Comments on: Huge File Storage in Database instead of File System

By: Akash Kava

Akash Kava — Sun, 25 Sep 2011 06:31:21 +0000

Conversion should not be done at time of serving the file, instead most appropriate way would be to convert and store a converted file along with original file. And conversion should be batched as well.

By: Sanjay Bhaskar

Sanjay Bhaskar — Fri, 15 Jul 2011 05:31:18 +0000

Very good inputs,

I’ve a question.

If we save files like (.doc, .swf, etc..) we need to save this as Binary data. Would it be a performance intensive which needs the data conversion while serving this request? In case of file system i don’t it need conversion, i assume its a direct access to the file with out any conversion.

Hope my question make sense.

By: Ghazanfar

Ghazanfar — Tue, 15 Feb 2011 04:11:53 +0000

Very nice article Akash,

i was really looking for the comparison about the file saving in database or file system but could not find any good resource on the internet, but i am fortunate in this morning to see your article. it is really beneficial for me and my organization to take decission, because we handle very huge data.

Regards,
Ghazanffer Javaid

By: Akash Kava

Akash Kava — Tue, 08 Feb 2011 15:47:31 +0000

Thanks for your comments, however it’s not only transactions, but being able to query, replication and disk performance etc also needs to be considered for huge systems.

By: Michiel van Otegem

Michiel van Otegem — Tue, 08 Feb 2011 10:59:36 +0000

Akash, I read your article with great interest. You make some very valid points about the disadvantages of a file system, and considering the number and size of the files that will have to be stored in the system I'm currently working on, database storage is definitly the way to go. I'd like to point out however that Windows Vista SP1 / Windows 2008 and up support transactions (see Wikipedia).

By: Akash Kava

Akash Kava — Tue, 11 Jan 2011 20:01:52 +0000

Well we have not used FULLTEXT so far, because we are storing images, videos and photos, but I have heard bad about MyISAM as it keeps on crashing for fulltext. However, implementing a small fulltext is not that difficult if your query isnt going to be complex enough. If you generate a table with WordFiles with columns Word and FileID and store every word with corresponding FileID where it appears and index your Word with B+, you will get better fulltext with innodb if your search is not going to be complex.

By: Adriano

Adriano — Thu, 06 Jan 2011 16:21:53 +0000

Thanks Akash,
so how do you manage with FULLTEXT search?

Do you use MyIsam tables only for FULLTEXT and then join the queries?

I, for instance, have this kind of situation:

I’m creating a system with a Unique ID for any kind of data in the DB.

So I could have a new article ( Table “Articles” ) with GUID ’1′,
then an image with GUID ’2′ ( Table ‘Images’)
then a PDF with GUID ’3′ ( Table ‘PDF’ ) and so on.

The maybe for instance the images are divided in 2 tables:

1 in the id_generator table ( autoincrement )
1 innoDB with only 3 binary data ( smal. medium, large ) and GUID,
1 MyIsam with image title, description, keywords etc.

This means, that to avoid concurrency, each time I put a new record in the DB,
I MUST lock the tables to make sure that no other new ‘row’ added in the DB has the same GUID.

Does it make sense? Or I’m about to make a mess?

Thanks.

By: Akash Kava

Akash Kava — Thu, 06 Jan 2011 10:33:42 +0000

MyISAM will not work, the reason is, MyIASM uses Table Locks, so for every update it will lock entire table instead of one row and that causes loss of data if the files are of little bigger size (4K>) , so I will not recommend MyIASM. We did try once and it resulted in loss of data.

By: Adriano

Adriano — Thu, 06 Jan 2011 07:17:15 +0000

Hi, very detailed article.

I have a question.

Do you think that it might work fine even in MyIsam System?

I’m not talking about Terabytes of data and powerful servers.
I just would like to save small attachments ( images, pdfs, docs ) in the same line of an article data, in a basic linux server.

Eg.: Title, description, keywords, body text, image ( 100K ), pdf ( 200K), word (100K)

Do you think that might work? Or it will slow down? especially when I will do a FULLTEXT search for instance on ( Title, description, keywords, body text ) ?

Thanks.

Great Job,
A

By: Sachin Modak

Sachin Modak — Mon, 21 Dec 2009 21:38:43 +0000

Hi Akash,

I was reading through this blog and I was wondering if you considered having a flat file/control file design to keep track of files? Also something about storing files in a database just does not seem very performance centric? I was wondering if we can discuss this sometime? BTW, we can even do a phone call. Let me know.

sachin.modak@gmail.com