Comments on: Huge File Storage in Database instead of File System http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/ Founder of NeuroSpeech Thu, 29 Jul 2010 10:04:23 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Sachin Modak http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-1094 Sachin Modak Mon, 21 Dec 2009 21:38:43 +0000 http://akashkava.com/blog/?p=127#comment-1094 Hi Akash, I was reading through this blog and I was wondering if you considered having a flat file/control file design to keep track of files? Also something about storing files in a database just does not seem very performance centric? I was wondering if we can discuss this sometime? BTW, we can even do a phone call. Let me know. sachin.modak@gmail.com Hi Akash,

I was reading through this blog and I was wondering if you considered having a flat file/control file design to keep track of files? Also something about storing files in a database just does not seem very performance centric? I was wondering if we can discuss this sometime? BTW, we can even do a phone call. Let me know.

sachin.modak@gmail.com

]]> By: Akash Kava http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-1091 Akash Kava Wed, 09 Dec 2009 16:34:24 +0000 http://akashkava.com/blog/?p=127#comment-1091 FileStream is good but I dont know if it is ACID compliant, it will be faster but I dont know about replication and transaction. As far as I know its raw file access within SQL Context, but it is not recommended over Blob. Yes if your files are big, and they are of less financial value, like all crap uploaded to youtube, for such data you can use filestream. But if you create a document management service, where your document (files) are of bigger financial value, then you should not use filestream. FileStream is good but I dont know if it is ACID compliant, it will be faster but I dont know about replication and transaction. As far as I know its raw file access within SQL Context, but it is not recommended over Blob. Yes if your files are big, and they are of less financial value, like all crap uploaded to youtube, for such data you can use filestream. But if you create a document management service, where your document (files) are of bigger financial value, then you should not use filestream.

]]> By: ypaez http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-1089 ypaez Sat, 05 Dec 2009 18:43:08 +0000 http://akashkava.com/blog/?p=127#comment-1089 Great job, nice & easy solution, but looking arround this topic I found that SQL Server 2008 include a new feature called "FILESTREAM Storage" http://technet.microsoft.com/en-us/library/bb933995.aspx Have you been tested this feature? Does it have a better performance that your solution? Great job, nice & easy solution, but looking arround this topic I found that SQL Server 2008 include a new feature called “FILESTREAM Storage” http://technet.microsoft.com/en-us/library/bb933995.aspx
Have you been tested this feature? Does it have a better performance that your solution?

]]> By: PUK http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-1087 PUK Wed, 28 Oct 2009 15:15:46 +0000 http://akashkava.com/blog/?p=127#comment-1087 I found this article helpful to bolster what I already instinctively felt that database is better than filesystem. It's much more portable, atomic, relational, transactional, etc, etc. so thanks. BTW, This area of the page (Leave a Reply) renders ok in IE (v7) but in Firefox the captcha input is over the email label! N.B. This is my 4th attempt at getting the captcha right... PUK I found this article helpful to bolster what I already instinctively felt that database is better than filesystem. It’s much more portable, atomic, relational, transactional, etc, etc. so thanks.

BTW, This area of the page (Leave a Reply) renders ok in IE (v7) but in Firefox the captcha input is over the email label!

N.B. This is my 4th attempt at getting the captcha right…

PUK

]]> By: Akash http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-374 Akash Wed, 26 Aug 2009 15:59:41 +0000 http://akashkava.com/blog/?p=127#comment-374 1. My post doesnt focus on which solution, MS or Linux to go for, it says about whether file system or database and I am not against anyone storing it in file system or not. 2. File system are not transactional, they are not ACID compliant. 3. People can still use mysql, we are not against anyone, we are talking here about locking and cpu overhead and disc life cycle and durability. 1. My post doesnt focus on which solution, MS or Linux to go for, it says about whether file system or database and I am not against anyone storing it in file system or not.
2. File system are not transactional, they are not ACID compliant.
3. People can still use mysql, we are not against anyone, we are talking here about locking and cpu overhead and disc life cycle and durability.

]]> By: Sarwar Faruque http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-373 Sarwar Faruque Wed, 26 Aug 2009 09:32:53 +0000 http://akashkava.com/blog/?p=127#comment-373 While having no intention of turning this into a Linux vs. Microsoft topic, have you considered using Linux for a file management system? Two years ago, I built a web based document management systems for an university. The system had user access permissions and document version tracking, running on a LAMP stack. While the permission and tracking part was shouldered by the database, all the files were stored on a separate Linux server with a scheduler automatically backing up everything in 2 hour intervals. To date, the number of documents have been steadily growing and while it hasn't hit the 1 million marker, it certainly has passed the half-million point in the last two years. While having no intention of turning this into a Linux vs. Microsoft topic, have you considered using Linux for a file management system? Two years ago, I built a web based document management systems for an university. The system had user access permissions and document version tracking, running on a LAMP stack. While the permission and tracking part was shouldered by the database, all the files were stored on a separate Linux server with a scheduler automatically backing up everything in 2 hour intervals. To date, the number of documents have been steadily growing and while it hasn’t hit the 1 million marker, it certainly has passed the half-million point in the last two years.

]]> By: Akash http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-372 Akash Wed, 26 Aug 2009 07:11:05 +0000 http://akashkava.com/blog/?p=127#comment-372 I appreciate your comments, unix/linux based systems optimized for file system storage only like facebook and other companies did can definitely be better then MS or DB solution. I especially mentioned that distributed file systems like hadoop and such systems basically requires good man power expert in linux/unix technologies mainly at C and compiler level. But they are difficult to find, hire and maintain. Hiring 1 system level expert is 100 times difficult then hiring 10 SQL+ASP.NET guys. Thats why I also mentioned that from 1GB to 1TB kind of solution, this database based system makes easier bet. Beyond that certainly you need dedicated staff to manage your files !! I appreciate your comments, unix/linux based systems optimized for file system storage only like facebook and other companies did can definitely be better then MS or DB solution. I especially mentioned that distributed file systems like hadoop and such systems basically requires good man power expert in linux/unix technologies mainly at C and compiler level. But they are difficult to find, hire and maintain. Hiring 1 system level expert is 100 times difficult then hiring 10 SQL+ASP.NET guys. Thats why I also mentioned that from 1GB to 1TB kind of solution, this database based system makes easier bet. Beyond that certainly you need dedicated staff to manage your files !!

]]> By: Florin Diaconeasa http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-366 Florin Diaconeasa Tue, 25 Aug 2009 20:52:54 +0000 http://akashkava.com/blog/?p=127#comment-366 hmm.. yes i didn't think it thru :) but also... holding such a number of files would prove to be better on a unix machine, especially open-source, as there are multiple solutions for this kind of problems. For example, stripping the rights management on a file system(which is not such a crazy idea, already being implemented by facebook) plus other optimizations. Not trying to start a war here :), just pointing out some facts. But then again, quite hard to do smthg like this when your company's profile is MS based hmm.. yes i didn’t think it thru :)

but also… holding such a number of files would prove to be better on a unix machine, especially open-source, as there are multiple solutions for this kind of problems.

For example, stripping the rights management on a file system(which is not such a crazy idea, already being implemented by facebook) plus other optimizations.

Not trying to start a war here :) , just pointing out some facts. But then again, quite hard to do smthg like this when your company’s profile is MS based

]]> By: Akash http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-364 Akash Tue, 25 Aug 2009 10:59:12 +0000 http://akashkava.com/blog/?p=127#comment-364 If you do different files per table, they grow very slowly leading to huge fragmentation, problem is there is no auto grow value, if you look at MySQL's data folder fragmentation and MS SQL's data folder, we had it first in MySQL and then we went to MS SQL. If you do different files per table, they grow very slowly leading to huge fragmentation, problem is there is no auto grow value, if you look at MySQL’s data folder fragmentation and MS SQL’s data folder, we had it first in MySQL and then we went to MS SQL.

]]> By: Florin Diaconeasa http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/comment-page-1/#comment-363 Florin Diaconeasa Tue, 25 Aug 2009 07:08:24 +0000 http://akashkava.com/blog/?p=127#comment-363 Another solution could be partitionioning your data using mysql 5.1+. This way you would have better performance in writing and reading also. Also, mysql innodb engine supports different files for each table, so it's not necessary to have only one big file, as the default mysql innodb configuration does. :) If interested in more details, you've got the email :) Another solution could be partitionioning your data using mysql 5.1+. This way you would have better performance in writing and reading also. Also, mysql innodb engine supports different files for each table, so it’s not necessary to have only one big file, as the default mysql innodb configuration does. :)

If interested in more details, you’ve got the email :)

]]>