Hadoop Distributed File System
Posted in scalability Wed, 23 Aug 2006 05:49:00 GMT
I just ran across the Hadoop DFS which is an open source alternative to distributed file systems such as GoogleFS, OneFS and others. GoogleFS and OneFS are both proprietary so it's nice to finally have a FOSS solution. MySpace uses OneFS. From the Hadoop Wiki:
Hadoop's Distributed File System is designed to reliably store very large files across machines in a large cluster. It is inspired by the Google File System. Hadoop DFS stores each file as a sequence of blocks, all blocks in a file except the last block are the same size. Blocks belonging to a file are replicated for fault tolerance. The block size and replication factor are configurable per file. Files in HDFS are "write once" and have strictly one writer at any time.
Until now, I had only been aware of MogileFS for FOSS solutions, however MogileFS is designed for smaller files such as images and the others are designed for very large files. It will be interesting to see how much traction Hadoop DFS gets since it could be very useful and a good FOSS compliment to MogileFS. Hadoop is part of the Lucene Apache project.
