Google File System - Design


GFS is optimized for Google's core data storage and usage needs (primarily the search engine), which can generate enormous amounts of data that needs to be retained; Google File System grew out of an earlier Google effort, "BigFiles", developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford. Files are divided into fixed-size chunks of 64 megabytes, similar to clusters or sectors in regular file systems, which are only extremely rarely overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google's computing clusters, dense nodes which consist of cheap, "commodity" computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data throughputs, even when it comes at the cost of latency.

A GFS cluster consists of multiple nodes.These nodes are divided into two types: one Master node and a large number of Chunkservers. Each file is divided into fixed-size chunks. Chunkservers store these chunks. Each chunk is assigned a unique 64-bit label by the master node at the time of creation, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated several times throughout the network, with the minimum being three, but even more for files that have high end-in demand or need more redundancy.

The Master server doesn't usually store the actual chunks, but rather all the metadata associated with the chunks, such as the tables mapping the 64-bit labels to chunk locations and the files they make up, the locations of the copies of the chunks, what processes are reading or writing to a particular chunk, or taking a "snapshot" of the chunk pursuant to replicate it (usually at the instigation of the Master server, when, due to node failures, the number of copies of a chunk has fallen beneath the set number). All this metadata is kept current by the Master server periodically receiving updates from each chunk server ("Heart-beat messages").

Permissions for modifications are handled by a system of time-limited, expiring "leases", where the Master server grants permission to a process for a finite period of time during which no other process will be granted permission by the Master server to modify the chunk. The modifying chunkserver, which is always the primary chunk holder, then propagates the changes to the chunkservers with the backup copies. The changes are not saved until all chunkservers acknowledge, thus guaranteeing the completion and atomicity of the operation.

Programs access the chunks by first querying the Master server for the locations of the desired chunks; if the chunks are not being operated on (e.g. no outstanding leases exist), the Master replies with the locations, and the program then contacts and receives the data from the chunkserver directly (similar to Kazaa and its supernodes).

As opposed to other file systems, GFS is not implemented in the kernel of an operating system, but is instead provided as a userspace library.

Read more about this topic:  Google File System

Other articles related to "design, designs":

56th Tony Awards - The Ceremony
... a Play, Direction of a Musical, Book of a Musical, Original Score, Choreography, Costume Design, Lighting Design and Scenic Design ...
Terminology - Process Design
... "Process design" (in contrast to "design process" mentioned above) refers to the planning of routine steps of a process aside from the expected result ... Processes (in general) are treated as a product of design, not the method of design ... found the term useful to describe the design of business processes as well as manufacturing processes ...
Boeing 747 - Design
... For more design details, see Boeing 747-400, 747-8, and 747SP ... The Boeing 747 is a large, wide-body (two-aisle) airliner with four wing-mounted engines ...
Shea Stadium - Features - Design
... The design also allowed for Shea Stadium to be expandable to 90,000 seats (by completely enclosing the grandstand), or to be later enclosed by a dome if ...
Very-large-scale Integration - Structured Design
... Structured VLSI design is a modular methodology originated by Carver Mead and Lynn Conway for saving microchip area by minimizing the interconnect fabrics area ... In complex designs this structuring may be achieved by hierarchical nesting ... Structured VLSI design had been popular in the early 1980s, but lost its popularity later because of the advent of placement and routing tools wasting a lot of ...

Famous quotes containing the word design:

    Humility is often only the putting on of a submissiveness by which men hope to bring other people to submit to them; it is a more calculated sort of pride, which debases itself with a design of being exalted; and though this vice transform itself into a thousand several shapes, yet the disguise is never more effectual nor more capable of deceiving the world than when concealed under a form of humility.
    François, Duc De La Rochefoucauld (1613–1680)

    For I choose that my remembrances of him should be pleasing, affecting, religious. I will love him as a glorified friend, after the free way of friendship, and not pay him a stiff sign of respect, as men do to those whom they fear. A passage read from his discourses, a moving provocation to works like his, any act or meeting which tends to awaken a pure thought, a flow of love, an original design of virtue, I call a worthy, a true commemoration.
    Ralph Waldo Emerson (1803–1882)

    We find that Good and Evil happen alike to all Men on this Side of the Grave; and as the principle Design of Tragedy is to raise Commiseration and Terror in the Minds of the Audience, we shall defeat this great End, if we always make Virtue and Innocence happy and successful.
    Joseph Addison (1672–1719)