HDF5 File Format Specification Version 2.0

This field contains a constant value and can be used to quickly identify a file as being an HDF5 file. The constant value is designed to allow easy identification of an HDF5 file and to allow certain types of data corruption to be detected. The file signature of an HDF5 file always contains the following values:

This signature both identifies the file as an HDF5 file and provides for immediate detection of common file-transfer problems. The first two bytes distinguish HDF5 files on systems that expect the first two bytes to identify the file type uniquely. The first byte is chosen as a non-ASCII value to reduce the probability that a text file may be misrecognized as an HDF5 file; also, it catches bad file transfers that clear bit 7. Bytes two through four name the format. The CR-LF sequence catches bad file transfers that alter newline sequences. The control-Z character stops file display under MS-DOS. The final line feed checks for the inverse of the CR-LF translation problem. (This is a direct descendent of the PNG file signature.)

This field is present in version 0+ of the superblock.

This value is used to determine the format of the information in the superblock. When the format of the information in the superblock is changed, the version number is incremented to the next integer and can be used to determine how the information in the superblock is formatted.

Values of 0, 1 and 2 are defined for this field. (The format of version 2 is described below, not here)

This field is present in version 0+ of the superblock.

This value is used to determine the format of the information in the file's free-space Information.

The only value currently valid in this field is '0', which indicates that the free space index is formatted as described below.

This field is present in version 0 and 1 of the superblock.

This value is used to determine the format of the information in the Root Group Symbol Table Entry. When the format of the information in that field is changed, the version number is incremented to the next integer and can be used to determine how the information in the field is formatted.

The only value currently valid in this field is '0', which indicates that the root group symbol table entry is formatted as described below.

This field is present in version 0 and 1 of the superblock.

This value is used to determine the format of the information in a shared object header message. Since the format of the shared header messages differs from the other private header messages, a version number is used to identify changes in the format.

The only value currently valid in this field is '0', which indicates that shared header messages are formatted as described below.

This field is present in version 0 and 1 of the superblock.

This value contains the number of bytes used to store addresses in the file. The values for the addresses of objects in the file are offsets relative to a base address, usually the address of the superblock signature. This allows a wrapper to be added after the file is created without invalidating the internal offset locations.

This field is present in version 0+ of the superblock.

This value contains the number of bytes used to store the size of an object.

This field is present in version 0+ of the superblock.

Each leaf node of a group B-tree will have at least this many entries but not more than twice this many. If a group has a single leaf node then it may have fewer entries.

This value must be greater than zero.

See the description of B-trees below.

This field is present in version 0 and 1 of the superblock.

Each internal node of a group B-tree will have at least this many entries but not more than twice this many. If the group has only one internal node then it might have fewer entries.

This value must be greater than zero.

See the description of B-trees below.

This field is present in version 0 and 1 of the superblock.

This value contains flags to indicate information about the consistency of the information contained within the file. Currently, the following bit flags are defined:

• Bit 0 set indicates that the file is opened for write-access.
• Bit 1 set indicates that the file has been verified for consistency and is guaranteed to be consistent with the format defined in this document.
• Bits 2-31 are reserved for future use.

This field is present in version 0+ of the superblock.

Each internal node of an indexed storage B-tree will have at least this many entries but not more than twice this many. If the index storage B-tree has only one internal node then it might have fewer entries.

This value must be greater than zero.

See the description of B-trees below.

This field is present in version 1 of the superblock.

This is the absolute file address of the first byte of the HDF5 data within the file. The library currently constrains this value to be the absolute file address of the superblock itself when creating new files; future versions of the library may provide greater flexibility. When opening an existing file and this address does not match the offset of the superblock, the library assumes that the entire contents of the HDF5 file have been adjusted in the file and adjusts the base address and end of file address to reflect their new positions in the file. Unless otherwise noted, all other file addresses are relative to this base address.

This field is present in version 0+ of the superblock.

Free-space management is not yet defined in the HDF5 file format and is not handled by the library. Currently this field always contains the undefined address.

This field is present in version 0 and 1 of the superblock.

This is the absolute file address of the first byte past the end of all HDF5 data. It is used to determine whether a file has been accidently truncated and as an address where file data allocation can occur if space from the free list is not used.

This field is present in version 0+ of the superblock.

This is the relative file address of the file driver information block which contains driver-specific information needed to reopen the file. If there is no driver information block then this entry should be the undefined address.

This field is present in version 0 and 1 of the superblock.

This is the symbol table entry of the root group, which serves as the entry point into the group graph for the file.

This field is present in version 0 and 1 of the superblock.

This field is the same as described for versions 0 and 1 of the superblock.

This field has a value of 2 and has the same meaning as for versions 0 and 1.

This field is the same as described for versions 0 and 1 of the superblock.

This field is the same as described for versions 0 and 1 of the superblock.

This field is the same as desribed for versions 0 and 1 except that it is smaller (the number of reserved bits has been reduced from 30 to 6).

This field is the same as described for versions 0 and 1 of the superblock.

The field is the address of the object header for the superblock extension. If there is no extension then this entry should be the undefined address.

This field is the same as described for versions 0 and 1 of the superblock.

This is the address of the root group object header, which serves as the entry point into the group graph for the file.

The checksum for the superblock.

The version number of the Driver Information Block. This document describes version 0.

The size in bytes of the Driver Information field.

This is an eight-byte ASCII string without null termination which identifies the driver and/or version number of the Driver Information Block. The predefined driver encoded in this field by the HDF5 library is identified by the letters NCSA followed by the first four characters of the driver name. If the Driver Information block is not the original version then the last letter(s) of the identification will be replaced by a version number in ASCII, starting with 0.

Identification for user-defined drivers is also eight-byte long. It can be arbitrary but should be unique to avoid the four character prefix "NCSA".

These fields are reserved and should always be zero.

This field Specifies the virtual address at which the member file starts.

N is the number of member files.

This field is the end of the allocated address for the member file.

This field is the null-terminated name of the member file and its length should be multiples of 8 bytes. Additional bytes will be padded with NULLs. The default naming convention is %s-X.h5, where X is one of the letters s (for superblock), b (for B-tree), r (for raw data), g (for global heap), l (for local heap), and o (for object header). The name of the whole HDF5 file will substitute the %s in the string.

This field is the size of the member file in the family of files.

The ASCII character string "TREE" is used to indicate the beginning of a B-link tree node. This gives file consistency checking utilities a better chance of reconstructing a damaged file.

Each B-link tree points to a particular type of data. This field indicates the type of data as well as implying the maximum degree K of the tree and the size of each Key field.

The node level indicates the level at which this node appears in the tree (leaf nodes are at level zero). Not only does the level indicate whether child pointers point to sub-trees or to data, but it can also be used to help file consistency checking utilities reconstruct damanged trees.

This determines the number of children to which this node points. All nodes of a particular type of tree have the same maximum degree, but most nodes will point to less than that number of children. The valid child pointers and keys appear at the beginning of the node and the unused pointers and keys appear at the end of the node. The unused pointers and keys have undefined values.

This is the relative file address of the left sibling of the current node. If the current node is the left-most node at this level then this field is the undefined address.

This is the relative file address of the right sibling of the current node. If the current node is the right-most node at this level then this field is the undefined address.

Each tree has 2K+1 keys with 2K child pointers interleaved between the keys. The number of keys and child pointers actually containing valid values is determined by the node's Entries Used field. If that field is N then the B-link tree contains N child pointers and N+1 keys.

The format and size of the key values is determined by the type of data to which this tree points. The keys are ordered and are boundaries for the contents of the child pointer; that is, the key values represented by child N fall between Key N and Key N+1. Whether the interval is open or closed on each end is determined by the type of data to which the tree points.

The format of the key depends on the node type. For nodes of node type 0 (group nodes), the key is formatted as follows:

For nodes of node type 1 (chunked raw data nodes), the key is formatted as follows:

The tree node contains file addresses of subtrees or data depending on the node level. Nodes at Level 0 point to data addresses, either raw data chunks or group nodes. Nodes at non-zero levels point to other nodes of the same B-tree.

For raw data chunk nodes, the child pointer is the address of a single raw data chunk. For group nodes, the child pointer points to a symbol table, which contains information for multiple symbol table entries.

The ASCII character string "BTHD" is used to indicate the header of a version 2 B-link tree node.

The version number for this B-tree header. This document describes version 0.

This field indicates the type of B-tree:

Value	Description
0	A "testing" B-tree, this value should not be used for storing records in actual HDF5 files.
1	This B-tree is used for indexing indirectly accessed, non-filtered 'huge' fractal heap objects.
2	This B-tree is used for indexing indirectly accessed, filtered 'huge' fractal heap objects.
3	This B-tree is used for indexing directly accessed, non-filtered 'huge' fractal heap objects.
4	This B-tree is used for indexing directly accessed, filtered 'huge' fractal heap objects.
5	This B-tree is used for indexing the 'name' field for links in indexed groups.
6	This B-tree is used for indexing the 'creation order' field for links in indexed groups.
7	This B-tree is used for indexing shared object header messages.
8	This B-tree is used for indexing the 'name' field for indexed attributes.
9	This B-tree is used for indexing the 'creation order' field for indexed attributes.

The format of records for each type is described below.

This is the size in bytes of all B-tree nodes.

This field is the size in bytes of the B-tree record.

This is the depth of the B-tree.

The percent full that a node needs to increase above before it is split.

The percent full that a node needs to be decrease below before it is split.

This is the address of the root B-tree node. A B-tree with no records will have the undefined address in this field.

This is the number of records in the root node.

This is the total number of records in the entire B-tree.

This is the checksum for the B-tree header.

The ASCII character string "BTIN" is used to indicate the internal node of a B-link tree.

The version number for this B-tree internal node. This document describes version 0.

This field is the type of the B-tree node. It should always be the same as the B-tree type in the header.

The size of this field is determined by the number of records for this node and the record size (from the header). The format of records depends on the type of B-tree.

This field is the address of the child node pointed to by the internal node.

This is the number of records in the child node pointed to by the corresponding Node Pointer.

The number of bytes used to store this field is determined by the maximum possible number of records able to be stored in the child node.

The maximum number of records in a child node is computed in the following way: Subtract the fixed size overhead for the child node (e.g. it's signature, version, checksum, etc. and one pointer triplet of information for the child node (because there is one more pointer triplet than records in each internal node)) from the size of nodes for the B-tree and dividing that result by the size of a record plus the pointer triplet of information stored to reach each child node from this node.

Note that leaf nodes don't encode any child pointer triplets, so the maximum number of records in a leaf node is just the node size minus the leaf node overhead, divided by the record size.

Also note that the first level of internal nodes above the leaf nodes don't encode the Total Number of Records in Child Node value in the child pointer triplets (since it is the same as the Number of Records in Child Node), so the maximum number of records in these nodes is computed with the equation above, but using (Child Pointer, Number of Records in Child Node) pairs instead of triplets.

The number of bytes used to encode this field is the least number of bytes required to encode the maximum number of records in a child node value for the child nodes below this level in the B-tree.

For example, if the maximum number of child records is 123, one byte will be used to encode these values in this node, if the maximum number of child records is 20000, two bytes will be used to encode these values in this node, etc. The maximum number of bytes used to encode these values is 8 (i.e. an unsigned 64-bit integer).

This is the total number of records for the node pointed to by the corresponding Node Pointer and all its children. This field exists only in nodes whose depth in the B-tree node is greater than 1 (i.e. the "twig" internal nodes, just above leaf nodes, don't store this field in their child node pointers).

The number of bytes used to store this field is determined by the maximum possible number of records able to be stored in the child node and its descendents.

The maximum possible number of records able to be stored in a child node and its descendents is computed iteratively, in the following way: The maximum number of records in a leaf node is computed, then that value is used to compute the maximum possible number of records in the first level of internal nodes above the leaf nodes. Multiplying these two values together determines the maximum possible number of records in child node pointers for the level of nodes two levels above leaf nodes. This process is continued up to any level in the B-tree.

The number of bytes used to encode this value is computed in the same way as for the Number of Records in Child Node field.

This is the checksum for this node.

The ASCII character string "BTLF" is used to indicate the leaf node of a version 2 B-link tree.

The version number for this B-tree leaf node. This document describes version 0.

This field is the type of the B-tree node. It should always be the same as the B-tree type in the header.

The size of this field is determined by the number of records for this node and the record size (from the header). The format of records depends on the type of B-tree.

This is the checksum for this node.

The address of the huge object in the file.

The length of the huge object in the file.

The heap ID for the huge object.

The address of the filtered huge object in the file.

The length of the filtered huge object in the file.

A 32-bit bitfield indicating which filters have been skipped for this chunk. Each filter has an index number in the pipeline (starting at 0, with the first filter to apply) and if that filter is skipped, the bit corresponding to its index is set.

The size of the de-filtered huge object in memory.

The heap ID for the huge object.

The address of the huge object in the file.

The length of the huge object in the file.

The address of the filtered huge object in the file.

The length of the filtered huge object in the file.

A 32-bit bitfield indicating which filters have been skipped for this chunk. Each filter has an index number in the pipeline (starting at 0, with the first filter to apply) and if that filter is skipped, the bit corresponding to its index is set.

The size of the de-filtered huge object in memory.

This field is hash value of the name for the link. The hash value is the Jenkins' lookup3 checksum algorithm applied to the link's name.

This is a 7-byte sequence of bytes and is the heap ID for the link record in the group's fractal heap.

This field is the creation order value for the link.

This is a 7-byte sequence of bytes and is the heap ID for the link record in the group's fractal heap.

This field Indicates the location where the message is stored:

Value	Description
0	Shared message is stored in shared message index heap.
1	Shared message is stored in object header.

This field is hash value of the shared message. The hash value is the Jenkins' lookup3 checksum algorithm applied to the shared message.

The number of objects which reference this message.

This is a 8-byte sequence of bytes and is the heap ID for the shared message in the shared message index's fractal heap.

This field Indicates the location where the message is stored:

Value	Description
0	Shared message is stored in shared message index heap.
1	Shared message is stored in object header.

This field is hash value of the shared message. The hash value is the Jenkins' lookup3 checksum algorithm applied to the shared message.

The object header message type of the shared message.

This field indicates that the shared message is the n'th message of its type in the specified object header.

The address of the object header containing the shared message.

This is a 8-byte sequence of bytes and is the heap ID for the attribute in the object's attribute fractal heap.

The object header message flags for the attribute message.

This field is the creation order value for the attribute.

This field is hash value of the name for the attribute. The hash value is the Jenkins' lookup3 checksum algorithm applied to the attribute's name.

This is a 8-byte sequence of bytes and is the heap ID for the attribute in the object's attribute fractal heap.

The object header message flags for the attribute message.

This field is the creation order value for the attribute.

The ASCII character string "SNOD" is used to indicate the beginning of a symbol table node. This gives file consistency checking utilities a better chance of reconstructing a damaged file.

The version number for the symbol table node. This document describes version 1. (There is no version '0' of the symbol table node)

Although all symbol table nodes have the same length, most contain fewer than the maximum possible number of link entries. This field indicates how many entries contain valid data. The valid entries are packed at the beginning of the symbol table node while the remaining entries contain undefined values.

Each link has an entry in the symbol table node. The format of the entry is described below. There are 2K entries in each group node, where K is the "Group Leaf Node K" value from the superblock.

This is the byte offset into the group's local heap for the name of the link. The name is null terminated.

Every object has an object header which serves as a permanent location for the object's metadata. In addition to appearing in the object header, some of the object's metadata can be cached in the scratch-pad space.

The cache type is determined from the object header. It also determines the format for the scratch-pad space:

Type:	Description:
0	No data is cached by the group entry. This is guaranteed to be the case when an object header has a link count greater than one.
1	Group object header metadata is cached in the scratch-pad space. This implies that the symbol table entry refers to another group.
2	The entry is a symbolic link. The first four bytes of the scratch-pad space are the offset into the local heap for the link value. The object header address will be undefined.

These four bytes are present so that the scratch-pad space is aligned on an eight-byte boundary. They are always set to zero.

This space is used for different purposes, depending on the value of the Cache Type field. Any metadata about an object represented in the scratch-pad space is duplicated in the object header for that object.

Furthermore, no data is cached in the group entry scratch-pad space if the object header for the object has a link count greater than one.

This is the file address for the root of the group's B-tree.

This is the file address for the group's local heap, in which are stored the group's symbol names.

The value of a symbolic link (that is, the name of the thing to which it points) is stored in the local heap. This field is the 4-byte offset into the local heap for the start of the link value, which is null terminated.

The ASCII character string "GCOL" is used to indicate the beginning of a collection. This gives file consistency checking utilities a better chance of reconstructing a damaged file.

Each collection has its own version number so that new collections can be added to old files. This document describes version one (1) of the collections (there is no version zero (0)).

This is the size in bytes of the entire collection including this field. The default (and minimum) collection size is 4096 bytes which is a typical file system block size. This allows for 127 16-byte heap objects plus their overhead (the collection header of 16 bytes and the 16 bytes of information about each heap object).

The objects are stored in any order with no intervening unused space.

Global Heap Object 0 (zero), when present, represents the free space in the collection. Free space always appears at the end of the collection. If the free space is too small to store the header for Object 0 (described below) then the header is implied and the collection contains no free space.

Each object has a unique identification number within a collection. The identification numbers are chosen so that new objects have the smallest value possible with the exception that the identifier 0 always refers to the object which represents all free space within the collection.

All heap objects have a reference count field. An object which is referenced from some other part of the file will have a positive reference count. The reference count for Object 0 is always zero.

Zero padding to align next field on an 8-byte boundary.

This is the size of the object data stored for the object. The actual storage space allocated for the object data is rounded up to a multiple of eight.

The object data is treated as a one-dimensional array of bytes to be interpreted by the caller.

This field is the address of the global heap collection where the data object is stored.

This field is the index of the data object within the global heap collection.

The ASCII character string "FRHP" is used to indicate the beginning of a fractal heap header. This gives file consistency checking utilities a better chance of reconstructing a damaged file.

This document describes version 0.

This is the length in bytes of heap object IDs for this heap.

This is the size in bytes of the encoded I/O Filter Information.

This field is the heap status flag and is a bit-field indicating additional information about the fractal heap.

Bit(s)	Description
0	If set, the ID value to use for huge object has wrapped around. If the value for the Next Huge Object ID has wrapped around, each new huge object inserted into the heap will require a search for an ID value.
1	If set, the direct blocks in the heap are checksummed.
2-7	Reserved

This is the maximum size of managed objects allowed in the heap. Objects greater than this this are 'huge' objects and will be stored in the file directly, rather than in a direct block for the heap.

This is the next ID value to use for a huge object in the heap.

This is the address of the v2 B-tree used to track huge objects in the heap. The type of records stored in the v2 B-tree will be determined by whether the address & length of a huge object can fit into a heap ID (if yes, it's a "directly" accessed huge object) and whether there is a filter used on objects in the heap.

This is the total amount of free space in managed direct blocks (in bytes).

This is the address of the Free-Space Manager for managed blocks.

This is the total amount of managed space in the heap (in bytes), essentially the upper bound of the heap's linear address space.

This is the total amount of managed space (in bytes) actually allocated in the heap. This can be less than the Amount of Managed Space in Heap field, if some direct blocks in the heap's linear address space are not allocated.

This is the linear heap offset where the next direct block should be allocated at (in bytes). This may be less than the Amount of Managed Space in Heap value because the heap's address space is increased by a "row" of direct blocks at a time, rather than by single direct block increments.

This is the number of managed objects in the heap.

This is the total size of huge objects in the heap (in bytes).

This is the number of huge objects in the heap.

This is the total size of tiny objects that are packed in heap IDs (in bytes).

This is the number of tiny objects that are packed in heap IDs.

This is the number of columns in the doubling table for managed blocks. This value must be a power of two.

This is the starting block size to use in the doubling table for managed blocks (in bytes). This value must be a power of two.

This is the maximum size allowed for a managed direct block. Objects inserted into the heap that are larger than this value (less the # of bytes of direct block prefix/suffix) are stored as 'huge' objects. This value must be a power of two.

This is the maximum size of the heap's linear address space for managed objects (in bytes). The value stored is the log2 of the actual value, that is: the # of bits of the address space. 'Huge' and 'tiny' objects aren't counted in this value, since they don't store objects in the linear address space of the heap.

This is the starting number of rows for the root indirect block. A value of 0 indicates that the root indirect block will have the maximum number of rows needed to address the heap's Maximum Heap Size.

This is the address of the root block for the heap. It can be the undefined address if there is no data in the heap. It either points to a direct block (if the Current # of Rows in the Root Indirect Block value is 0), or an indirect block.

This is the current number of rows in the root indirect block. A value of 0 indicates that Address of Root Block points to direct block instead of indirect block.

This is the size of the root direct block, if filters are applied to heap objects (in bytes). This field is only stored in the header if the I/O Filters' Encoded Length is greater than 0.

This is the filter mask for the root direct block, if filters are applied to heap objects. This mask has the same format as that used for the filter mask in chunked raw data records in a v1 B-tree. This field is only stored in the header if the I/O Filters' Encoded Length is greater than 0.

This is the I/O filter information encoding direct blocks and huge objects, if filters are applied to heap objects. This field is encoded as a Filter Pipeline message. The size of this field is determined by I/O Filters' Encoded Length.

This is the checksum for the header.