next up previous contents
Next: Dataset Structure Up: Specification of the Format Previous: Definitions and Concepts   Contents


Overview of the Dataset Structure

A dataset written in the df standard consists of a series of records. A record is defined to be a series of bytes which are to be read or written together. An example would be a Fortran record as written with a single WRITE statement. The concept of a record is especially important for certain languages (such as Fortran) and for certain binary data representations (such as XDR). On the other hand, a record might consist of a set of bytes in a uniform bit stream with no delimiters. Each record defined in the df format begins with a specific numeric code which identifies its type. Record types are organized into functional groups. Within a group, and from group to group, a lower numeric ID code generally indicates its placement further towards the beginning of the dataset.

Because the df format should be usable with any storage architecture or language, it was designed to function with purely sequential data streams used with languages (such as Fortran) which need to know how long a record is before it is read. These design constraints do not prevent datasets from being accessed in a non-sequential fashion or by more sophisticated languages, but the format must at least work for this lowest common denominator.

In order that the next record type in a dataset be known (so that its length is known as well), the order of the records is fixed. In addition, the length of some records depends upon information read in previous records.

All of the information needed to read and decode the dataset is present in the dataset. Metadata is recorded in appropriate records in the proper record groups. Some groups are composed of only one record type, while others consist of several. Some record types are mandatory, and others are optional. Some types of records can be written more than once, and these will be further indexed so that they can be processed in a particular order. See Section 2.3.2 for a discussion of what indices mean in various contexts.

Following the metadata come the data records themselves. The information in the previous records fully describe the structure and format of the data, making it possible to read.



Subsections
next up previous contents
Next: Dataset Structure Up: Specification of the Format Previous: Definitions and Concepts   Contents
Eric Nash 2003-09-25