A Dataset is broken into several components. Each component has a different purpose:

Each component is described in detail below.

Body

Body is the principle content of a dataset. A dataset body is the subject which all other fields describe and qualify.

Supported Data Formats:

• csv - comma-separated values
• json - javascript object notation
• xlsx - microsoft excel open xml spreadsheet
• cbor - concise binary object representation

The structure of the data stored is arbitrary, with one important exception: the top level of body must be either an object or an array. scalar types like int, bool, float, or strings are not valid types. Keep in mind that it's perfectly valid to wrap a scalar type (for example, a string) in an array to obtain a valid body.

Commit

Commit encapsulates information about changes to a dataset in relation to other entries in a given history. Commit is directly analogous to the concept of a Commit Message in the git version control system. A full commit defines the administrative metadata of a dataset, answering "who made this dataset, when, and why".

commit fields:

additional data types:

User

example

  {
    "id": "user_id_12340584",
    "fullname": "sean carter",
    "email": "hova@jayz.com"
  }

Meta

Meta contains human-readable descriptive metadata that qualifies and distinguishes a dataset. Well-defined Meta should aid in making datasets Findable by describing a dataset in generalizable taxonomies that can aggregate across other dataset documents. Because dataset documents are intended to interoperate with many other data storage and cataloging systems, meta fields and conventions are derived from existing metadata formats whenever possible.

All of the meta fields below must be well-formed, valid values. However: The Meta section of a dataset supports arbitrary metadata. This means you can place additional values not listed here & qri will store them as-is, without any additional validation.

Another important note here: qri software doesn't leverage things like identifier, downloadPath, homePath, etc.. Qri considers all fields here descriptive metadata, as opposed to structural metadata. In practice this means qri User interfaces may leverage the meta component for the purposes of correlation and display. Information stored in the meta section is not intended for use by machines to interpret the dataset. For example, setting the identifier is of no significance to qri, it's included here for interoperability with other specifications like DCAT

meta fields:

additional data types:

Citation

example:

  {
    "name" : "sean carter",
    "url" : "https://jayz.com",
    "email" : "hova@jayz.com"
  }

License

example:

  {
    "type" : "CC-BY-2",
    "url" : "https://creativecommons.org/licenses/by/2.0/"
  }

Structure

Structure defines the characteristics of a dataset document necessary for a machine to interpret the dataset body. Structure fields are things like the encoding data format (JSON,CSV,etc.), length of the dataset body in bytes, stored in a rigid form intended for machine use. A well defined structure & accompanying software should allow the end user to spend more time focusing on the data itself.

Two dataset documents that both have a defined structure will have some degree of natural interoperability, depending first on the amount of detail provided in a dataset's structure, and then by the natural comparability of the datasets.

structure fields:

Transform

Transform is a record of executing a transformation on data. Transforms can theoretically be anything from an SQL query, a jupyter notebook, the state of an ETL pipeline, etc, so long as the input is zero or more datasets, and the output is a single dataset Ideally, transforms should contain all the machine-necessary bits to deterministicly execute the algorithm referenced in "ScriptPath".

transform fields: