Skip to content

esm-catalog-spec.md

Latest commit

 

History

History
145 lines (115 loc) · 10.6 KB

esm-catalog-spec.md

File metadata and controls

145 lines (115 loc) · 10.6 KB

ESM Catalog Specification

This documents mirrors the [ESM Collection Specification](https://github.com/NCAR/esm-collection-spec/blob/master/collection-spec/collection-spec.md) and is updated as the specification evolves.

Overview

This document explains the structure and content of an ESM Catalog. A catalog provides metadata about the catalog, telling us what we expect to find inside and how to open it. The catalog is described is a single json file, inspired by the STAC spec.

The ESM Catalog specification consists of three parts:

Catalog Specification

The catalog specification provides metadata about the catalog, telling us what we expect to find inside and how to open it. The descriptor is a single json file, inspired by the STAC spec.

{
  "esmcat_version": "0.1.0",
  "id": "sample",
  "description": "This is a very basic sample ESM catalog.",
  "catalog_file": "sample_catalog.csv",
  "attributes": [
    {
      "column_name": "activity_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_activity_id.json"
    },
    {
      "column_name": "source_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_source_id.json"
    }
  ],
  "assets": {
    "column_name": "path",
    "format": "zarr"
  }
}

Catalog

The collection points to a single catalog. A catalog is a CSV file. The meaning of the columns in the csv file is defined by the parent collection.

activity_id,source_id,path
CMIP,ACCESS-CM2,gs://pangeo-data/store1.zarr
CMIP,GISS-E2-1-G,gs://pangeo-data/store1.zarr

Assets (Data Files)

The data assets can be either netCDF or Zarr. They should be either URIs or full filesystem paths.

Catalog fields

Element Type Description
esmcat_version string REQUIRED. The ESM Catalog version the catalog implements.
id string REQUIRED. Identifier for the catalog.
title string A short descriptive one-line title for the catalog.
description string REQUIRED. Detailed multi-line description to fully explain the catalog. CommonMark 0.28 syntax MAY be used for rich text representation.
catalog_file string REQUIRED. Path to a the CSV file with the catalog contents.
catalog_dict array If specified, it is mutually exclusive with catalog_file. An array of dictionaries that represents the data that would otherwise be in the csv.
attributes [Attribute Object] REQUIRED. A list of attribute columns in the data set.
assets Assets Object REQUIRED. Description of how the assets (data files) are referenced in the CSV catalog file.
aggregation_control Aggregation Control Object OPTIONAL. Description of how to support aggregation of multiple assets into a single xarray data set.

Attribute Object

An attribute object describes a column in the catalog CSV file. The column names can optionally be associated with a controlled vocabulary, such as the CMIP6 CVs, which explain how to interpret the attribute values.

Element Type Description
column_name string REQUIRED. The name of the attribute column. Must be in the header of the CSV file.
vocabulary string Link to the controlled vocabulary for the attribute in the format of a URL.

Assets Object

An assets object describes the columns in the CSV file relevant for opening the actual data files.

Element Type Description
column_name string REQUIRED. The name of the column containing the path to the asset. Must be in the header of the CSV file.
format string The data format. Valid values are netcdf, zarr, zarr2, zarr3, opendap or reference (kerchunk reference files). If specified, it means that all data in the catalog is the same type.
format_column_name string The column name which contains the data format, allowing for variable data types in one catalog. Mutually exclusive with format. 
 Zarr v3 is built on asynchronous operations, and requires `xarray_open_kwargs` to contain the following dictionary fragment:
 ```python
xarray_open_kwargs = {
    "storage_options" : {
        "remote_options" : {
            "async": true,
            ...
        },
        ...
    },
   ...
}
 ```

In contrast, Zarr v2 is synchronous and instead requires:

 ```python
xarray_open_kwargs = {
    "storage_options" : {
        "remote_options" : {
            "async": false,
            ...
        },
        ...
    },
   ...
}
 ```

If zarr2 or zarr3 is specified in the format field, the async flag will be set automatically. If you specify zarr as the format, you must set the async flag manually in the xarray_open_kwargs.

Aggregation Control Object

An aggregation control object defines neccessary information to use when aggregating multiple assets into a single xarray data set.

Element Type Description
variable_column_name string REQUIRED. Name of the attribute column in csv file that contains the variable name.
groupby_attrs array Column names (attributes) that define data sets that can be aggegrated.
aggregations [Aggregation Object] OPTIONAL. List of aggregations to apply to query results

Aggregation Object

An aggregation object describes types of operations done during the aggregation of multiple assets into a single xarray data set.

Element Type Description
type string REQUIRED. Type of aggregation operation to apply. Valid values include: join_new, join_existing, union
attribute_name string Name of attribute (column) across which to aggregate.
options object OPTIONAL. Aggregration settings that are passed as keywords arguments to xarray.concat() or xarray.merge(). For join_existing, it must contain the name of the existing dimension to use (for e.g.: something like {'dim': 'time'}).