[RFC] Hierarchical Collections

[m-mohr]

The following is a proposal, how STAC APIs could implement a grouping for Collections as the flat list of Collections is considered not very user-friendly by many.

• Proposal
  • Example
• Related Extensions and Issues
  • Children
  • Browseable
  • Catalogs Endpoint
  • Related issues
• Implementations

Proposal

STAC API - Core, including its Landing Page, can have links with relation type child and item that point to other STAC Catalogs, Collections, and Items. This can be used to create arbitrarily complex hierarchies similar to static STAC catalogs.

Thus arbitrarily complex hierarchies can be served via STAC APIs. The traversal can lead to a lot of requests, as such it is recommended to implement STAC API - Children for more efficient data retrieval.

Generally, the STAC APIs implementing Core can flexibly choose how the hierchical STAC structure looks like, it's pretty much like a static STAC catalog, just powered by an API server in the background. Nevertheless, many STAC APIs implement STAC API - Collections / Features. It offers rich functionalities to provide and filter Collections and Items through a flat interface the hierarchy is Landing Page -> Collections -> Items. With the growing number of Collections, providers want to group/structure their Collections to avoid overly long lists of Collections.

A combination of STAC API - Core, Children, and Collections / Features can support such as usecase. The general idea is to add the child link relation to the landing page, to create a structure that links to the existing STAC Collections provided via STAC API - Collections while still keeping the abilities of STAC API - Collections and Collection Search.

An open point up for discussion is, how these structures could be created by be the API implementation.

• One way of doing it would be for example through the Themes extension or keywords in Collections. The keywords / concepts could imply a hierarchy.
• Another way would be to create a structure in a file and then simply use Collection Search to find collections based on a specific criteria that is defined per leaf in the structure.
• Generally, any way that makes sense for API implementations can be chosen.

Example

Landing Page (STAC API - Core)
+ Collections (rel=data, STAC API - Collections)
  + Sentinel-1 (Collection)
    + Items (rel=items, STAC API - Features)
  + Sentinel-2 (Collection)
    + Items (rel=items)
  + Landsat (Collection)
    + Items (rel=items)
  + Drone Flight 1 (Collection)
    + Items (rel=items)
  + Drone Flight 2 (Collection)
    + Items (rel=items)
  + ...
+ Providers (rel=child)
  + ESA (rel=child)
    + Sentinel-1 (Collection, rel=child)
      + Items (rel=items)
    + Sentinel-2 (Collection, rel=child)
      + Items (rel=items)
  + NASA (rel=child)
    + Landsat (Collection, rel=child)
      + Items (rel=items)
  + My Drone Company (rel=child)
    + Drone Flight 1 (Collection, rel=child)
      + Items (rel=items)
    + Drone Flight 2 (Collection, rel=child)
      + Items (rel=items)
+ Flight Type
  + Satellite (Catalog, rel=child)
    + Sentinel-1 (Collection, rel=child)
      + Items (rel=items, STAC API - Features)
    + Sentinel-2 (Collection, rel=child)
      + Items (rel=items)
    + Landsat (Collection, rel=child)
      + Items (rel=items)
  + Airborne (Catalog, rel=child)
    + Drone Flight 1 (Collection, rel=child)
      + Items (rel=items)
    + Drone Flight 2 (Collection, rel=child)
      + Items (rel=items)
+ ... (any further groupings could be provided)

This provides a nice way to structure collections in various ways (e.g. SKOSMOS), multiple paths can lead to the same Collection.
Collection Search and a hierarchical way to browse through the magnitute of collections can lead to a much better UX than the plain collection lists.

Related Extensions and Issues

Children

The Landing Page, and all subsequent catalogs (Providers, ESA, NASA, My Drone Company, Flight Type, Satellite, etc.) could all implement the proposed update to STAC API - Children and provide a link with relation type children to reduce the number of API requests significantly.

Browseable

Please note that item links from Collections in this case can be found via the items relation type, not via the item relation type. As such it does not fullfill the requirements of STAC API - Browseable.
This is due to the fact the STAC API - Browsable requires item relation types to be present, which is not the case here. Instead items is provided, to keep the implementation overhead small and allow search on the /collections/{collectionId}/items endpoint.

Catalogs Endpoint

The extension Catalogs endpoint is somewhat similar, but catered more towards serviing multiple APIs from the same API instance. it only allows a single level of depth, but in principle the two extensions can be used together. It might not integrate as well with existing implementations as it defines new endpoints.

Related issues

• OGC API - Common: Hierarchical Collections
• STAC API Spec: Priority: child links vs. collections
• STAC API Spec: Create a new link relation like "parent-catalog" or "api-parent"
• SFEOS: Where is the 'catalog'?

Implementations

I'd assume that many implementations can already handle this if they are aware of static STAC and STAC APIs as it resuses the link relation types of existing specifications.

For instance, STAC Browser would support such a usecase out of the box. By default it would show both the list of Collections and the newly added child links combined, but setting the config option apiCatalogPriority to childs would hide the list of collections and only show the more user-friendly STAC hierarchy that is defined through child links. It still keeps the advantages of Collection Search for example.

[jonhealy1]

@m-mohr I think that there is a fundamental system design challenge that this hierarchical proposal introduces: The complexity of "Write" operations in a Graph.

The hierarchy described in this RFC makes perfect sense in a Read-Only context (like a Static Catalog on S3). In that world, the "Truth" is the file structure, and links are generated offline or in batch processes.

However, a STAC API is often a Read-Write system. If we encourage complex hierarchies (e.g., Provider -> Project -> Collection), we must consider how a client builds and maintains this structure via HTTP.

Relying on "Manual Link Maintenance" (forcing the client to PUT updates to parent and child documents to stitch the graph together) poses significant architectural risks:

1. Lack of Atomicity (The "Broken Link" Risk)

In a standard REST API, requests are independent.

• Scenario: A user wants to move Collection C from Catalog A to Catalog B.
• Required Client Actions:
  1. PUT /catalogs/A (Remove child link).
  2. PUT /catalogs/B (Add child link).
  3. PUT /collections/C (Update parent link).
• The Risk: If the network fails after Step 2, the graph is corrupted. Collection C might be linked from B, but still think it belongs to A.
• The Fix: The API needs a transactional endpoint to handle the move atomically on the server side.

2. The Orphan Problem (Referential Integrity)

• Scenario: A user sends DELETE /catalogs/A.
• The Static Mindset: "Just delete the file."
• The API Reality: If Catalog A had children, those children are now Orphans. They exist in the database, but are unreachable via navigation. They become "Dark Data."
• The Requirement: The API must enforce Referential Integrity. It should either:
  • Reject the delete (409 Conflict: Catalog is not empty).
  • Cascade the delete (delete all children).
  • Logic like this belongs in the Server/Database layer, not the Client layer.

3. Concurrency & Race Conditions

• Scenario: User X adds a child to Catalog A. User Y simultaneously removes a child from Catalog A.
• The Risk: Both users GET the catalog, modify the links array locally, and PUT it back. The last write wins, and one of the changes is silently lost.
• The Requirement: To safely manage a mutable graph, we need atomic operations (e.g., POST /catalogs/{id}/children), rather than replacing the full document state via PUT.

4. Scale & Refactoring

• Scenario: We want to introduce a new hierarchy layer (e.g., grouping 5,000 existing Collections under a new "Theme" catalog).
• The Manual Way: The client must script 5,000 HTTP requests to update every single Collection. This is slow, error-prone, and floods the network.
• The Server Way: A server-side operation (like an Elasticsearch update_by_query) can perform this re-parenting in milliseconds.

Conclusion

While describing the hierarchy via child links works perfectly for Discovery, relying on manual link manipulation for Management is dangerous.

If we want to standardize hierarchical STAC APIs, we implicitly need to standardize the Transactional Behaviors (Create, Move, Cascade Delete) required to maintain that hierarchy safely. Without that, we risk defining a spec that works great for readers but is brittle for writers.

[m-mohr]

The title of your RFC is hierarchical collections, but collections, in my mind, don't naturally contain sub-collections. Collections are like specialized Catalogs that group together like Items. Catalogs can be hierarchical I think, but not Collections.

I'm happy to change the title/name to something more appropiate if a proposal is made.

[jonhealy1]

Based on the feedback you have given - I have no doubt that you strongly oppose our extensions. You have been highly critical - that's ok - I'm not saying you shouldn't be - I just would not spend time creating another extension proposal right now.

[jonhealy1]

Change it to hierarchical catalogs maybe :). People have been asking for /catalogs for years.

[jonhealy1]

I think if I force our Users to use /catalogs by making the addition not optional or by making default=True, then people can complain, but as an option, it's introduced as something that Users can try out if they want.

[m-mohr]

I just would not spend time creating another extension proposal right now.

That's fine, as I said, it can evolve whenever there's a need for it. It seemed you cared for transactional endpoints, so the obvious proposal is to write it up. I'll not do it at this point, I have no usecase for it yet.

Change it to hierarchical catalogs maybe :)

We structure Collections though, at the end you find Collections (and eventually Items). The question was always: How can we group our long list of collections and this is a proposed answer for it. Maybe "Cataloging API Collections" is better? Not sure...

[jonhealy1]

[m-mohr]

This is a proposal for now. RFCs are published on stac-spec as it's the (only) discussion forum for all STAC specifications.

Whether this will be a stac-api-spec part or an extension is to be decided.

The STAC API spec also ignores lifecycle questions and leaves it to extensions (i.e. Transactions and Collection Transactions), so I'm not sure what your argument is here?

Collection Transactions did not exist for a long time and were added by me long after stac-api-spec existed, just because I felt it was missing and without anyone specifically requesting it. So stac-api-specl worked without it for a significant time. So I don't see that having transactional APIs for this proposal is a hard requirement either. You can implement this without transactional API, if you want. But a transactional API can be added whenever needed in addition by whoever has the need for it.

[jonhealy1]

This just didn't seem right in the stac-spec, but I guess like you said, it's the only discussion forum. I'm just trying to learn how things work still.

[jonhealy1]

I wanted to link some of the things I have been working on related to this RFC:

• catalogs-endpoint stac api extension
• skos-registry stac api extension
• PR to add the Children extension to our experimental Catalogs-endpoint implementation
• PR to add type query parameter to the Children api extension
• Issue to ask that /children be extended to any catalog or collection