docs: update docs and examples (#141)

helen-m-lin · web-flow · commit 85f36e807dfc · 2025-04-23T09:30:37.000-07:00
* docs: update README

* docs: update User and Contributor Guides

* docs: update SSH examples

* docs: update rest api examples
diff --git a/README.md b/README.md
@@ -6,8 +6,7 @@
 
 API to interact with a few AIND databases. We have two primary databases:
 
-1. A document database (DocDB) to store
-   unstructured json documents. The DocDB contains AIND metadata.
+1. A document database (DocDB) to store unstructured JSON documents. The DocDB contains AIND metadata.
 2. A relational database to store structured tables.
 
 ## Installation
diff --git a/docs/source/Contributing.rst b/docs/source/Contributing.rst
@@ -32,7 +32,7 @@ For development,
    off of ``main``
 
 Consult the `Branches and Pull Requests <#branches-and-pull-requests>`__
-and `Release Cycles <#release-cycles>`__ for more details.
+and `Release Cycles <#release-cycles>`__ sections for more details.
 
 From the root directory, run:
 
@@ -164,7 +164,7 @@ Hotfixes
 ~~~~~~~~
 
 -  A ``hotfix`` branch is created off of ``main``
--  A Pull Request into is ``main`` is opened, reviewed, and merged into
+-  A Pull Request into ``main`` is opened, reviewed, and merged into
    ``main``
 -  A new ``tag`` with a patch bump is created, and a new ``release`` is
    deployed
diff --git a/docs/source/ExamplesDocDBDirectConnection.rst b/docs/source/ExamplesDocDBDirectConnection.rst
@@ -1,12 +1,16 @@
-Examples - DocDB Direct Connection
-==================================
+Examples - DocDB Direct Connection (SSH)
+========================================
 
-This page provides examples for interact with the Document Database (DocDB)
-using the provided Python client.
+This page provides examples to interact with the Document Database (DocDB)
+using the provided Python client and SSH tunneling.
 
 It is assumed that the required credentials are set in environment.
 Please refer to the User Guide for more details.
 
+.. note::
+
+    It is recommended to use the REST API client instead of the SSH client for ease of use.
+
 
 Querying Metadata
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -43,7 +47,7 @@ Filter Example 1: Get records with a certain subject_id
 
 
 With projection (recommended):
-      
+
 .. code:: python
 
   with DocumentDbSSHClient(credentials=credentials) as doc_db_client:
@@ -115,94 +119,8 @@ Aggregation Example 1: Get all subjects per breeding group
           doc_db_client.collection.aggregate(pipeline=agg_pipeline)
       )
       print(f"Total breeding groups: {len(result)}")
-      print(f"First 3 breeding groups and corresponding subjects:")
+      print("First 3 breeding groups and corresponding subjects:")
       print(json.dumps(result[:3], indent=3))
 
 For more info about aggregations, please see MongoDB documentation:
-https://www.mongodb.com/docs/manual/aggregation/
-
-
-Updating Metadata
-~~~~~~~~~~~~~~~~~~~~~~
-
-Below is an example of how to update records in DocDB using ``DocumentDbSSHClient``.
-
-.. code:: python
-
-  import logging
-
-  from aind_data_access_api.document_db_ssh import (
-      DocumentDbSSHClient,
-      DocumentDbSSHCredentials,
-  )
-
-  logging.basicConfig(level="INFO")
-
-  def _process_docdb_record(record: dict, doc_db_client: DocumentDbSSHClient, dryrun: bool) -> None:
-      """
-      Process record. This example updates the data_description.name field
-      if it does not match the record.name field.
-
-      Parameters
-      ----------
-      record : dict
-
-      """
-      _id = record.get("_id")
-      name = record.get("name")
-      location = record.get("location")
-      if _id:
-          if record.get("data_description") and record["data_description"].get("name") != name:
-              # update specific fields(s) only
-              new_fields = {
-                  "data_description.name": name
-              }
-              update_docdb_record_partial(record_id=_id, new_fields=new_fields, doc_db_client=doc_db_client, dryrun=dryrun)
-          # else:
-          #     logging.info(f"Record for {location} does not need to be updated.")
-      else:
-          logging.warning(f"Record for {location} does not have an _id field! Skipping.")
-
-
-  def update_docdb_record_partial(record_id: str, new_fields: dict, doc_db_client: DocumentDbSSHClient, dryrun: bool) -> None:
-      """
-      Update record in docdb by updating specific fields only.
-      Parameters
-      ----------
-      record_id : str
-          The _id of the record to update.
-      new_fields : dict
-          New fields to update. E.g. {"data_description.name": "new_name"}
-
-      """
-      if dryrun:
-          logging.info(f"(dryrun) doc_db_client.collection.update_one (partial): {record_id}")
-      else:
-          logging.info(f"doc_db_client.collection.update_one (partial): {record_id}")
-          response = doc_db_client.collection.update_one(
-              {"_id": record_id},
-              {"$set": new_fields},
-              upsert=False,
-          )
-          logging.info(response.raw_result)
-            
-          
-  if __name__ == "__main__":
-      credentials = DocumentDbSSHCredentials()    # credentials in environment
-      dryrun = True
-      filter = {"location": {"$regex": ".*s3://aind-open-data.*"}}
-      projection = None
-      
-      with DocumentDbSSHClient(credentials=credentials) as doc_db_client:
-          db_name = doc_db_client.database_name
-          col_name = doc_db_client.collection_name
-          # count = doc_db_client.collection.count_documents(filter)
-          # logging.info(f"{db_name}.{col_name}: Found {count} records with {filter}")
-
-          logging.info(f"{db_name}.{col_name}: Starting to scan for {filter}.")
-          records = doc_db_client.collection.find(
-              filter=filter,
-          )
-          for record in records:
-              _process_docdb_record(record=record, doc_db_client=doc_db_client, dryrun=dryrun)
-          logging.info(f"{db_name}.{col_name}:Finished scanning through DocDb.")
+https://www.mongodb.com/docs/manual/aggregation/
diff --git a/docs/source/ExamplesDocDBRestApi.rst b/docs/source/ExamplesDocDBRestApi.rst
@@ -1,7 +1,7 @@
 Examples - DocDB REST API
 ==================================
 
-This page provides examples for interact with the Document Database (DocDB)
+This page provides examples to interact with the Document Database (DocDB)
 REST API using the provided Python client.
 
 
@@ -46,7 +46,7 @@ Filter Example 1: Get records with a certain subject_id
 
 
 With projection (recommended):
-      
+
 .. code:: python
 
   filter = {"subject.subject_id": "731015"}
@@ -110,13 +110,13 @@ Aggregation Example 1: Get all subjects per breeding group
               "subject_ids": {"$addToSet": "$subject.subject_id"},
               "count": {"$sum": 1},
           }
-    }
+      }
   ]
   result = docdb_api_client.aggregate_docdb_records(
       pipeline=agg_pipeline
   )
   print(f"Total breeding groups: {len(result)}")
-  print(f"First 3 breeding groups and corresponding subjects:")
+  print("First 3 breeding groups and corresponding subjects:")
   print(json.dumps(result[:3], indent=3))
 
 For more info about aggregations, please see MongoDB documentation:
@@ -125,7 +125,7 @@ https://www.mongodb.com/docs/manual/aggregation/
 Advanced Example: Custom Session Object
 -------------------------------------------
 
-It's possible to attach a custom Session to retry certain requests errors
+It's possible to attach a custom Session to retry certain requests errors:
 
 .. code:: python
 
@@ -157,6 +157,31 @@ It's possible to attach a custom Session to retry certain requests errors
     ) as docdb_api_client:
         records = docdb_api_client.retrieve_docdb_records(limit=10)
 
+Utility Methods
+---------------
+
+A few utility methods are provided in the :mod:`aind_data_access_api.utils` module
+to help with interacting with the DocDB API.
+
+For example, to fetch records that match any value in a list of subject IDs:
+
+.. code:: python
+
+    from aind_data_access_api.utils import fetch_records_by_filter_list
+
+    records = fetch_records_by_filter_list(
+        docdb_api_client=docdb_api_client,
+        filter_key="subject.subject_id",
+        filter_values=["731015", "741137", "789012"],
+        projection={
+            "name": 1,
+            "location": 1,
+            "subject.subject_id": 1,
+            "data_description.project_name": 1,
+        },
+    )
+    print(f"Found {len(records)} records. First 3 records:")
+    print(json.dumps(records[:3], indent=3))
 
 
 Updating Metadata
@@ -166,7 +191,10 @@ Updating Metadata
 2. **Query DocDB**: Filter for the records you want to update.
 3. **Update DocDB**: Use ``upsert_one_docdb_record`` or ``upsert_list_of_docdb_records`` to update the records.
 
-Please note that records are read and written as dictionaries from DocDB (not Pydantic models).
+.. note::
+
+    Records must be read and written as dictionaries from DocDB (not Pydantic models).
+
 For example, to update the "instrument" and "session" metadata of a record in DocDB:
 
 .. code:: python
@@ -214,7 +242,10 @@ You can also make updates to individual nested fields:
   )
   print(response.json())
 
-Please note that while DocumentDB supports fieldnames with special characters ("$" and "."), they are not recommended.
-There may be issues querying or updating these fields.
+.. note::
+
+    While DocumentDB supports fieldnames with special characters ("$" and "."), they are not recommended.
+    There may be issues querying or updating these fields.
 
-It is recommended to avoid these special chars in dictionary keys, e.g. ``{"abc.py": "data"}`` can be written as ``{"filename": "abc.py", "some_file_property": "data"}`` instead.
+    It is recommended to avoid these special chars in dictionary keys. E.g. ``{"abc.py": "data"}`` can be
+    written as ``{"filename": "abc.py", "some_file_property": "data"}`` instead.
diff --git a/docs/source/UserGuide.rst b/docs/source/UserGuide.rst
@@ -8,7 +8,7 @@ with AIND databases.
 We have two primary databases:
 
 1. A `document database (DocDB) <#document-database-docdb>`__ to store
-   unstructured json documents. The DocDB contains AIND metadata.
+   unstructured JSON documents. The DocDB contains AIND metadata.
 2. A `relational database <#rds-tables>`__ to store structured tables.
 
 Document Database (DocDB)
@@ -30,7 +30,7 @@ The DocDB can be accessed through a public read-only REST API or
 through a direct connection using SSH. For a direct connection,
 it is assumed you have the appropriate credentials.
 
-REST API (Read-Only)
+REST API
 ~~~~~~~~~~~~~~~~~~~~~~
 
 1. A GET request to ``https://api.allenneuraldynamics.org/v1/metadata_index/data_assets``
@@ -47,7 +47,7 @@ REST API (Read-Only)
    response = requests.get(URL, params={"filter": json.dumps(filter), "limit": limit})
    print(response.json())
 
-2. Alternatively, we provide a Python client:
+2. **We provide a Python client (recommended):**
 
 .. code:: python
 
@@ -78,7 +78,7 @@ with our document database.
 
 To connect:
 
-1. If provided a temporary SSH password, please first run ``ssh {ssh-username}@{ssh_host}``
+1. If provided a temporary SSH password, please first run ``ssh {ssh_username}@{ssh_host}``
    and set a new password.
 2. Download the full version of `MongoDB Compass <https://www.mongodb.com/try/download/compass>`__.
 3. When connecting, click “Advanced Connection Options” and use the configurations below.
@@ -107,7 +107,7 @@ To connect:
      - SSL/TLS Connection
      - OFF
    * - Proxy/SSH
-     - SSH Tunnel/ Proxy Method	
+     - SSH Tunnel/ Proxy Method
      - SSH with Password
    * -
      - SSH Hostname
@@ -119,10 +119,16 @@ To connect:
      - SSH Username
      - ``ssh_username``
    * -
-     - SSH Username
+     - SSH Password
      - ``ssh_password``
-   
-4. You should be able to see the home page with the ``metadata-index`` database.
+   * - (Optional) Advanced
+     - Read Preference
+     - Secondary Preferred
+   * - 
+     - Replica Set Name
+     - rs0
+
+4. You should be able to see the home page with the ``metadata_index`` database.
    It should have 1 single collection called ``data_assets``.
 5. If provided with a temporary DocDB password, please change it using the embedded
    mongo shell in Compass, and then reconnect.
@@ -180,6 +186,7 @@ To use the client:
 
 RDS Tables
 ------------------
+
 We have some convenience methods to interact with our Relational Database. You can create a client by 
 explicitly setting credentials, or downloading from AWS Secrets Manager.
 
