Skip to content

Commit f9627f3

Browse files
helen-m-linhelen-m-lin
authored
Merge pull request #142 from AllenNeuralDynamics/release-1.2.1
Release 1.2.1
2 parents 8ede68c + e32a930 commit f9627f3

File tree

7 files changed

+203
-115
lines changed

7 files changed

+203
-115
lines changed

‎CODE_OF_CONDUCT.md

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
2+
# Contributor Covenant Code of Conduct
3+
4+
## Our Pledge
5+
6+
We as members, contributors, and leaders pledge to make participation in our
7+
community a harassment-free experience for everyone, regardless of age, body
8+
size, visible or invisible disability, ethnicity, sex characteristics, gender
9+
identity and expression, level of experience, education, socio-economic status,
10+
nationality, personal appearance, race, religion, or sexual identity
11+
and orientation.
12+
13+
We pledge to act and interact in ways that contribute to an open, welcoming,
14+
diverse, inclusive, and healthy community.
15+
16+
## Our Standards
17+
18+
Examples of behavior that contributes to a positive environment for our
19+
community include:
20+
21+
* Demonstrating empathy and kindness toward other people
22+
* Being respectful of differing opinions, viewpoints, and experiences
23+
* Giving and gracefully accepting constructive feedback
24+
* Accepting responsibility and apologizing to those affected by our mistakes,
25+
and learning from the experience
26+
* Focusing on what is best not just for us as individuals, but for the
27+
overall community
28+
29+
Examples of unacceptable behavior include:
30+
31+
* The use of sexualized language or imagery, and sexual attention or
32+
advances of any kind
33+
* Trolling, insulting or derogatory comments, and personal or political attacks
34+
* Public or private harassment
35+
* Publishing others' private information, such as a physical or email
36+
address, without their explicit permission
37+
* Other conduct which could reasonably be considered inappropriate in a
38+
professional setting
39+
40+
## Enforcement Responsibilities
41+
42+
Community leaders are responsible for clarifying and enforcing our standards of
43+
acceptable behavior and will take appropriate and fair corrective action in
44+
response to any behavior that they deem inappropriate, threatening, offensive,
45+
or harmful.
46+
47+
Community leaders have the right and responsibility to remove, edit, or reject
48+
comments, commits, code, wiki edits, issues, and other contributions that are
49+
not aligned to this Code of Conduct, and will communicate reasons for moderation
50+
decisions when appropriate.
51+
52+
## Scope
53+
54+
This Code of Conduct applies within all community spaces, and also applies when
55+
an individual is officially representing the community in public spaces.
56+
Examples of representing our community include using an official email address,
57+
posting via an official social media account, or acting as an appointed
58+
representative at an online or offline event.
59+
60+
## Enforcement
61+
62+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
63+
reported to the community leaders responsible for enforcement at
64+
[email protected].
65+
All complaints will be reviewed and investigated promptly and fairly.
66+
67+
All community leaders are obligated to respect the privacy and security of the
68+
reporter of any incident.
69+
70+
## Enforcement Guidelines
71+
72+
Community leaders will follow these Community Impact Guidelines in determining
73+
the consequences for any action they deem in violation of this Code of Conduct:
74+
75+
### 1. Correction
76+
77+
**Community Impact**: Use of inappropriate language or other behavior deemed
78+
unprofessional or unwelcome in the community.
79+
80+
**Consequence**: A private, written warning from community leaders, providing
81+
clarity around the nature of the violation and an explanation of why the
82+
behavior was inappropriate. A public apology may be requested.
83+
84+
### 2. Warning
85+
86+
**Community Impact**: A violation through a single incident or series
87+
of actions.
88+
89+
**Consequence**: A warning with consequences for continued behavior. No
90+
interaction with the people involved, including unsolicited interaction with
91+
those enforcing the Code of Conduct, for a specified period of time. This
92+
includes avoiding interactions in community spaces as well as external channels
93+
like social media. Violating these terms may lead to a temporary or
94+
permanent ban.
95+
96+
### 3. Temporary Ban
97+
98+
**Community Impact**: A serious violation of community standards, including
99+
sustained inappropriate behavior.
100+
101+
**Consequence**: A temporary ban from any sort of interaction or public
102+
communication with the community for a specified period of time. No public or
103+
private interaction with the people involved, including unsolicited interaction
104+
with those enforcing the Code of Conduct, is allowed during this period.
105+
Violating these terms may lead to a permanent ban.
106+
107+
### 4. Permanent Ban
108+
109+
**Community Impact**: Demonstrating a pattern of violation of community
110+
standards, including sustained inappropriate behavior, harassment of an
111+
individual, or aggression toward or disparagement of classes of individuals.
112+
113+
**Consequence**: A permanent ban from any sort of public interaction within
114+
the community.
115+
116+
## Attribution
117+
118+
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
119+
version 2.0, available at
120+
[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
121+
122+
Community Impact Guidelines were inspired by
123+
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
124+
125+
For answers to common questions about this code of conduct, see the FAQ at
126+
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available
127+
at [https://www.contributor-covenant.org/translations][translations].
128+
129+
[homepage]: https://www.contributor-covenant.org
130+
[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
131+
[Mozilla CoC]: https://github.com/mozilla/diversity
132+
[FAQ]: https://www.contributor-covenant.org/faq
133+
[translations]: https://www.contributor-covenant.org/translations

‎README.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,7 @@
66

77
API to interact with a few AIND databases. We have two primary databases:
88

9-
1. A document database (DocDB) to store
10-
unstructured json documents. The DocDB contains AIND metadata.
9+
1. A document database (DocDB) to store unstructured JSON documents. The DocDB contains AIND metadata.
1110
2. A relational database to store structured tables.
1211

1312
## Installation

‎docs/source/Contributing.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ For development,
3232
off of ``main``
3333

3434
Consult the `Branches and Pull Requests <#branches-and-pull-requests>`__
35-
and `Release Cycles <#release-cycles>`__ for more details.
35+
and `Release Cycles <#release-cycles>`__ sections for more details.
3636

3737
From the root directory, run:
3838

@@ -164,7 +164,7 @@ Hotfixes
164164
~~~~~~~~
165165

166166
- A ``hotfix`` branch is created off of ``main``
167-
- A Pull Request into is ``main`` is opened, reviewed, and merged into
167+
- A Pull Request into ``main`` is opened, reviewed, and merged into
168168
``main``
169169
- A new ``tag`` with a patch bump is created, and a new ``release`` is
170170
deployed

‎docs/source/ExamplesDocDBDirectConnection.rst

Lines changed: 11 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
1-
Examples - DocDB Direct Connection
2-
==================================
1+
Examples - DocDB Direct Connection (SSH)
2+
========================================
33

4-
This page provides examples for interact with the Document Database (DocDB)
5-
using the provided Python client.
4+
This page provides examples to interact with the Document Database (DocDB)
5+
using the provided Python client and SSH tunneling.
66

77
It is assumed that the required credentials are set in environment.
88
Please refer to the User Guide for more details.
99

10+
.. note::
11+
12+
It is recommended to use the REST API client instead of the SSH client for ease of use.
13+
1014

1115
Querying Metadata
1216
~~~~~~~~~~~~~~~~~~~~~~
@@ -43,7 +47,7 @@ Filter Example 1: Get records with a certain subject_id
4347
4448
4549
With projection (recommended):
46-
50+
4751
.. code:: python
4852
4953
with DocumentDbSSHClient(credentials=credentials) as doc_db_client:
@@ -115,94 +119,8 @@ Aggregation Example 1: Get all subjects per breeding group
115119
doc_db_client.collection.aggregate(pipeline=agg_pipeline)
116120
)
117121
print(f"Total breeding groups: {len(result)}")
118-
print(f"First 3 breeding groups and corresponding subjects:")
122+
print("First 3 breeding groups and corresponding subjects:")
119123
print(json.dumps(result[:3], indent=3))
120124
121125
For more info about aggregations, please see MongoDB documentation:
122-
https://www.mongodb.com/docs/manual/aggregation/
123-
124-
125-
Updating Metadata
126-
~~~~~~~~~~~~~~~~~~~~~~
127-
128-
Below is an example of how to update records in DocDB using ``DocumentDbSSHClient``.
129-
130-
.. code:: python
131-
132-
import logging
133-
134-
from aind_data_access_api.document_db_ssh import (
135-
DocumentDbSSHClient,
136-
DocumentDbSSHCredentials,
137-
)
138-
139-
logging.basicConfig(level="INFO")
140-
141-
def _process_docdb_record(record: dict, doc_db_client: DocumentDbSSHClient, dryrun: bool) -> None:
142-
"""
143-
Process record. This example updates the data_description.name field
144-
if it does not match the record.name field.
145-
146-
Parameters
147-
----------
148-
record : dict
149-
150-
"""
151-
_id = record.get("_id")
152-
name = record.get("name")
153-
location = record.get("location")
154-
if _id:
155-
if record.get("data_description") and record["data_description"].get("name") != name:
156-
# update specific fields(s) only
157-
new_fields = {
158-
"data_description.name": name
159-
}
160-
update_docdb_record_partial(record_id=_id, new_fields=new_fields, doc_db_client=doc_db_client, dryrun=dryrun)
161-
# else:
162-
# logging.info(f"Record for {location} does not need to be updated.")
163-
else:
164-
logging.warning(f"Record for {location} does not have an _id field! Skipping.")
165-
166-
167-
def update_docdb_record_partial(record_id: str, new_fields: dict, doc_db_client: DocumentDbSSHClient, dryrun: bool) -> None:
168-
"""
169-
Update record in docdb by updating specific fields only.
170-
Parameters
171-
----------
172-
record_id : str
173-
The _id of the record to update.
174-
new_fields : dict
175-
New fields to update. E.g. {"data_description.name": "new_name"}
176-
177-
"""
178-
if dryrun:
179-
logging.info(f"(dryrun) doc_db_client.collection.update_one (partial): {record_id}")
180-
else:
181-
logging.info(f"doc_db_client.collection.update_one (partial): {record_id}")
182-
response = doc_db_client.collection.update_one(
183-
{"_id": record_id},
184-
{"$set": new_fields},
185-
upsert=False,
186-
)
187-
logging.info(response.raw_result)
188-
189-
190-
if __name__ == "__main__":
191-
credentials = DocumentDbSSHCredentials() # credentials in environment
192-
dryrun = True
193-
filter = {"location": {"$regex": ".*s3://aind-open-data.*"}}
194-
projection = None
195-
196-
with DocumentDbSSHClient(credentials=credentials) as doc_db_client:
197-
db_name = doc_db_client.database_name
198-
col_name = doc_db_client.collection_name
199-
# count = doc_db_client.collection.count_documents(filter)
200-
# logging.info(f"{db_name}.{col_name}: Found {count} records with {filter}")
201-
202-
logging.info(f"{db_name}.{col_name}: Starting to scan for {filter}.")
203-
records = doc_db_client.collection.find(
204-
filter=filter,
205-
)
206-
for record in records:
207-
_process_docdb_record(record=record, doc_db_client=doc_db_client, dryrun=dryrun)
208-
logging.info(f"{db_name}.{col_name}:Finished scanning through DocDb.")
126+
https://www.mongodb.com/docs/manual/aggregation/

‎docs/source/ExamplesDocDBRestApi.rst

Lines changed: 40 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Examples - DocDB REST API
22
==================================
33

4-
This page provides examples for interact with the Document Database (DocDB)
4+
This page provides examples to interact with the Document Database (DocDB)
55
REST API using the provided Python client.
66

77

@@ -46,7 +46,7 @@ Filter Example 1: Get records with a certain subject_id
4646
4747
4848
With projection (recommended):
49-
49+
5050
.. code:: python
5151
5252
filter = {"subject.subject_id": "731015"}
@@ -110,13 +110,13 @@ Aggregation Example 1: Get all subjects per breeding group
110110
"subject_ids": {"$addToSet": "$subject.subject_id"},
111111
"count": {"$sum": 1},
112112
}
113-
}
113+
}
114114
]
115115
result = docdb_api_client.aggregate_docdb_records(
116116
pipeline=agg_pipeline
117117
)
118118
print(f"Total breeding groups: {len(result)}")
119-
print(f"First 3 breeding groups and corresponding subjects:")
119+
print("First 3 breeding groups and corresponding subjects:")
120120
print(json.dumps(result[:3], indent=3))
121121
122122
For more info about aggregations, please see MongoDB documentation:
@@ -125,7 +125,7 @@ https://www.mongodb.com/docs/manual/aggregation/
125125
Advanced Example: Custom Session Object
126126
-------------------------------------------
127127

128-
It's possible to attach a custom Session to retry certain requests errors
128+
It's possible to attach a custom Session to retry certain requests errors:
129129

130130
.. code:: python
131131
@@ -157,6 +157,31 @@ It's possible to attach a custom Session to retry certain requests errors
157157
) as docdb_api_client:
158158
records = docdb_api_client.retrieve_docdb_records(limit=10)
159159
160+
Utility Methods
161+
---------------
162+
163+
A few utility methods are provided in the :mod:`aind_data_access_api.utils` module
164+
to help with interacting with the DocDB API.
165+
166+
For example, to fetch records that match any value in a list of subject IDs:
167+
168+
.. code:: python
169+
170+
from aind_data_access_api.utils import fetch_records_by_filter_list
171+
172+
records = fetch_records_by_filter_list(
173+
docdb_api_client=docdb_api_client,
174+
filter_key="subject.subject_id",
175+
filter_values=["731015", "741137", "789012"],
176+
projection={
177+
"name": 1,
178+
"location": 1,
179+
"subject.subject_id": 1,
180+
"data_description.project_name": 1,
181+
},
182+
)
183+
print(f"Found {len(records)} records. First 3 records:")
184+
print(json.dumps(records[:3], indent=3))
160185
161186
162187
Updating Metadata
@@ -166,7 +191,10 @@ Updating Metadata
166191
2. **Query DocDB**: Filter for the records you want to update.
167192
3. **Update DocDB**: Use ``upsert_one_docdb_record`` or ``upsert_list_of_docdb_records`` to update the records.
168193

169-
Please note that records are read and written as dictionaries from DocDB (not Pydantic models).
194+
.. note::
195+
196+
Records must be read and written as dictionaries from DocDB (not Pydantic models).
197+
170198
For example, to update the "instrument" and "session" metadata of a record in DocDB:
171199

172200
.. code:: python
@@ -214,7 +242,10 @@ You can also make updates to individual nested fields:
214242
)
215243
print(response.json())
216244
217-
Please note that while DocumentDB supports fieldnames with special characters ("$" and "."), they are not recommended.
218-
There may be issues querying or updating these fields.
245+
.. note::
246+
247+
While DocumentDB supports fieldnames with special characters ("$" and "."), they are not recommended.
248+
There may be issues querying or updating these fields.
219249

220-
It is recommended to avoid these special chars in dictionary keys, e.g. ``{"abc.py": "data"}`` can be written as ``{"filename": "abc.py", "some_file_property": "data"}`` instead.
250+
It is recommended to avoid these special chars in dictionary keys. E.g. ``{"abc.py": "data"}`` can be
251+
written as ``{"filename": "abc.py", "some_file_property": "data"}`` instead.

0 commit comments

Comments
 (0)