Merge pull request #83 from stackhpc/mkdocs

docs: Add mkdocs documentation

Author: markgoddard

.github/workflows/publish-docs.yml

@@ -0,0 +1,17 @@
+---
+name: Publish documentation
+on:
+  push:
+    branches:
+      - main
+jobs:
+  docs-publish:
+    name: Publish documentation
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - run: pip install -r docs/requirements.txt
+      - run: mkdocs gh-deploy --force

.github/workflows/pull-request.yml

@@ -15,6 +15,18 @@ jobs:
       - name: Build
         run: make build
   docs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - name: Install mkdocs
+        run: pip install -r docs/requirements.txt
+      - run: mkdocs build --strict
+  rustdocs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout

CONTRIBUTING.md

@@ -9,289 +9,38 @@ The work is funded by the
 and is done in collaboration with the
 [University of Reading](http://www.reading.ac.uk/).
 
-Documentation is available on [docs.rs](https://docs.rs/reductionist/latest/reductionist/).
+Documentation for the Reductionist application is hosted on [GitHub](https://stackhpc.github.io/reductionist-rs).
+Documentation for the source code is available on [docs.rs](https://docs.rs/reductionist/latest/reductionist/).
 
 This is a performant implementation of the active storage server.
 The original Python functional prototype is available [here](https://github.com/stackhpc/s3-active-storage-prototype).
 
-## Concepts
+Note: The original S3 Active Storage project was renamed to Reductionist, to avoid confusion due to overuse of the term Active Storage.
 
-The Reductionist server supports the application of reductions to S3 objects that contain numeric binary data. These reductions are specified by making a HTTP post request to the active storage proxy service.
+## Features
 
-The Reductionist server does not attempt to infer the datatype - it must be told the datatype to use based on knowledge that the client already has about the S3 object.
+Reductionist provides the following features:
 
-For example, if the original object has the following URL:
+* HTTP(S) API with JSON request data
+* Access to data stored in S3-compatible storage
+* Basic numerical operations on multi-dimensional arrays (count, min, max, select, sum)
+* Perform calculations on a selection/slice of an array
+* Perform calculations allowing for missing data
+* Compressed data (GZip, Zlib)
+* Filtered data (byte shuffle)
+* Data with non-native byte order (endianness)
+* Server resource (CPU, memory, files) management
+* [Prometheus](https://prometheus.io/) metrics
+* Tracing with an option to send data to [Jaeger](https://www.jaegertracing.io/)
+* Ansible-based containerised deployment
 
-```
-http[s]://s3.example.org/my-bucket/path/to/object
-```
+## Related projects
 
-Then Reductionist server could be used by making post requests to specfic reducer endpoints:
-
-```
-http[s]://s3-proxy.example.org/v1/{reducer}/
-```
-
-with a JSON payload of the form:
-
-```
-{
-    // The URL for the S3 source
-    // - required
-    "source": "https://s3.example.com/,
-
-    // The name of the S3 bucket
-    // - required
-    "bucket": "my-bucket",
-
-    // The path to the object within the bucket
-    // - required
-    "object": "path/to/object",
-
-    // The data type to use when interpreting binary data
-    // - required
-    "dtype": "int32|int64|uint32|uint64|float32|float64",
-
-    // The byte order (endianness) of the data
-    // - optional, defaults to native byte order of Reductionist server
-    "byte_order": "big|little",
-
-    // The offset in bytes to use when reading data
-    // - optional, defaults to zero
-    "offset": 0,
-
-    // The number of bytes to read
-    // - optional, defaults to the size of the entire object
-    "size": 128,
-
-    // The shape of the data (i.e. the size of each dimension) 
-    // - optional, defaults to a simple 1D array
-    "shape": [20, 5],
-
-    // Indicates whether the data is in C order (row major)
-    // or Fortran order (column major, indicated by 'F')
-    // - optional, defaults to 'C'
-    "order": "C|F",
-
-    // An array of [start, end, stride] tuples indicating the data to be operated on
-    // (if given, you must supply one tuple per element of "shape")
-    // - optional, defaults to the whole array
-    "selection": [
-        [0, 19, 2],
-        [1, 3, 1]
-    ],
-
-    // Algorithm used to compress the data
-    // - optional, defaults to no compression
-    "compression": {"id": "gzip|zlib"},
-
-    // List of algorithms used to filter the data
-    // - optional, defaults to no filters
-    "filters": [{"id": "shuffle", "element_size": 4}],
-
-    // Missing data description
-    // - optional, defaults to no missing data
-    // - exactly one of the keys below should be specified
-    // - the values should match the data type (dtype)
-    "missing": {
-        "missing_value": 42,
-        "missing_values": [42, -42],
-        "valid_min": 42,
-        "valid_max": 42,
-        "valid_range": [-42, 42],
-    }
-}
-```
-
-The currently supported reducers are `max`, `min`, `sum`, `select` and `count`. All reducers return the result using the same datatype as specified in the request except for `count` which always returns the result as `int64`.
-
-The proxy returns the following headers to the HTTP response:
-
-* `x-activestorage-dtype`: The data type of the data in the response payload. One of `int32`, `int64`, `uint32`, `uint64`, `float32` or `float64`.
-* `x-activestorage-byte-order`: The byte order of the data in the response payload. Either `big` or `little`.
-* `x-activestrorage-shape`: A JSON-encoded list of numbers describing the shape of the data in the response payload. May be an empty list for a scalar result.
-* `x-activestorage-count`: The number of non-missing array elements operated on while performing the requested reduction. This header is useful, for example, to calculate the mean over multiple requests where the number of items operated on may differ between chunks.
-
-[//]: <> (TODO: No OpenAPI support yet).
-[//]: <> (For a running instance of the proxy server, the full OpenAPI specification is browsable as a web page at the `{proxy-address}/redoc/` endpoint or in raw JSON form at `{proxy-address}/openapi.json`.)
-
-## Caveats
-
-This is a very early-stage project, and as such supports limited functionality.
-
-In particular, the following are known limitations which we intend to address:
-
-  * Error handling and reporting is minimal
-  * No support for missing data
-  * No support for encrypted objects
-
-## Running
-
-There are various ways to run the Reductionist server.
-
-### Production deployment
-
-Reductionist provides an Ansible playbook to easily deploy it and supporting
-services to one or more hosts. See the [deployment
-README](deployment/README.md) for details.
-
-### Running in a container
-
-The simplest method is to run it in a container using a pre-built image:
-
-```sh
-docker run -it --detach --rm --net=host --name reductionist ghcr.io/stackhpc/reductionist-rs:latest
-```
-
-Images are published to [GitHub Container Registry](https://github.com/stackhpc/reductionist-rs/pkgs/container/reductionist-rs) when the project is released.
-The `latest` tag corresponds to the most recent release, or you can use a specific release e.g. `0.1.0`.
-
-This method does not require access to the source code.
-
-### Building a container image
-
-If you need to use unreleased changes, but still want to run in a container, it is possible to build an image.
-First, clone this repository:
-
-```sh
-git clone https://github.com/stackhpc/reductionist-rs.git
-cd reductionist-rs
-```
-
-```sh
-make build
-```
-
-The image will be tagged as `reductionist`.
-The image may be pushed to a registry, or deployed locally.
-
-```sh
-make run
-```
-
-## Build
-
-If you prefer not to run the Reductionist server in a container, it will be necessary to build a binary.
-Building locally may also be preferable during development to take advantage of incremental compilation.
-
-### Prerequisites
-
-This project is written in Rust, and as such requires a Rust toolchain to be installed in order to build it.
-The Minimum Supported Rust Version (MSRV) is 1.66.1, due to a dependency on the [AWS SDK](https://github.com/awslabs/aws-sdk-rust).
-It may be necessary to use [rustup](https://rustup.rs/) rather than the OS provided Rust toolchain to meet this requirement.
-See the [Rust book](https://doc.rust-lang.org/book/ch01-01-installation.html) for toolchain installation.
-
-### Build and run Reductionist
-
-First, clone this repository:
-
-```sh
-git clone https://github.com/stackhpc/reductionist-rs.git
-cd reductionist-rs
-```
-
-Next, use Cargo to build the package:
-
-```sh
-cargo build --release
-```
-
-The active storage server may be run using Cargo:
-
-```sh
-cargo run --release
-```
-
-Or installed to the system:
-
-```sh
-cargo install --path . --locked
-```
-
-Then run:
-
-```sh
-reductionist
-```
-
-## Testing
-
-For simple testing purposes Minio is a convenient object storage server.
-
-### Deploy Minio object storage
-
-Start a local [Minio](https://min.io/) server which serves the test data:
-
-```sh
-./scripts/minio-start
-```
-
-The Minio server will run in a detached container and may be stopped:
-
-```sh
-./scripts/minio-stop
-```
-
-Note that object data is not preserved when the container is stopped.
-
-### Upload some test data
-
-A script is provided to upload some test data to minio.
-In a separate terminal, set up the Python virtualenv then upload some sample data:
-
-```sh
-# Create a virtualenv
-python3 -m venv ./venv
-# Activate the virtualenv
-source ./venv/bin/activate
-# Install dependencies
-pip install scripts/requirements.txt
-# Upload some sample data to the running minio server
-python ./scripts/upload_sample_data.py
-```
-
-### Compliance test suite
-
-Proxy functionality can be tested using the [S3 active storage compliance suite](https://github.com/stackhpc/s3-active-storage-compliance-suite).
-
-### Making requests to active storage endpoints
-
-Request authentication is implemented using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication) with the username and password consisting of your S3 Access Key ID and Secret Access Key, respectively. These credentials are then used internally to authenticate with the upstream S3 source using [standard AWS authentication methods](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html)
-
-A basic Python client is provided in `scripts/client.py`.
-First install dependencies in a Python virtual environment:
-
-```sh
-# Create a virtualenv
-python3 -m venv ./venv
-# Activate the virtualenv
-source ./venv/bin/activate
-# Install dependencies
-pip install scripts/requirements.txt
-```
-
-Then use the client to make a request:
-```sh
-venv/bin/python ./scripts/client.py sum --server http://localhost:8080 --source http://localhost:9000 --username minioadmin --password minioadmin --bucket sample-data --object data-uint32.dat --dtype uint32
-```
-
----
-
-## Documentation
-
-The source code is documented using [rustdoc](https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html).
-Documentation is available on [docs.rs](https://docs.rs/reductionist/latest/reductionist/).
-It is also possible to build the documentation locally:
-
-```sh
-cargo doc --no-deps
-```
-
-The resulting documentation is available under `target/doc`, and may be viewed in a web browser using file:///path/to/reductionist/target/doc/reductionist/index.html.
+* [PyActiveStorage](https://github.com/valeriupredoi/PyActiveStorage) is a Python library which performs reductions on numerical data in data sources such as netCDF4. It has support for delegating computation to Reductionist when the data is stored in an S3-compatible object store.
 
 ## Contributing
 
-See [CONTRIBUTING.md](CONTRIBUTING.md) for information about contributing to Reductionist.
+See the [contributor guide](https://stackhpc.github.io/reductionist-rs/contributing.html) for information about contributing to Reductionist.
 
 ## License