Metadata front matter for apps (#2372)

* Initialize Software catalog

* Serve catalog.json

Plenty of refactoring/reorganization

* Add software catalog appendix

* Allow for getting site_url from environment

Default to production url

* Remove extra whitespace

* Refactoring

* Software catalog, phase 1

* Ignore file for revision-date plugin

* Ignore commit for revision date plugin

* Remove leftover scripts

* Temporary amendment to FAQ

* Fixes for catalog-hook

* Refactor type hints for older Python version

---------

Co-authored-by: Rasmus Kronberg <[email protected]>

Author: joonas-somero

.git-revision-date-ignore-revs

@@ -0,0 +1,3 @@
+# Add metadata front matter
+d3b1642782b24ba19b6957d6e49ef67899ac132a
+d7aa3afdab66ff2e74dae96d5181676cb9aa93cb

.gitignore

@@ -6,3 +6,9 @@ site/
 .DS_Store
 .idea/
 __pycache__/
+
+# These are generated by catalog-hook and only
+# written to the filesystem when MKDOCS_ENV=test:
+docs/apps/index.md
+docs/apps/by_discipline.md
+docs/apps/by_system.md

.travis.yml

@@ -9,6 +9,7 @@ before_install:
 env:
   global:
     - ENABLED_GIT_REVISION_DATE=false
+    - MKDOCS_ENV=test
 script:
 
   # Checks all files for commonly misspelled English words with client9's misspell

Dockerfile

@@ -32,8 +32,6 @@ RUN if [ ! -d ".git" ]; then \
     git init && \
     git switch --force $repo_branch; \
     fi && \
-    bash scripts/generate_alpha.sh && \
-    bash scripts/generate_by_system.sh && \
     bash scripts/generate_new.sh && \
     bash scripts/generate_glossary.sh && \
     mkdocs build -d /usr/share/nginx/html

FAQ.md

@@ -355,13 +355,17 @@ Documentation for _Material for MkDocs_ has a [search feature](https://squidfunk
 
 ## How do I add a license tag to an application page?
 
-The license tag is added inside a YAML front matter. The first lines in the Markdown file should be
-the front matter. Please note the enclosing dashes. A template for the front matter is
+The license tag is added inside the YAML front matter. Temporarily, the license type should be placed as a
+list item under `tags:` _and_ as a string in `license_type:`:
 
 ```yaml
 ---
 tags:
   - <license>
+catalog:
+  # ...
+  license_type: <license>
+  # ...
 ---
 ```
 
@@ -372,36 +376,22 @@ The application will then be included on the Applications by license page automa
 
 ## How do I tag an application as available under a web interface?
 
-As with [adding a license tag](#how-do-I-add-a-license-tag-to-an-application-page), an application
-can be tagged as available in a particular web interface. The application will then be listed under
-that web interface on the _Applications by availability_ page. The tags for web interfaces that an
-application is available on are added under the `system` key in the front matter.
+In the YAML front matter. The following would, for example, tag the application as available on
+Puhti, Mahti _and_ Puhti web interface.
 
 ```yaml
 ---
-system:
-  - www-<system1>
-  - www-<system2>
+catalog:
+  # ...
+  available_on:
+    - Puhti
+    - Mahti
+    - web_interfaces:
+        - Puhti
+  # ...
 ---
 ```
 
-where `<system1>` or `<system2>` is one of the systems where a web interface is available, for
-example (prefixed with `www-`) `mahti` or `puhti`.
-
-As a temporary workaround, to prevent an application getting listed under a system for merely
-mentioning the system, edit the [skip_system.txt](scripts/skip_system.txt) file.
-
-```text
-Workaround:
-
-...
-
-SKIP_<system1> application.md
-SKIP_<system2> application.md
-```
-
-where `<system1>` and `<system2>` are as above.
-
 ## How do I add footnotes?
 
 Usage of the footnotes feature is described

csc-overrides/assets/stylesheets/catalog.css

@@ -0,0 +1,57 @@
+.md-typeset .app-heading-row {
+  display: flex;
+  justify-content: start;
+  align-content: center;
+  margin: 0 0 1.25em;
+}
+
+.md-typeset .app-heading {
+  margin: unset;
+}
+
+.md-typeset .app-description {
+  font-weight: 300;
+  text-align: center;
+  color: var(--c-tertiary-800);
+  margin-left: .5ch;
+}
+
+.md-typeset .app-heading-row .app-description {
+  font-size: 1.50em;
+}
+
+.md-typeset .app-link {
+  margin-right: 1em;
+}
+
+.md-typeset nav .app-tags {
+  margin: 0 0 .75em;
+}
+
+.md-typeset c-accordion-item {
+  margin-bottom: 1em;
+}
+
+.md-typeset .alpha-toc {
+  max-width: 80%;
+}
+
+.md-typeset .alpha-toc ul {
+  display: flex;
+  justify-content: start;
+  flex-wrap: wrap;
+  margin-left: 0;
+}
+
+.md-typeset .alpha-toc ul li {
+  list-style-type: none;
+  margin: 0 .25em;
+  text-align: center;
+}
+
+.md-typeset .alpha-toc a {
+  padding: .125em .5em;
+  border-radius: 25%;
+  font-size: large;
+  font-weight: var(--csc-font-weight--bold);
+}

csc-overrides/assets/stylesheets/extra.css

@@ -15,5 +15,6 @@
 @import url(./tags.css);
 @import url(./title.css);
 @import url(./video.css);
+@import url(./catalog.css);
 
 @import url(./sensitive-data.css);

csc-overrides/assets/stylesheets/tags.css

@@ -1,17 +1,17 @@
-.md-tag {
+.md-typeset .md-tag {
   border: 1px solid var(--md-primary-fg-color);
-  color: var(--md-primary-fg-color) !important;
-  background-color: transparent !important;
+  color: var(--md-primary-fg-color);
+  background-color: transparent;
   border-radius: 15px;
-  font-weight: 400 !important;
-  text-decoration: none !important;
+  font-weight: 400;
+  text-decoration: none;
 }
 
-:target>.md-tag {
-  color: var(--md-primary-bg-color) !important;
-  background-color: var(--md-primary-fg-color) !important;
+.md-typeset :target>.md-tag {
+  color: var(--md-primary-bg-color);
+  background-color: var(--md-primary-fg-color);
 }
 
-a.md-tag:is(:active, :hover) {
-  background-color: var(--md-accent-bg-color) !important;
+.md-typeset .md-tag:is(:active, :hover) {
+  background-color: var(--md-accent-bg-color);
 }

csc-overrides/assets/stylesheets/variables.css

@@ -1,7 +1,7 @@
 /* Museo Sans font */
 @import url("https://use.typekit.net/csv4mjm.css");
 /* CSC Design System custom properties */
-@import url(https://cdn.jsdelivr.net/npm/@cscfi/csc-ui@2.1.11/dist/styles/css/theme.css);
+@import url(https://cdn.jsdelivr.net/npm/@cscfi/csc-ui@2.3.0/dist/styles/css/theme.css);
 
 :root {
   --md-text-font: "museo-sans";

development/packages.txt

@@ -5,3 +5,4 @@ mkdocs-git-revision-date-localized-plugin
 mkdocs-redirects
 mkdocs-section-index
 feedparser
+pyhumps

docs/apps/abaqus.md

@@ -1,6 +1,14 @@
 ---
 tags:
   - Academic
+catalog:
+  name: ABAQUS
+  description: Dassault Systemes' SIMULIA academic research suite
+  license_type: Academic
+  disciplines:
+    - Computational Engineering
+  available_on:
+    - Puhti
 ---
 
 # ABAQUS

docs/apps/accelerated-visualization.md

@@ -1,9 +1,17 @@
 ---
 tags:
   - Free
-system:
-  - www-puhti
-  - www-lumi
+catalog:
+  name: Accelerated visualization
+  description: A selection of GPU accelerated visualization applications
+  license_type: Free
+  disciplines:
+    - Miscellaneous
+  available_on:
+    - web_interfaces:
+        - LUMI
+        - Puhti
+    - LUMI
 ---
 
 # Accelerated visualization

docs/apps/alphafold.md

@@ -1,6 +1,15 @@
 ---
 tags:
   - Free
+catalog:
+  name: Alphafold
+  description: Protein 3D structure prediction
+  license_type: Free
+  disciplines:
+    - Biosciences
+  available_on:
+    - Puhti
+    - Mahti
 ---
 
 # Alphafold
@@ -9,20 +18,20 @@ AlphaFold is an AI system developed by [DeepMind](https://www.deepmind.com/) tha
 
 [TOC]
 
-# AlphaFold 3
+## AlphaFold 3
 
 AlphaFold 3 is available on Mahti.
 
-## License
+### License
 
 The AlphaFold 3 inference code is available under a [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en) license.
 The model parameters are available under a separate terms of use agreement and have to be obtained by each user directly from Google as described in the [AlphaFold 3 documentation](https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#obtaining-model-parameters).
 
-## Available
+### Available
 
 -   Mahti: 3.0.1
 
-## Usage
+### Usage
 
 To initialize on Mahti use:
 ```bash
@@ -34,14 +43,14 @@ To print the available command line options:
 run_alphafold --helpshort
 ```
 
-### Database
+#### Database
 
 The genetic databases needed for evolutionary search are hosted at `/mnt/datasets/alphafold`.
 CSC maintains a single version of these databases. If you need a different version, you can download it yourself.
 See [download instructions](https://github.com/google-deepmind/alphafold3/blob/main/docs/installation.md#obtaining-genetic-databases) and the Job Script Examples.
 At time of writing the databases were about 700 GB and it took 30 minutes to download them.
 
-### Job Script Examples
+#### Job Script Examples
 All the examples here use the example input from the [AlphaFold 3 GitHub page](https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#installation-and-running-your-first-prediction):
 ```json
 {
@@ -60,7 +69,7 @@ All the examples here use the example input from the [AlphaFold 3 GitHub page](h
 }
 ```
 
-#### Data pipeline job
+##### Data pipeline job
 Since GPUs are not needed for the first stage of the workflow, it may make sense to perform this on a CPU node as follows:
 ```bash
 #!/bin/bash
@@ -78,7 +87,7 @@ module load alphafold/3.0.1
 srun time run_alphafold --json_path=af_input/fold_input.json --output_dir=af_output --db_dir=/mnt/datasets/alphafold --norun_inference --run_data_pipeline --jackhmmer_n_cpu=8
 ```
 
-#### Inference job
+##### Inference job
 And then perform the second stage on a GPU node.
 ```bash
 #!/bin/bash
@@ -97,7 +106,7 @@ module load alphafold/3.0.1
 time run_alphafold --json_path=af_output/2pv7/2pv7_data.json --model_dir=</path/to/dir/containing/weight/file/> --output_dir=af_output --run_inference --norun_data_pipeline
 ```
 
-#### Data pipeline job using fast local disk
+##### Data pipeline job using fast local disk
 It is also possible to copy the databases to the node local disk.
 Since copying the databases to the local disk introduces some overhead (during testing it took about 40 minutes), this may only lead to overall performance gains when running many large queries in bulk.
 ```bash
@@ -121,7 +130,7 @@ srun ls $LOCAL_SCRATCH
 srun time run_alphafold --json_path=af_input/fold_input.json --output_dir=af_output2 --db_dir=$LOCAL_SCRATCH --norun_inference --run_data_pipeline --jackhmmer_n_cpu=8
 ```
 
-#### Download databases
+##### Download databases
 CSC hosts these databases under `/mnt/datasets/alphafold`. If you need a newer version you can download it with this job script and [this](https://github.com/google-deepmind/alphafold3/blob/main/fetch_databases.sh) download script.
 ```bash
 #!/bin/bash
@@ -138,30 +147,30 @@ export DB_DIR=/scratch/${SLURM_JOB_ACCOUNT}/${USER}/db_dir
 bash <path/to/script/>/fetch_databases.sh $DB_DIR
 ```
 
-#### Chaining Jobs
+##### Chaining Jobs
 To run the data pipeline first and then start the inference job as soon as the first one is finished, you can chain them like this:
 ```bash
 sbatch run_datapipeline.slurm
 sbatch --dependency=afterok:<JOBID> run_inference.slurm
 ```
 
-## More Information
+### More Information
 See [AlphaFold 3 documentation](https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#alphafold-3).
 
-# AlphaFold 2
+## AlphaFold 2
 
 Alphafold 2 is available on Puhti.
 
-## License
+### License
 
 Free to use and open source under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
 
-## Available
+### Available
 
 -   Puhti: 2.0.1, 2.3.0, 2.3.2-1
 
 
-## Usage
+### Usage
 
 To initialize in Puhti use:
 
@@ -187,7 +196,7 @@ $ALPHAFOLD_DIR/alphafold_gpu.slurm
 ```
 
 
-### Which version to use
+#### Which version to use
 
 AlphaFold analysis consists of three stages:
   - Multiple sequence alignment (CPU only)
@@ -206,7 +215,7 @@ It may be difficult to know beforehand, so do some testing. If the run is taking
 more than 3-4 hours on CPU, you should try using GPU.
 
 
-### Database
+#### Database
 
 AlphaFold needs a set of sequence databases to run. The total size of of these
 databases is almost 3 TiB.
@@ -245,7 +254,7 @@ cp -r /scratch/project_12345/alphafold_db .
 export ALPHAFOLD_DATADIR=$LOCAL_SCRATCH/alphafold_db
 ```
 
-## More Information
+### More Information
 
 *   [AlphaFold Homepage](https://github.com/google-deepmind/alphafold/)
 *   The Puhti installation is based on [Alphafold_singularity](https://github.com/prehensilecode/alphafold_singularity)