Improve the documentation about external service (integrations) (#1073)

Signed-off-by: tdruez <[email protected]>

Author: tdruez

docs/application-settings.rst

@@ -24,8 +24,11 @@ file.
 
     docker compose restart web worker
 
+Instance settings
+-----------------
+
 DATABASE
---------
+^^^^^^^^
 
 The database can be configured using the following settings::
 
@@ -38,7 +41,7 @@ The database can be configured using the following settings::
 .. _scancodeio_settings_require_authentication:
 
 SCANCODEIO_REQUIRE_AUTHENTICATION
----------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 By default, the ScanCode.io Web UI and REST API are available without any
 authentication.
@@ -59,7 +62,7 @@ authentication system in the REST API.
 .. _scancodeio_settings_workspace_location:
 
 SCANCODEIO_WORKSPACE_LOCATION
------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This setting defines the workspace location of a given project.
 The **workspace** is the directory where **all of the project's files are stored**
@@ -74,7 +77,7 @@ See :ref:`project_workspace` for more details.
 .. _scancodeio_settings_config_dir:
 
 SCANCODEIO_CONFIG_DIR
----------------------
+^^^^^^^^^^^^^^^^^^^^^
 
 The location of the :guilabel:`.scancode/` configuration directory within the project
 codebase.
@@ -91,7 +94,7 @@ it to ScanCode.io. The expected location of the attribution template is::
   .scancode/templates/attribution.html
 
 SCANCODEIO_PROCESSES
---------------------
+^^^^^^^^^^^^^^^^^^^^
 
 By default, multiprocessing is enabled and configured to use an optimal number of CPUs
 available on the machine. You can control the number of parallel processes available
@@ -114,7 +117,7 @@ To disable both multiprocessing and threading, use "-1"::
 .. _scancodeio_settings_async:
 
 SCANCODEIO_ASYNC
-----------------
+^^^^^^^^^^^^^^^^
 
 When enabled, pipeline runs are **executed asynchronously**, meaning that users can
 continue using the app while the pipeline are run in the background.
@@ -139,7 +142,7 @@ synchronously in the web server process.
         export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
 
 SCANCODEIO_TASK_TIMEOUT
------------------------
+^^^^^^^^^^^^^^^^^^^^^^^
 
 Maximum time allowed for a pipeline to complete.
 The pipeline run will be stopped and marked as failed if that limit is reached.
@@ -152,7 +155,7 @@ The value is a string with specify unit including hour, minute, second
 Default: ``24h``
 
 SCANCODEIO_SCAN_FILE_TIMEOUT
-----------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Maximum time allowed for a file to be analyzed when scanning a codebase.
 
@@ -165,7 +168,7 @@ Default: ``120`` (2 minutes)
 .. _scancodeio_settings_pipelines_dirs:
 
 SCANCODEIO_PIPELINES_DIRS
--------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This setting defines any additional locations that ScanCode.io will search in
 for pipelines.
@@ -177,7 +180,7 @@ of additional pipelines directories::
 .. _scancodeio_settings_policies_file:
 
 SCANCODEIO_POLICIES_FILE
-------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
 
 This setting defines the location of the policies file, or ``policies.yml``.
 A valid policies file is required to enable compliance-related features.
@@ -208,15 +211,15 @@ A valid policies file is required to enable compliance-related features.
     this feature.
 
 SCANCODEIO_PAGINATE_BY
-----------------------
+^^^^^^^^^^^^^^^^^^^^^^
 
 The number of objects display per page for each object type can be customized with the
 following setting::
 
     SCANCODEIO_PAGINATE_BY=project=30,error=50,resource=100,package=100,dependency=100
 
 SCANCODEIO_REST_API_PAGE_SIZE
------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 A numeric value indicating the number of objects returned per page in the REST API::
 
@@ -228,7 +231,7 @@ Default: ``50``
     Using a large page size may have an impact on performances.
 
 SCANCODEIO_LOG_LEVEL
---------------------
+^^^^^^^^^^^^^^^^^^^^
 
 By default, only a minimum of logging messages is displayed in the console, mostly
 to provide some progress about pipeline run execution.
@@ -254,7 +257,7 @@ The web server can be started in DEBUG mode with:
     $ SCANCODEIO_LOG_LEVEL=DEBUG make run
 
 TIME_ZONE
----------
+^^^^^^^^^
 
 A string representing the time zone for the current ScanCode.io installation. By
 default the ``UTC`` time zone is used::
@@ -265,30 +268,48 @@ default the ``UTC`` time zone is used::
     You can view a detailed list of time zones `here.
     <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones>`_
 
+.. _scancodeio_settings_external_services:
+
+External services (integrations)
+--------------------------------
+
 .. _scancodeio_settings_purldb:
 
 PURLDB
-------
+^^^^^^
+
+A public instance of **PurlDB** is accessible at https://public.purldb.io/.
 
-Provide the URL and API key of your `PurlDB <https://github.com/nexB/purldb/>`_
-instance.
+Alternatively, you can deploy your own instance of PurlDB by
+following the instructions provided in the documentation at
+https://purldb.readthedocs.io/.
 
- ::
+To configure your local environment, set the ``PURLDB_URL`` in your ``.env`` file::
 
-    PURLDB_URL=https://your-purldb-domain/
-    PURLDB_API_KEY=apikeyexample
+    PURLDB_URL=https://public.purldb.io/
+
+While using the public PurlDB instance, providing an API key is optional.
+However, if authentication is enabled on your PurlDB instance, you can provide the
+API key using ``PURLDB_API_KEY``::
+
+    PURLDB_API_KEY=insert_your_api_key_here
 
 .. _scancodeio_settings_vulnerablecode:
 
 VULNERABLECODE
---------------
+^^^^^^^^^^^^^^
 
-You can either run your own instance of
+You have the option to either deploy your instance of
 `VulnerableCode <https://github.com/nexB/vulnerablecode/>`_
-or connect to the public one.
+or connect to the `public instance <https://public.vulnerablecode.io/>`_.
 
-Authentication is provided using an API key that you can obtain by registering at
-https://public.vulnerablecode.io/account/request_api_key/ ::
+To configure your local environment, set the ``VULNERABLECODE_URL`` in your ``.env``
+file::
 
     VULNERABLECODE_URL=https://public.vulnerablecode.io/
-    VULNERABLECODE_API_KEY=apikeyexample
+
+When using the public VulnerableCode instance, providing an API key is optional.
+However, if authentication is enabled on your VulnerableCode instance,
+you can provide the API key using ``VULNERABLECODE_API_KEY``::
+
+    VULNERABLECODE_API_KEY=insert_your_api_key_here

docs/built-in-pipelines.rst

@@ -44,8 +44,14 @@ Analyse Docker Windows Image
 
 .. _pipeline_find_vulnerabilities:
 
-Find Vulnerabilities
---------------------
+Find Vulnerabilities (addon)
+----------------------------
+
+.. warning::
+    This pipeline requires access to a VulnerableCode database.
+    Refer to :ref:`scancodeio_settings_vulnerablecode` to configure access to
+    VulnerableCode in your ScanCode.io instance.
+
 .. autoclass:: scanpipe.pipelines.find_vulnerabilities.FindVulnerabilities()
     :members:
     :member-order: bysource
@@ -84,10 +90,30 @@ Map Deploy To Develop
     :members:
     :member-order: bysource
 
+.. _pipeline_match_to_purldb:
+
+Match to PurlDB (addon)
+-----------------------
+
+.. warning::
+    This pipeline requires access to a PurlDB service.
+    Refer to :ref:`scancodeio_settings_purldb` to configure access to PurlDB in your
+    ScanCode.io instance.
+
+.. autoclass:: scanpipe.pipelines.match_to_purldb.MatchToPurlDB()
+    :members:
+    :member-order: bysource
+
 .. _pipeline_populate_purldb:
 
-Populate PurlDB
----------------
+Populate PurlDB (addon)
+-----------------------
+
+.. warning::
+    This pipeline requires access to a PurlDB service.
+    Refer to :ref:`scancodeio_settings_purldb` to configure access to PurlDB in your
+    ScanCode.io instance.
+
 .. autoclass:: scanpipe.pipelines.populate_purldb.PopulatePurlDB()
     :members:
     :member-order: bysource

docs/faq.rst

@@ -43,18 +43,26 @@ Here are some general guidelines based on different input scenarios:
 These pipelines will automatically execute the necessary steps to scan and create the
 packages, dependencies, and resources for your project based on the input data provided.
 
-After running one of the above pipelines, you may further **enhance your project data**
-by running some of the following additional pipelines:
+After executing one of the pipelines mentioned above, you have the option to
+**augment your project's data** by executing additional pipelines, often referred to
+as **addon** pipelines.
+These additional pipelines offer further enhancements and modifications to your
+existing data, allowing for more comprehensive analysis and insights.
 
 - If you wish to **find vulnerabilities** for packages and dependencies, you can use the
   :ref:`find_vulnerabilities <pipeline_find_vulnerabilities>` pipeline.
   Note that setting up :ref:`VulnerableCode <scancodeio_settings_vulnerablecode>` is
   required for this pipeline to function properly.
 
-- To **populate the PurlDB** with your project discovered packages, use the
-  :ref:`populate_purldb <pipeline_populate_purldb>` pipeline.
-  Please ensure that you have set up
-  :ref:`PurlDB <scancodeio_settings_purldb>` before running this pipeline.
+- To **populate PurlDB with the packages discovered in your project**,
+  use the :ref:`populate_purldb <pipeline_populate_purldb>` pipeline.
+  Before executing this pipeline, make sure to set up
+  :ref:`PurlDB <scancodeio_settings_purldb>`.
+
+- To **match your project codebase resources to PurlDB for Package matches**,
+  utilize the :ref:`match_to_purldb <pipeline_match_to_purldb>` pipeline.
+  It's essential to set up :ref:`PurlDB <scancodeio_settings_purldb>` before executing
+  this pipeline.
 
 What is the difference between scan_codebase and scan_single_package pipelines?
 -------------------------------------------------------------------------------

docs/tutorial_vulnerablecode_integration.rst

@@ -16,22 +16,20 @@ Configure VulnerableCode integration
 .. warning::
     The ``find_vulnerabilities`` pipeline requires access to a VulnerableCode database.
 
-You can either run your own instance of VulnerableCode or connect to the public one.
-Authentication is provided using an API key that you can obtain by registering at
-https://public.vulnerablecode.io/account/request_api_key/
+You have the option to either deploy your instance of
+`VulnerableCode <https://github.com/nexB/vulnerablecode/>`_
+or connect to the `public instance <https://public.vulnerablecode.io/>`_.
 
-Set the ``VULNERABLECODE_URL`` and ``VULNERABLECODE_API_KEY`` in your local settings:
-  - in the ``docker.env`` file if your run with docker
-  - in the ``.env`` for a local development deployment
-
-For example::
+To configure your local environment, set the ``VULNERABLECODE_URL`` in your ``.env``
+file::
 
     VULNERABLECODE_URL=https://public.vulnerablecode.io/
-    VULNERABLECODE_API_KEY=c1fa7dc1fd0a408880ba2dfdf63c1124abca9477
 
-.. note::
-    Optionally contact nexB support at [email protected] with your API user email if
-    you are doing a larger scale evaluation and need to ease API throttling limitations.
+**Restarting the services is required following any changes to .env:**
+
+.. code-block:: bash
+
+    docker compose restart web worker
 
 Run the ``find_vulnerabilities`` pipeline
 -----------------------------------------

scanpipe/pipelines/match_to_purldb.py

@@ -26,12 +26,14 @@
 
 class MatchToPurlDB(Pipeline):
     """
-    Check CodebaseResources of a Project against PurlDB for Package matches.
+    Match the codebase resources of a project against PurlDB to identify packages.
 
-    This involves creating a JSON scan of the Project codebase, sending it to
-    MatchCode on PurlDB, waiting for match results, creating DiscoveredPackages
-    from the match results Package data and associating the proper
-    CodebaseResources to those DiscoveredPackges.
+    This process involves:
+
+    1. generating a JSON scan of the project codebase
+    2. transmitting it to MatchCode on PurlDB and awaiting match results
+    3. creating discovered packages from the package data obtained
+    4. associating the codebase resources with those discovered packages
     """
 
     download_inputs = False