Exception when running `populate_purldb` on project where previous run was `load_sbom`

[ghsa-retrieval]

Describe the bug
In some cases it appears that the populated_purldb pipeline on a project that previously did load_sbom fails.

Invalid purl: type is a required argument.

Traceback:
  File "/opt/scancodeio/aboutcode/pipeline/__init__.py", line 199, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/populate_purldb.py", line 48, in populate_purldb_with_discovered_dependencies
    purldb.populate_purldb_with_discovered_dependencies(
  File "/opt/scancodeio/scanpipe/pipes/purldb.py", line 369, in populate_purldb_with_discovered_dependencies
    packages = [{"purl": purl} for purl in get_unique_resolved_purls(project)]
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/purldb.py", line 298, in get_unique_resolved_purls
    return {str(PackageURL(*values)) for values in distinct_combinations}
                ^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packageurl/__init__.py", line 369, in __new__
    raise ValueError(f"Invalid purl: {key} is a required argument.")

The PURLs in the SBOM look like this: pkg:pypi/beautifulsoup4@4.13.3

System configuration

• Which version of ScanCode.io are you running?
  • 34.9.5 + custom patch for RQ worker
• Are you running the app using Docker?
  • No; custom helm charts running on Kubernetes
• On which OS?
  • Linux
• What inputs are you using?
  • An SBOM containing python packages generated with cdxgen
• Which pipeline are you running?
  • load_sbom
  • populate_purldb

To Reproduce
Update 2025-03-26:

• It appears that any SBOM that results in dependencies being identified by load_sbom will trigger this error
• Package URL and type columns are not populated in the table shown by ScanCode.io (see screenshot) which may indicate that the internal model is not properly populated or populate_purldb is access the wrong fields

Original:
Not quite clear what the cause is. We are using an SBOM containing pypi packages that trigger the exception after the following steps:

1. Use Load Packages from SBOM in DejaCode. It will create a project and pipeline in ScanCode.io
2. Once the packages have been imported into DejaCode, manually add the pipeline populate_purldb

• The intention was to make PurlDB gather information on all identified packages
• Instead of a success the job fails within seconds due to the error above

Expected behavior
The expected behavior is that populate_purldb works with regular PURLs found in SBOMs, may this be pypi or other packages managers.

Ideally this would also be run by default if PurlDB is configured, since otherwise this requires manual intervention when these pipelines get triggered from DejaCode. If the SBOM contains no download URL just the PURL, then it is essential that the PurlDB is being populated so that Improve Packages from PurlDB has data to work with

Screenshots

[ghsa-retrieval]

It would greatly help to know what kind of input data causes this issue and if there is a work around.

[mjherzog]

@ghsa-retrieval We also need to accommodate the use case where someone is using ScanCode.io without PURLDB.

[ghsa-retrieval]

@mjherzog of course, but if ScanCode.io is configured to use PurlDB it would seem sensible to avoid roundtrips and manual intervention. See the following discussion of steps that we currently have to perform, if populate_purldb works correctly: aboutcode-org/dejacode#289

However, that in itself is unrelated to the exception and the root cause of this issue.

[ghsa-retrieval]

It appears this issue happens whenever there are not just packages but also dependencies identified during a load_sbom pipeline. The reason appears to be that package type and package url fields in the the internal representation are not populated or the populate_purldb pipeline accesses the wrong ones. I've noticed when clicking on the dependencies in ScanCode.io the first columns are not populated.

[ghsa-retrieval]

I can confirm that emptying the dependencies list in a CycloneDX SBOM results in only the packages being listed in ScanCode.io, no dependencies, and subsequently the populate_purldb pipeline works successfully (mostly). However, I've noticed that some PURLs are apparently not supported. A project with npm PURLs works without issue with the adapted SBOM, but it refuses to process PURLs to pypi.

Python project
When using populate_purldb after running load_sbom on a modified SBOM with an empty dependencies list:

2025-03-26 09:12:03.395 Pipeline [populate_purldb] starting
2025-03-26 09:12:03.402 Step [populate_purldb_with_discovered_packages] starting
2025-03-26 09:12:03.463 Populating PurlDB with 239 PURLs from DiscoveredPackage
2025-03-26 09:12:03.959 Couldn't index 239 unsupported PURLs
2025-03-26 09:12:03.962 Step [populate_purldb_with_discovered_packages] completed in 1 seconds
2025-03-26 09:12:03.968 Step [populate_purldb_with_discovered_dependencies] starting
2025-03-26 09:12:04.045 Populating PurlDB with 0 PURLs from DiscoveredDependency
2025-03-26 09:12:04.194 No PURLs found. Skipping.
2025-03-26 09:12:04.218 Populating PurlDB with 0 unresolved PURLs from DiscoveredDependency
2025-03-26 09:12:04.328 No PURLs found. Skipping.
2025-03-26 09:12:04.331 Step [populate_purldb_with_discovered_dependencies] completed in 0 seconds
2025-03-26 09:12:04.336 Pipeline completed in 1 seconds

NPM project
When using populate_purldb after running load_sbom on a modified SBOM with an empty dependencies list:

2025-03-26 09:10:52.228 Pipeline [populate_purldb] starting
2025-03-26 09:10:52.246 Step [populate_purldb_with_discovered_packages] starting
2025-03-26 09:10:52.746 Populating PurlDB with 1,546 PURLs from DiscoveredPackage
2025-03-26 09:10:53.593 Progress: 12% (2/16) ETA: 5 seconds
2025-03-26 09:10:54.970 Progress: 25% (4/16) ETA: 6 seconds
2025-03-26 09:10:56.298 Progress: 37% (6/16) ETA: 6 seconds
2025-03-26 09:10:57.569 Progress: 50% (8/16) ETA: 5 seconds
2025-03-26 09:10:58.929 Progress: 62% (10/16) ETA: 4 seconds
2025-03-26 09:11:00.413 Progress: 75% (12/16) ETA: 3 seconds
2025-03-26 09:11:01.706 Progress: 87% (14/16) ETA: 1 seconds
2025-03-26 09:11:03.519 Progress: 100% (16/16)
2025-03-26 09:11:03.973 Successfully queued 1,546 PURLs for indexing in PurlDB
2025-03-26 09:11:03.996 Step [populate_purldb_with_discovered_packages] completed in 12 seconds
2025-03-26 09:11:04.012 Step [populate_purldb_with_discovered_dependencies] starting
2025-03-26 09:11:04.091 Populating PurlDB with 0 PURLs from DiscoveredDependency
2025-03-26 09:11:04.235 No PURLs found. Skipping.
2025-03-26 09:11:04.266 Populating PurlDB with 0 unresolved PURLs from DiscoveredDependency
2025-03-26 09:11:04.372 No PURLs found. Skipping.
2025-03-26 09:11:04.401 Step [populate_purldb_with_discovered_dependencies] completed in 0 seconds
2025-03-26 09:11:04.416 Pipeline completed in 12 seconds

Edit: The issue with pypi packages appears to be coming from purldb and not in ScanCode.io. This can be ignored for this issue. See aboutcode-org/purldb#599