More documentation needed on `sbomtype` option

[robross0606]

The tool offers an --sbomtype option but I have been unable to find any documentation on what values are supported for the option flag. The README says:

SPDX, CycloneDX and Syft are all in scope for this repo.

sbom-scorecard score --help simply says:

      --sbomtype string       type of sbom being evaluated (default "guess")

Where are the available options documented other than in the source code?

[robross0606]

I also cannot find any documentation on which version(s) (1.4? 1.6?) or format(s) (XML? JSON?) of the CycloneDX specification are supported. For example, I suspect that only the XML flavor of CycloneDX is currently supported, but the tool doesn't document that anywhere. I'm only guessing because this happened:

> sbom-scorecard score --debug --sbomtype cdx cyclonedx-sbom.json
0 total packages
0% have versions.
0% have licenses.
0% have package digest.
0% have purls.
0% have CPEs.
Has creation info? false
Spec valid? false
==
╔═══╤══════════════════╤════════╤══════════════════════════════════╗
║ # │     Criteria     │ Points │            Reasoning             ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ 1 │ Spec Compliance  │ 0/25   │ Couldn't parse the SBOM          ║
║ 2 │ Package ID       │ 0/20   │ No packages                      ║
║ 3 │ Package Versions │ 0/20   │                                  ║
║ 4 │ Package Licenses │ 0/20   │                                  ║
║ 5 │ Creation Info    │ 0/15   │ SBOM was not generated by a tool ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║                    Total points: 0/100 or 0%                     ║
║                                                                  ║
╚═══╧══════════════════╧════════╧══════════════════════════════════╝

Considering --debug was used here, this is quite non-informative about why the SBOM couldn't be parsed. There are no error messages and no information about what it was attempting to do. Even something as simple as this would be more informative:

Parsing /my/file/cyclonedx-sbom.json as CycloneDX SBOM XML...
0 total packages
0% have versions.
0% have licenses.
0% have package digest.
0% have purls.
0% have CPEs.
Has creation info? false
Spec valid? false
==
╔═══╤══════════════════╤════════╤══════════════════════════════════╗
║ # │     Criteria     │ Points │            Reasoning             ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ 1 │ Spec Compliance  │ 0/25   │ Couldn't parse the CycloneDX SBOM XML          ║
║ 2 │ Package ID       │ 0/20   │ No packages                      ║
║ 3 │ Package Versions │ 0/20   │                                  ║
║ 4 │ Package Licenses │ 0/20   │                                  ║
║ 5 │ Creation Info    │ 0/15   │ SBOM was not generated by a tool ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║                    Total points: 0/100 or 0%                     ║
║                                                                  ║
╚═══╧══════════════════╧════════╧══════════════════════════════════╝

[robross0606]

After reviewing the source I am more confused because CycloneDX/cyclonedx-go is used which appears to support both JSON and XML formats. And this source appears to attempt both flavors. However, when I attempt to parse JSON output from either trivy or cdxgen, I am getting this out of sbom-scorecard:

> sbom-scorecard score --debug ./trivy-sbom.json
Guessed: spdx
0 total packages
0 total files
0% have licenses.
0% have package digest.
0% have package versions.
0% have purls.
0% have CPEs.
0% have file digest.
Spec valid? true
Has creation info? false
==
╔═══╤══════════════════╤════════╤════════════════════════╗
║ # │     Criteria     │ Points │       Reasoning        ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━╢
║ 1 │ Spec Compliance  │ 25/25  │                        ║
║ 2 │ Package ID       │ 0/20   │ No packages            ║
║ 3 │ Package Versions │ 0/20   │ No packages            ║
║ 4 │ Package Licenses │ 0/20   │ No packages            ║
║ 5 │ Creation Info    │ 0/15   │ No creation info found ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━╢
║              Total points: 25/100 or 25%               ║
║                                                        ║
╚═══╧══════════════════╧════════╧════════════════════════╝

Or this when directly specifying the sbomtype:

╚═══╧══════════════════╧════════╧══════════════════════════════════╝
> sbom-scorecard score --debug --sbomtype cdx ./trivy-sbom.json
0 total packages
0% have versions.
0% have licenses.
0% have package digest.
0% have purls.
0% have CPEs.
Has creation info? false
Spec valid? false
==
╔═══╤══════════════════╤════════╤══════════════════════════════════╗
║ # │     Criteria     │ Points │            Reasoning             ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ 1 │ Spec Compliance  │ 0/25   │ Couldn't parse the SBOM          ║
║ 2 │ Package ID       │ 0/20   │ No packages                      ║
║ 3 │ Package Versions │ 0/20   │                                  ║
║ 4 │ Package Licenses │ 0/20   │                                  ║
║ 5 │ Creation Info    │ 0/15   │ SBOM was not generated by a tool ║
╟━━━┼━━━━━━━━━━━━━━━━━━┼━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║                    Total points: 0/100 or 0%                     ║
║                                                                  ║
╚═══╧══════════════════╧════════╧══════════════════════════════════╝

Here is the SBOM file for reference:
trivy-sbom.json

[robross0606]

Well, one mystery is solved. The reason "guess" is thinking this is an spdx file is because the determineSbomType() algorithm is pretty flawed. All it does is parse the entire file into memory, lowercase it, and then search for the string "spdx" in the string. This is both not scalable (memory and cpu on a large input file) and doesn't work if the CycloneDX file happens to have the letters "spdx" in it (which this one does). Is it fair to say that the documentation should recommend never using "guess" if it can be helped?

Perhaps it would be better to at least search for more specific data like:

• "SPDXID"
• "spdxVersion"

[dirsigler]

Having the same issues here, neither Trivy nor Syft SBOMs seem to work and just result in meaningless "debug" information.