Ensure that we check the YAML syntax of license data files in tests.

[akirataguchi115]

it is difficult for humans to predict how a given yaml document will parse [1]. For instance https://github.com/aboutcode-org/scancode-licensedb/blob/main/docs/lgpl-2.0-fltk.yml is an invalid yaml document although it doesn't seem like it. I discovered this while parsing the yamls of the repository with pyyaml.

Should we deprecate and remove all of the yaml version from this repo?

1: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell

[pombredanne]

@akirataguchi115 re:

Should we deprecate and remove all of the yaml version from this repo?

I agree that YAML can be problematic, and the spec is one of the most complex document that I have seen, so much so that we use yaml frontmatter now for license original data in https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/3dslicer-1.0.LICENSE

Yet, when used minimally, YAML is highly readable by humans and machines. And it can wrap long strings which makes big difference with JSON when using VCS.

Having the text part of a YAML attribute is likely a mistake, and front matter may be better.

As some folks may rely on this data format, I would rather have tests to validate that the YAML data is correct and can parse alright, rather than dropping it.

Can I interest you in adding a test? This would be best done in the command line that creates the data at https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/license_db.py and this could simply fail the data dump. And this would likely fail the test at https://github.com/aboutcode-org/scancode-toolkit/blob/develop/tests/licensedcode/test_license_db.py#L26 that tests the dump.

[pombredanne]

TIL: https://www.yaml.info/learn/quote.html#indent via yaml/pyyaml#717 (comment)

[pombredanne]

And a solution to get likely correct YAML would be to:

• indent by one the text (though this may be not 100% needed)
• use the "1" notation in text: |1 to indicate that this is using a one space indent (which is the default unless there are more than one space 😩 )

[akirataguchi115]

As some folks may rely on this data format, I would rather have tests to validate that the YAML data is correct and can parse alright, rather than dropping it.

You're right. It seems I was too hasty when I thought dropping YAML as an offered data format would be the best choice.

I would be interested in adding a test 👍 Thanks for asking!

This would be best done in the command line that creates the data at /src/licensedcode/license_db.py and this could simply fail the data dump. And this would likely fail the test at /tests/licensedcode/test_license_db.py#L26 that tests the dump.

What do the words "data" and "data dump" mean in this context? Shouldn't I create a test in the CI/CD pipeline of scancode-licensedb instead so that it would always fail if there exists an invalid yaml file in the repository during a merge or a pull request?

[pombredanne]

The data stored here is entirely generated from ScanCode toolkit, so this is the place to fix this.

[akirataguchi115]

Ah makes sense. I'm actually writing my thesis right now so I might not be able to make the test anytime soon. I'll have a look in my free time though if it would be simple and rewarding enough for me to create the test.

To double make sure I understood it correct: I should create a license dump in YAML format like in lgpl-2.0-fltk.yml in src/licensedcode/license_db.py and then I should also modify tests/licensedcode/test_license_db.py so that they should fail on a YAML file like the lgpl-2.0-fltkl.yml the way it is now?

[pombredanne]

@akirataguchi115 re: your thesis, do not hesitate to reach out for input or feedback! What is the topic?

As for the test, a simple enough approach could be change the code in

scancode-toolkit/src/licensedcode/license_db.py

Lines 134 to 138 in 9a340fc

	write_file(
	output_path,
	f"{lic.key}.yml",
	saneyaml.dump(license_data_with_text, indent=2)
	)

from

        write_file(
            output_path,
            f"{lic.key}.yml",
            saneyaml.dump(license_data_with_text, indent=2)
        )

to something more or less like this:

        as_yaml  = saneyaml.dump(license_data_with_text, indent=2)
        # this will crash loudly if the YAML is invalid
        saneyaml.load(as_yaml)
        write_file(
            output_path,
            f"{lic.key}.yml",
            as_yaml
        )

The test suite already attempts to dump all the licenses, so this should fail there too without extra code.

[akirataguchi115]

The test suite already attempts to dump all the licenses
Ahh that helps me understand this better. Excited to try out that approach! Thank you.

The topic is public software licenses: https://akirataguchi115.github.io/mscthesis/HY-CS-main.pdf

[pombredanne]

The check is already available in

scancode-toolkit/src/licensedcode/models.py

Line 664 in 395091c

if thorough:

.... but we do not run it may be?

Because the check works:

>>> li.validate(thorough=True, licenses=lics)
({'lgpl-2.0-fltk': ['No SPDX license key'], 'GLOBAL': ['Error invalid YAML text at: lgpl-2.0-fltk, failed during saneyaml.load()Run scancode-reindex-licenses --load-dump to fix this.']}, {}, {})

BUT note that lgpl-2.0-fltk is deprecated.
I am moving this issues to SCTK.

[pombredanne]

We need to make the "thorough" check True in

scancode-toolkit/tests/licensedcode/test_models.py

Line 89 in 395091c

licenses=models.load_licenses(with_deprecated=False),