Document test and validation principles

[pombredanne]

@mjherzog wrote in a chat https://aboutcode-org.slack.com/archives/C08LBQMA7DE/p1756676978878109 :

It seems that we need to shift our PURL validation discussions up a level to document some key principles:
One example is several PURL type definitions are coming through with version required. Obviously a version is very important and required for practical use in a vuln detection/reporting context but is it required for validation?

• See also swift - Why the version is required ? #606

Another example is qualifiers where we should not reject unregistered qualifiers, but do not want to validate the correctness of qualifiers that are registered for a purl type. This suggests that we need to define at least error vs warning messages - e.g., an error for a registered qualifier that is not correctly expressed vs a warning for an unregistered qualiifer

One good place to start would be for you to update /docs/tests.md to document some of the key principles / techniques. I am suggesting that you ( @pombredanne ) update this first to express your ideas behind the purl-test schema and then we can focus the higher level community discussion there. The current text is a direct copy from the Test section of PURL-SPECIFICATION.rst (still with the reference to test-suite-data.json)

• See also Update /docs/tests.md for migration from test-suite-data.json to /tests #597

[giterlizzi]

Another example is qualifiers where we should not reject unregistered qualifiers, but do not want to validate the correctness of qualifiers that are registered for a purl type. This suggests that we need to define at least error vs warning messages - e.g., an error for a registered qualifier that is not correctly expressed vs a warning for an unregistered qualiifer

In URI-PackageURL (Perl) I started to implement a check to find unknown qualifiers using the list of "Known PURL qualifiers" + "PURL type-specific qualifiers".

Raising an exception when it finds an unknown qualifier has a catastrophic effect on the 900+ tests I run for PURL.

I chose to issue a "warn" that does not block the execution flow of the program/script but only gives a message to the user, because I think qualifiers should be flexible as we may discover new use cases in the future.

With the same way (raising a warning), I check whether a checksum have the correct length for the specified algorithm (e.g., sha1 --> 40, sha512 --> 128, etc.) or whether it contains invalid characters.

[prabhu]

Include me in the camp that advocates for both name and version being mandatory in a purl across as many types as possible. Sure there will be types like generic where version may not be known. But say for types like github and bitbucket, it is not impossible for the end users to construct better PURLs (including version and some useful and standard qualifiers)

[mjherzog]

We should revisit the discussion regarding why decided to not make version required at the schema level.

[matt-phylum]

With the same way (raising a warning), I check whether a checksum have the correct length for the specified algorithm (e.g., sha1 --> 40, sha512 --> 128, etc.) or whether it contains invalid characters.

I don't think it's that uncommon for hashes to be truncated. Maybe the user wants to be warned about this, but having libraries log warnings about it may not be helpful.

it is not impossible for the end users to construct better PURLs

It is sometimes impossible. If you're talking about the dependencies of a package, an exact version is not specified, and a PURL cannot be constructed if a version is required. The best you can do is add a qualifier with the version range.

[maennchen]

@mjherzog I don't think types should be allowed to require versions. In some cases, it makes sense to identify a package and not a specific version of that package. One case I can think of is the CVE schema RFD for Purl:

https://github.com/CVEProject/cve-schema/blob/e4fe53e746da43bc299057aa3e7bc2f1679165c5/rfds/0002-supporting-package-urls.md?plain=1#L151-L153

In that specific instance, the Purl is only used to identify the version and other fields in the CVE are used to specify affected version ranges.

[prabhu]

@maennchen, In an ideal world, CVE records must have both an accurate list of purls (with version) as well as vers. Projects such as OSV painstakingly add the exact version list for CVEs as an enhancement. The problem with weakening the spec is that people will create purls with just the type and some name and completely forget the version or vers. All such purl-like strings will continue to remain as valid PURLs but not actionable.

[mjherzog]

Agree that a PURL without version is not actionable for vulnerability analysis but it will usually be actionable for licensing. There is also the use case for dependencies where you want to use PURL (w/o version) and VERS.
So we need a nuanced approach to what is required, when and why.