Validators report STAC as valid although it is not

[m-mohr]

The JSON Schemas are not very complete with regards to (mostly) assets, links and summaries, e.g. #1241. Many related requirements are not checked. The Validators that we have all report something similar to a boolean "valid" or "invalid" although they can't fully guarantee the actual file is valid. Thus, implementors claim their implementation is valid although it is not and blame us for that.

In JSON Schema draft-07 we can't easily validate assets anyway due to them being an object of objects. We'd need to update to the upcoming draft and then it's still a major effort to make the JSON Schemas more complete (and they will likely never be fully complete).

So I'm wondering whether we should update the validators to explain that?
So if the validation status is currently "valid", maybe it should just say something like "no obvious errors found (but that doesn't guarantee correctness)".

Thoughts?

[jonhealy1]

We could add a disclaimer message to stac-validator output but it would be hard to move away from boolean validation. We could use passed which is maybe closer to the truth than valid? We can't guarantee validity but we can say that an object has passed or failed jsonschema validation with certainty.

[m-mohr]

You are the native speaker, aren't you? :-)
But here's my first proposal:

Schema-based STAC validation may be incomplete and should only be considered as a first indicator of validity.

Happy to improve it based on your suggestions.

[m-mohr]

Maybe we should write up somewhere what the known limitations are and link to that issue. This discussion is probably a bit vague right now.

[jonhealy1]

I think that would be really good. The message you wrote is perfect I think.

[m-mohr]

Cool, I've opened issues on my repos already:

• Add validation warning moregeo-it/stac-node-validator#90
• Add validation warning stac-browser#610

[jonhealy1]

Ok looks good, I did the same in stac-check/ validator

• Add validator warning stac-utils/stac-validator#261
• Add validation warning stac-utils/stac-check#133