feat: Add opam source (https://github.com/ocaml/security-advisories) to testing

[hannesm]

ref #4581

Now that #4582 has been merged...

[another-rex]

/gcbrun

[another-rex]

LGTM! Just a note on the OPAM database entries, it would be better to generate them with indentation rather than all on one line, as it makes it easy for consumers to see the changes to a record in the git history.

[another-rex]

@hannesm Seems like we are getting rate limited when enumerating versions.

A couple of options here:

1. It seems like the records we are getting are pre-enumerated already anyway, so the easiest way is to just skip enumeration for ocaml records and rely on the record itself being correctly enumerated.
2. Apparently we get higher rate limits if we have an API token, though it does mean a bit more complexity on our part.

Let me know if you're happy with option 1., otherwise we can look into option 2.

@michaelkedar fyi

[hannesm]

Thanks @another-rex.

@hannesm Seems like we are getting rate limited when enumerating versions.

I'm sorry to hear.

A couple of options here:

1. It seems like the records we are getting are pre-enumerated already anyway, so the easiest way is to just skip enumeration for ocaml records and rely on the record itself being correctly enumerated.

I'm not sure I fully understand what you mean, but indeed the osv data we generate carries an array of versions that are affected. And this is filled by the tooling that generates the json. So you can rely on that data.

2. Apparently we get higher rate limits if we have an API token, though it does mean a bit more complexity on our part.

Let me know if you're happy with option 1., otherwise we can look into option 2.

As mentioned, if I understand that correctly, I'm fine with 1. I don't know what needs to be done in the osv code to accommodate this option.

[hannesm]

I tried to find the test instance database where I'd be able to see the imported json data etc. -- any chance you can provide me with a way how to browse that data? At https://test.osv.dev I don't see any imported opam advisories.

Thanks.