Add API for automated discovery of new packages

[rmccue]

To facilitate discoverability of new packages, it'd be useful for aggregators to be able to discover new packages which are available.

There are multiple sources this data could come from:

• Vendors: Vendors could publish a list of which packages they have published, which could come from one or many repositories.
• Repositories: Repositories could publish a list of all packages they host, which could come from one or many vendors.
• Analytics: "Discovered" packages could be automatically found using analytics data.

It would be useful for aggregators to be able to subscribe to many of these feeds so that they can find new packages, without needing authors to submit them manually. This is analogous to search engine spiders automatically finding new websites.

From discussion with @toderash, the primary piece of data we want here is the DID. For a very large source (such as a big repo or the analytics system), there could be a large number of packages, so providing an optional parameter for conditional fetching is handy - ie ?since=<time>

[afragen]

The following endpoint does work and should be useful.

https://fair.git-updater.com/wp-json/minifair/v1/packages

{
"git-updater":"did:plc:afjf7gsjzsqmgc7dlhb553mv"
"handbook-callout-blocks":"did:plc:deoui6ztyx6paqajconl67rz"
}

[toderash]

@chuckadams take note for AC.

We have an end point in Mini-FAIR for this now, but may need to revise the approach to scale for use by a very large repo with many packages. Currently we can get a list of packages and then query each one to check the current version, so the ?since=<time> idea would be a method of retrieving only the relevant diffs. I think we still need another query per package to get the version number (and prime the cache with the updated package if we want, if it's actually being cached and not served directly) but at least you'd only be querying for what you know has been updated.

[afragen]

@toderash since the /packages/ endpoint returns the slug and the DID, and since Mini-FAIR currently requires Git Updater you could grab all the data using /wp-json/git-updater/v1/update-api/?slug=xxx

[toderash]

@afragen true, though I don't want to bake it in that way if there's a mid-term plan for Mini-FAIR to have a DID and be updated that way rather than via GU since that would then trigger required changes in AC (possibly elsewhere) rather than only within Mini-FAIR.

[afragen]

@toderash not sure what my comment has to do with Mini-FAIR potentially having a DID.

[toderash]

@afragen maybe I misunderstood or misstated something there; it's not about having a DID per se. What I was getting at is that if the Mini-FAIR plugin is going to be served/updated from a FAIR repo (e.g. a Mini-FAIR instance serving updates to plugins including FAIR & Mini-FAIR for DID-based federation), then it's likely that GU would be removed from it at some point. On that basis, I wasn't wanting to rely on GU endpoints because that change within the Mini-FAIR plugin would trigger a required change in a different project, viz. AC.

[afragen]

My understanding is that Mini-FAIR isn't being updated via FAIR. Need to check with @rmccue

Also, my understanding is there will always be an integration for Mini-FAIR whether Git Updater, EDD, or something else.

[toderash]

Yeah, we should clarify that. It isn't being updated via FAIR currently, but I understand that to be the plan. I thought this was in the 1.0 milestone, but that's where we'll need to fill in some detail. Maybe we should add this to the agenda for Monday's tech call.

[afragen]

Is Mini-FAIR in the 1.0 milestone or does it have it's own milestones?

[toderash]

It's in 1.0 but we really need to review the 1.0 list and ratify what's doable and on what timeline. The main thing we're lacking atm is an indication of how all the things will get done. The goals are good, if they have a plan!