Provider performance regression

[gammazero]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

v0.34.1

Config

Description

There is a reported performance regression betweek Kubo v0.32.1 (boxo v0.24.3) and Kubo v0.34.1 (boxo v0.29.1)

Initial Report

From: @acejam

The issue: We have a handful of Kubo nodes that are receiving a significant level of add 's per second and ultimately each bulk reprovide run takes too long, causing a snowball effect when the next run kicks off. The net result is a delay in making a DHT announcement.

Bitswap Config

"Internal": {
    "Bitswap": {
      "EngineBlockstoreWorkerCount": 600,
      "EngineTaskWorkerCount": 600,
      "MaxOutstandingBytesPerPeer": 20971520,
      "ProvideEnabled": true,
      "TaskWorkerCount": 600
    }
  },

Build Latest boxo (main branch)

Built the latest commit from boxo into Kubo 0.34.1 and it did not produce any improvement unfortunately.

Conclusion

@acejam Confirmed that downgrading to Kubo 0.32.1 improves providing performance significantly - probably a 1000x+ gain. Based on the fullrtdht logs, it appears that both bulkMessageSend() and Provide() are called in fullrt/dht.go - leading to possibly two announcements per key?
From what I can tell, bulkMessageSend() still runs very slowly, but the extra Provide() call (which is not called in latest kubo/boxo) appears to be helping tremendously.

Possible Areas to investigate

It is worth noting that the kubo v0.33.0 release notes state:

From now on, Bitswap will not be doing any providing on-the-side and all announcements are managed by the reprovider. In some cases, when the reproviding queue is full with other elements, this may cause additional delays, but more likely this will result in improved block-providing behaviour overall.

It may also be reasonable to look for opportunities for horizontal scaling (i.e. adding more concurrent workers doing provide work), to ensure each Kubo announces fast enough

[guillaumemichel]

• feat: Prevent multiple instances of "ipfs routing reprovide" running together. boxo#834 ( associated with Prevent multiple instances of "ipfs bitswap reprovide" running at the same time #10513) prevent multiple reprovide to run concurrently.
• After Bitswap: move providing -> Exchange-layer, providerQueryManager -> routing boxo#641, Bitswap doesn't directly call fullrt.Provide() anymore. All provides go through the boxo/provider module.

It is possible that new provides are in competition with keys being reprovided during a reprovide. If a bulk (re)provide is already in process, all fresh CIDs have to wait for the bulk (re)provide to finish before being provided. Hence it may be really slow to provide new keys during a reprovide.

I am implementing a solution to avoid these spikes when reproviding. However, in the meanwhile, it seems reasonable to prioritize providing new keys over reproviding older keys by having a dedicated queue for new provides (see ipfs/boxo#907).

Additionally, increasing fullrt BulkSendParallelism should accelerate (re)provides, at the expense of being more resource intensive. If it helps we could add this parameter to kubo config.

[acejam]

Thank you for filing this issue @gammazero. Here are some more details:

Our use case involves using Kubo to pin lots of content of varying sizes. This is done by making calls to /api/v0/add.

This content only resides on a Kubo node for a short amount of time. After some time, the data is transferred to another system, then unpinned by calling /api/v0/pin/rm and then subsequently removed by an out of band GC process. In fact, most of these CID's are actually never re-provided since we run with a Reprovider setting of 12h and a pinned strategy. It is only the performance of the initial provide that actually matters here. Therefore, I don't believe there is any contention of provide vs reprovide present - which also means changing FIFO to LIFO on the queues won't result in any noticeable change.

We are now running 0.34.1 with a BulkSendParallelism value of 40. While this has helped a small amount compared to a vanilla install of 0.34.1, provide performance of new content is still significantly slower when compared to Kubo 0.32.1. (5 minutes vs 5 seconds)

Here are some basic log snippets:

bulk send: 70 peers for group size 12
bulk sending: 70.1% done - 1680/2395 done

We also added some debug logging to help us better understand how chunkSize (chunkSize := (len(sortedKeys) * dht.bucketSize * 2) / numPeers) is calculated:

chunkSize: 1, len(sortedKeys): 1, dht.bucketSize: 20, numPeers: 9244

A bulk send of 2,395 keys will take just about 5 minutes to run. That means the last CID in the batch cannot be found on the DHT for a full 5 minutes. We also noted that there are cases where the provider system has a mind of it's own, and appears to wait for various conditions to be met before starting occasionally. @hsanjuan noted a similar observation here: #10595 (comment)

It is clear that the removal of direct calls to fullrt.Provide() from 0.33.0 and newer (ipfs/boxo#641) have caused this regression. Is there a reason why these calls to Provide() were removed?