feat(provide): detailed ipfs provide stat (#11019)

* feat: provide stats

* added N/A

* format

* workers stats alignment

* ipfs provide stat --all --compact

* consolidating compact stat

* update column alignment

* flags combinations errors

* command description

* change schedule AvgPrefixLen to float

* changelog

* alignments

* provide stat description draft

* rephrased provide-stats.md

* linking provide-stats.md from command description

* documentation test

* fix: refactor provide stat command type handling

- add extractSweepingProvider() helper to reduce nested type switching
- extract lowWorkerThreshold constant for worker availability check
- fix --lan error handling to work with buffered providers

* docs: add clarifying comments

* fix(commands): improve provide stat compact mode

- prevent panic when both columns are empty
- fix column alignment with UTF-8 characters
- only track col0MaxWidth for first column (as intended)

* test: add tests for ipfs provide stat command

- test basic functionality, flags, JSON output
- test legacy provider behavior
- test integration with content scheduling
- test disabled provider configurations
- add parseSweepStats helper with t.Helper()

* docs: improve provide command help text

- update tagline to "Control and monitor content providing"
- simplify help descriptions
- make error messages more consistent
- update tests to match new error messages

* metrics rename

```
Next reprovide at:
Next prefix:
```
updated to:
```
Next region prefix:
Next region reprovide:
```

* docs: improve Provide system documentation clarity

Enhance documentation for the Provide system to better explain how provider
records work and the differences between sweep and legacy modes.

Changes to docs/config.md:
- Provide section: add clear explanation of provider records and their role
- Provide.DHT: add provider record lifecycle and two provider systems overview
- Provide.DHT.Interval: explain relationship to expiration, contrast sweep vs legacy behavior
- Provide.DHT.SweepEnabled: rewrite to explain legacy problem, sweep solution, and efficiency gains
- Monitoring section: prioritize command-line tools (ipfs provide stat) before Prometheus

Changes to core/commands/provide.go:
- ipfs provide stat help: add explanation of provider records, TTL expiration, and how sweep batching works

Changes to docs/changelogs/v0.39.md:
- Add context about why stats matter for monitoring provider health
- Emphasize real-time monitoring workflow with watch command
- Explain what users can observe (rates, queues, worker availability)

* depend on latest kad-dht master

* docs: nits

---------

Co-authored-by: Marcin Rataj <[email protected]>

Author: guillaumemichel

core/commands/provide.go

@@ -10,6 +10,8 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
 
 - [Overview](#overview)
 - [🔦 Highlights](#-highlights)
+  - [📊 Detailed statistics for Sweep provider with `ipfs provide stat`](#-detailed-statistics-for-sweep-provider-with-ipfs-provide-stat)
+  - [🪦 Deprecated `go-ipfs` name no longer published](#-deprecated-go-ipfs-name-no-longer-published)
 - [📦️ Important dependency updates](#-important-dependency-updates)
 - [📝 Changelog](#-changelog)
 - [👨‍👩‍👧‍👦 Contributors](#-contributors)
@@ -18,13 +20,55 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
 
 ### 🔦 Highlights
 
+#### 📊 Detailed statistics for Sweep provider with `ipfs provide stat`
+
+The experimental Sweep provider system ([introduced in
+v0.38](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.38.md#-experimental-sweeping-dht-provider))
+now has detailed statistics available through `ipfs provide stat`.
+
+These statistics help you monitor provider health and troubleshoot issues,
+especially useful for nodes providing large content collections. You can quickly
+identify bottlenecks like queue backlog, worker saturation, or connectivity
+problems that might prevent content from being announced to the DHT.
+
+**Default behavior:** Displays a brief summary showing queue sizes, scheduled
+CIDs/regions, average record holders, ongoing/total provides, and worker status
+when resources are constrained.
+
+**Detailed statistics with `--all`:** View complete metrics organized into sections:
+
+- **Connectivity**: DHT connection status
+- **Queues**: Pending provide and reprovide operations
+- **Schedule**: CIDs/regions scheduled for reprovide
+- **Timings**: Uptime, reprovide cycle information
+- **Network**: Peer statistics, keyspace region sizes
+- **Operations**: Ongoing and past provides, rates, errors
+- **Workers**: Worker pool utilization and availability
+
+**Real-time monitoring:** For continuous monitoring, run
+`watch ipfs provide stat --all --compact` to see detailed statistics refreshed
+in a 2-column layout. This lets you observe provide rates, queue sizes, and
+worker availability in real-time. Individual sections can be displayed using
+flags like `--network`, `--operations`, or `--workers`, and multiple flags can
+be combined for custom views.
+
+**Dual DHT support:** For Dual DHT configurations, use `--lan` to view LAN DHT
+provider statistics instead of the default WAN DHT stats.
+
+> [!NOTE]
+> These statistics are only available when using the Sweep provider system
+> (enabled via
+> [`Provide.DHT.SweepEnabled`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providedhtsweepenabled)).
+> Legacy provider shows basic statistics without flag support.
+
 #### 🪦 Deprecated `go-ipfs` name no longer published
 
 The `go-ipfs` name was deprecated in 2022 and renamed to `kubo`. Starting with this release, we have stopped publishing Docker images and distribution binaries under the old `go-ipfs` name.
 
 Existing users should switch to:
+
 - Docker: `ipfs/kubo` image (instead of `ipfs/go-ipfs`)
-- Binaries: download from https://dist.ipfs.tech/kubo/ or https://github.com/ipfs/kubo/releases
+- Binaries: download from <https://dist.ipfs.tech/kubo/> or <https://github.com/ipfs/kubo/releases>
 
 For Docker users, the legacy `ipfs/go-ipfs` image name now shows a deprecation notice directing you to `ipfs/kubo`.
 

docs/changelogs/v0.39.md

@@ -1910,10 +1910,17 @@ Type: `duration`
 
 ## `Provide`
 
-Configures CID announcements to the routing system, including both immediate
-announcements for new content (provide) and periodic re-announcements
-(reprovide) on systems that require it, like Amino DHT. While designed to support
-multiple routing systems in the future, the current default configuration only supports providing to the Amino DHT.
+Configures how your node advertises content to make it discoverable by other
+peers.
+
+**What is providing?** When your node stores content, it publishes provider
+records to the routing system announcing "I have this content". These records
+map CIDs to your peer ID, enabling content discovery across the network.
+
+While designed to support multiple routing systems in the future, the current
+default configuration only supports [providing to the Amino DHT](#providedht).
+
+<!-- TODO: See the [Reprovide Sweep blog post](https://blog.ipfs.tech/2025-reprovide-sweep/) for detailed performance comparisons. -->
 
 ### `Provide.Enabled`
 
@@ -1964,13 +1971,39 @@ Type: `optionalString` (unset for the default)
 
 Configuration for providing data to Amino DHT peers.
 
+**Provider record lifecycle:** On the Amino DHT, provider records expire after
+[`amino.DefaultProvideValidity`](https://github.com/libp2p/go-libp2p-kad-dht/blob/v0.34.0/amino/defaults.go#L40-L43).
+Your node must re-announce (reprovide) content periodically to keep it
+discoverable. The [`Provide.DHT.Interval`](#providedhtinterval) setting
+controls this timing, with the default ensuring records refresh well before
+expiration or negative churn effects kick in.
+
+**Two provider systems:**
+
+- **Sweep provider**: Divides the DHT keyspace into regions and systematically
+  sweeps through them over the reprovide interval. This batches CIDs allocated
+  to the same DHT servers, dramatically reducing the number of DHT lookups and
+  PUTs needed. Spreads work evenly over time with predictable resource usage.
+
+- **Legacy provider**: Processes each CID individually with separate DHT
+  lookups. Works well for small content collections but struggles to complete
+  reprovide cycles when managing thousands of CIDs.
+
 #### Monitoring Provide Operations
 
-You can monitor the effectiveness of your provide configuration through metrics exposed at the Prometheus endpoint: `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`).
+**Quick command-line monitoring:** Use `ipfs provide stat` to view the current
+state of the provider system. For real-time monitoring, run
+`watch ipfs provide stat --all --compact` to see detailed statistics refreshed
+continuously in a 2-column layout.
 
-Different metrics are available depending on whether you use legacy mode (`SweepEnabled=false`) or sweep mode (`SweepEnabled=true`). See [Provide metrics documentation](https://github.com/ipfs/kubo/blob/master/docs/metrics.md#provide) for details.
+**Long-term monitoring:** For in-depth or long-term monitoring, metrics are
+exposed at the Prometheus endpoint: `{Addresses.API}/debug/metrics/prometheus`
+(default: `http://127.0.0.1:5001/debug/metrics/prometheus`). Different metrics
+are available depending on whether you use legacy mode (`SweepEnabled=false`) or
+sweep mode (`SweepEnabled=true`). See [Provide metrics documentation](https://github.com/ipfs/kubo/blob/master/docs/metrics.md#provide)
+for details.
 
-To enable detailed debug logging for both providers, set:
+**Debug logging:** For troubleshooting, enable detailed logging by setting:
 
 ```sh
 GOLOG_LOG_LEVEL=error,provider=debug,dht/provider=debug
@@ -1982,12 +2015,24 @@ GOLOG_LOG_LEVEL=error,provider=debug,dht/provider=debug
 #### `Provide.DHT.Interval`
 
 Sets how often to re-announce content to the DHT. Provider records on Amino DHT
-expire after [`amino.DefaultProvideValidity`](https://github.com/libp2p/go-libp2p-kad-dht/blob/v0.34.0/amino/defaults.go#L40-L43),
-also known as Provider Record Expiration Interval.
+expire after [`amino.DefaultProvideValidity`](https://github.com/libp2p/go-libp2p-kad-dht/blob/v0.34.0/amino/defaults.go#L40-L43).
+
+**Why this matters:** The interval must be shorter than the expiration window to
+ensure provider records refresh before they expire. The default value is
+approximately half of [`amino.DefaultProvideValidity`](https://github.com/libp2p/go-libp2p-kad-dht/blob/v0.34.0/amino/defaults.go#L40-L43),
+which accounts for network churn and ensures records stay alive without
+overwhelming the network with unnecessary announcements.
 
-An interval of about half the expiration window ensures provider records
-are refreshed well before they expire. This keeps your content continuously
-discoverable accounting for network churn without overwhelming the network with too frequent announcements.
+**With sweep mode enabled
+([`Provide.DHT.SweepEnabled`](#providedhtsweepenabled)):** The system spreads
+reprovide operations smoothly across this entire interval. Each keyspace region
+is reprovided at scheduled times throughout the period, ensuring announcements
+periodically happen every interval.
+
+**With legacy mode:** The system attempts to reprovide all CIDs as quickly as
+possible at the start of each interval. If reproviding takes longer than this
+interval (common with large datasets), the next cycle is skipped and provider
+records may expire.
 
 - If unset, it uses the implicit safe default.
 - If set to the value `"0"` it will disable content reproviding to DHT.
@@ -2055,46 +2100,60 @@ Type: `optionalInteger` (non-negative; `0` means unlimited number of workers)
 
 #### `Provide.DHT.SweepEnabled`
 
-Whether Provide Sweep is enabled. If not enabled, the legacy
-[`boxo/provider`](https://github.com/ipfs/boxo/tree/main/provider) is used for
-both provides and reprovides.
-
-Provide Sweep is a resource efficient technique for advertising content to
-the Amino DHT swarm. The Provide Sweep module tracks the keys that should be periodically reprovided in
-the `Keystore`. It splits the keys into DHT keyspace regions by proximity (XOR
-distance), and schedules when reprovides should happen in order to spread the
-reprovide operation over time to avoid a spike in resource utilization. It
-basically sweeps the keyspace _from left to right_ over the
-[`Provide.DHT.Interval`](#providedhtinterval) time period, and reprovides keys
-matching to the visited keyspace region.
-
-Provide Sweep aims at replacing the inefficient legacy `boxo/provider`
-module, and is currently opt-in. You can compare the effectiveness of sweep mode vs legacy mode by monitoring the appropriate metrics (see [Monitoring Provide Operations](#monitoring-provide-operations) above).
-
-Whenever new keys should be advertised to the Amino DHT, `kubo` calls
-`StartProviding()`, triggering an initial `provide` operation for the given
-keys. The keys will be added to the `Keystore` tracking which keys should be
-reprovided and when they should be reprovided. Calling `StopProviding()`
-removes the keys from the `Keystore`. However, it is currently tricky for
-`kubo` to detect when a key should stop being advertised. Hence, `kubo` will
-periodically refresh the `Keystore` at each [`Provide.DHT.Interval`](#providedhtinterval)
-by providing it a channel of all the keys it is expected to contain according
-to the [`Provide.Strategy`](#providestrategy). During this operation,
-all keys in the `Keystore` are purged, and only the given ones remain scheduled.
+Enables the sweep provider for efficient content announcements. When disabled,
+the legacy [`boxo/provider`](https://github.com/ipfs/boxo/tree/main/provider) is
+used instead.
+
+**The legacy provider problem:** The legacy system processes CIDs one at a
+time, requiring a separate DHT lookup (10-20 seconds each) to find the 20
+closest peers for each CID. This sequential approach typically handles less
+than 10,000 CID over 22h ([`Provide.DHT.Interval`](#providedhtinterval)). If
+your node has more CIDs than can be reprovided within
+[`Provide.DHT.Interval`](#providedhtinterval), provider records start expiring
+after
+[`amino.DefaultProvideValidity`](https://github.com/libp2p/go-libp2p-kad-dht/blob/v0.34.0/amino/defaults.go#L40-L43),
+making content undiscoverable.
+
+**How sweep mode works:** The sweep provider divides the DHT keyspace into
+regions based on keyspace prefixes. It estimates the Amino DHT size, calculates
+how many regions are needed (sized to contain at least 20 peers each), then
+schedules region processing evenly across
+[`Provide.DHT.Interval`](#providedhtinterval). When processing a region, it
+discovers the peers in that region once, then sends all provider records for
+CIDs allocated to those peers in a batch. This batching is the key efficiency:
+instead of N lookups for N CIDs, the number of lookups is bounded by a constant
+fraction of the Amino DHT size (e.g., ~3,000 lookups when there are ~10,000 DHT
+servers), regardless of how many CIDs you're providing.
+
+**Efficiency gains:** For a node providing 100,000 CIDs, sweep mode reduces
+lookups by 97% compared to legacy. The work spreads smoothly over time rather
+than completing in bursts, preventing resource spikes and duplicate
+announcements. Long-running nodes reprovide systematically just before records
+would expire, keeping content continuously discoverable without wasting
+bandwidth.
+
+**Implementation details:** The sweep provider tracks CIDs in a persistent
+keystore. New content added via `StartProviding()` enters the provide queue and
+gets batched by keyspace region. The keystore is periodically refreshed at each
+[`Provide.DHT.Interval`](#providedhtinterval) with CIDs matching
+[`Provide.Strategy`](#providestrategy) to ensure only current content remains
+scheduled. This handles cases where content is unpinned or removed.
 
 > <picture>
 >   <source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/f6e06b08-7fee-490c-a681-1bf440e16e27">
 >   <source media="(prefers-color-scheme: light)" srcset="https://github.com/user-attachments/assets/e1662d7c-f1be-4275-a9ed-f2752fcdcabe">
 >   <img alt="Reprovide Cycle Comparison" src="https://github.com/user-attachments/assets/e1662d7c-f1be-4275-a9ed-f2752fcdcabe">
 > </picture>
 >
-> The diagram above visualizes the performance patterns:
+> The diagram compares performance patterns:
 >
-> - **Legacy mode**: Individual (slow) provides per CID, can struggle with large datasets
-> - **Sweep mode**: Even distribution matching the keyspace sweep described with low resource usage
-> - **Accelerated DHT**: Hourly traffic spikes with high resource usage
+> - **Legacy mode**: Sequential processing, one lookup per CID, struggles with large datasets
+> - **Sweep mode**: Smooth distribution over time, batched lookups by keyspace region, predictable resource usage
+> - **Accelerated DHT**: Hourly network crawls creating traffic spikes, high resource usage
 >
-> Sweep mode provides similar effectiveness to Accelerated DHT but with steady resource usage - better for machines with limited CPU, memory, or network bandwidth.
+> Sweep mode achieves similar effectiveness to the Accelerated DHT client but with steady resource consumption.
+
+You can compare the effectiveness of sweep mode vs legacy mode by monitoring the appropriate metrics (see [Monitoring Provide Operations](#monitoring-provide-operations) above).
 
 > [!NOTE]
 > This feature is opt-in for now, but will become the default in a future release.

docs/config.md

@@ -7,9 +7,9 @@ go 1.25
 replace github.com/ipfs/kubo => ./../../..
 
 require (
-	github.com/ipfs/boxo v0.35.1-0.20251016232905-37006871a40e
+	github.com/ipfs/boxo v0.35.0
 	github.com/ipfs/kubo v0.0.0-00010101000000-000000000000
-	github.com/libp2p/go-libp2p v0.44.0
+	github.com/libp2p/go-libp2p v0.43.0
 	github.com/multiformats/go-multiaddr v0.16.1
 )
 
@@ -82,7 +82,7 @@ require (
 	github.com/ipfs/go-ds-flatfs v0.5.5 // indirect
 	github.com/ipfs/go-ds-leveldb v0.5.2 // indirect
 	github.com/ipfs/go-ds-measure v0.2.2 // indirect
-	github.com/ipfs/go-ds-pebble v0.5.5 // indirect
+	github.com/ipfs/go-ds-pebble v0.5.3 // indirect
 	github.com/ipfs/go-dsqueue v0.1.0 // indirect
 	github.com/ipfs/go-fs-lock v0.1.1 // indirect
 	github.com/ipfs/go-ipfs-cmds v0.15.0 // indirect
@@ -98,7 +98,7 @@ require (
 	github.com/ipfs/go-peertaskqueue v0.8.2 // indirect
 	github.com/ipfs/go-test v0.2.3 // indirect
 	github.com/ipfs/go-unixfsnode v1.10.2 // indirect
-	github.com/ipld/go-car/v2 v2.16.0 // indirect
+	github.com/ipld/go-car/v2 v2.15.0 // indirect
 	github.com/ipld/go-codec-dagpb v1.7.0 // indirect
 	github.com/ipld/go-ipld-prime v0.21.0 // indirect
 	github.com/ipshipyard/p2p-forge v0.6.1 // indirect
@@ -115,15 +115,15 @@ require (
 	github.com/libp2p/go-doh-resolver v0.5.0 // indirect
 	github.com/libp2p/go-flow-metrics v0.3.0 // indirect
 	github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect
-	github.com/libp2p/go-libp2p-kad-dht v0.35.1 // indirect
+	github.com/libp2p/go-libp2p-kad-dht v0.35.2-0.20251017193437-abd04263daac // indirect
 	github.com/libp2p/go-libp2p-kbucket v0.8.0 // indirect
 	github.com/libp2p/go-libp2p-pubsub v0.14.2 // indirect
 	github.com/libp2p/go-libp2p-pubsub-router v0.6.0 // indirect
 	github.com/libp2p/go-libp2p-record v0.3.1 // indirect
 	github.com/libp2p/go-libp2p-routing-helpers v0.7.5 // indirect
 	github.com/libp2p/go-libp2p-xor v0.1.0 // indirect
 	github.com/libp2p/go-msgio v0.3.0 // indirect
-	github.com/libp2p/go-netroute v0.3.0 // indirect
+	github.com/libp2p/go-netroute v0.2.2 // indirect
 	github.com/libp2p/go-reuseport v0.4.0 // indirect
 	github.com/libp2p/go-yamux/v5 v5.0.1 // indirect
 	github.com/libp2p/zeroconf/v2 v2.2.0 // indirect
@@ -141,7 +141,7 @@ require (
 	github.com/multiformats/go-multiaddr-dns v0.4.1 // indirect
 	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
 	github.com/multiformats/go-multibase v0.2.0 // indirect
-	github.com/multiformats/go-multicodec v0.10.0 // indirect
+	github.com/multiformats/go-multicodec v0.9.2 // indirect
 	github.com/multiformats/go-multihash v0.2.3 // indirect
 	github.com/multiformats/go-multistream v0.6.1 // indirect
 	github.com/multiformats/go-varint v0.1.0 // indirect
@@ -177,7 +177,7 @@ require (
 	github.com/prometheus/common v0.66.1 // indirect
 	github.com/prometheus/procfs v0.17.0 // indirect
 	github.com/quic-go/qpack v0.5.1 // indirect
-	github.com/quic-go/quic-go v0.55.0 // indirect
+	github.com/quic-go/quic-go v0.54.1 // indirect
 	github.com/quic-go/webtransport-go v0.9.0 // indirect
 	github.com/rogpeppe/go-internal v1.14.1 // indirect
 	github.com/spaolacci/murmur3 v1.1.0 // indirect
@@ -212,22 +212,22 @@ require (
 	go.uber.org/zap/exp v0.3.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go4.org v0.0.0-20230225012048-214862532bf5 // indirect
-	golang.org/x/crypto v0.43.0 // indirect
-	golang.org/x/exp v0.0.0-20251009144603-d2f985daa21b // indirect
-	golang.org/x/mod v0.29.0 // indirect
-	golang.org/x/net v0.46.0 // indirect
+	golang.org/x/crypto v0.42.0 // indirect
+	golang.org/x/exp v0.0.0-20250911091902-df9299821621 // indirect
+	golang.org/x/mod v0.28.0 // indirect
+	golang.org/x/net v0.44.0 // indirect
 	golang.org/x/sync v0.17.0 // indirect
-	golang.org/x/sys v0.37.0 // indirect
-	golang.org/x/telemetry v0.0.0-20251008203120-078029d740a8 // indirect
-	golang.org/x/text v0.30.0 // indirect
+	golang.org/x/sys v0.36.0 // indirect
+	golang.org/x/telemetry v0.0.0-20250908211612-aef8a434d053 // indirect
+	golang.org/x/text v0.29.0 // indirect
 	golang.org/x/time v0.12.0 // indirect
-	golang.org/x/tools v0.38.0 // indirect
+	golang.org/x/tools v0.37.0 // indirect
 	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
 	gonum.org/v1/gonum v0.16.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20250825161204-c5933d9347a5 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20250825161204-c5933d9347a5 // indirect
 	google.golang.org/grpc v1.75.0 // indirect
-	google.golang.org/protobuf v1.36.10 // indirect
+	google.golang.org/protobuf v1.36.9 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	lukechampine.com/blake3 v1.4.1 // indirect
 )