You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(pebble): support pinning FormatMajorVersion (#10789)
* Upgrade to pebble v2.0.3
- Configure latest pebble database format at init
- Do not automatically ratchet database format if set in config
- Daemon messge about new available pebble format
- Document pebble config with formatMajorVersion
- Add warning to users running badger, nudging them to switch to flatfs or pebble
- docs: explain Pebble's `FormatMajorVersion`
- Use pebbleds instead of badgerds in t0060-daemon.sh
- Print badgerds warning message to stderr
-[Pebble Database Format Config](#pebble-database-format-config)
22
23
-[📦️ Important dependency updates](#-important-dependency-updates)
23
24
-[📝 Changelog](#-changelog)
24
25
-[👨👩👧👦 Contributors](#-contributors)
@@ -113,10 +114,24 @@ but for reprovides only.
113
114
> [!NOTE]
114
115
> `ipfs stats provider` still works, but is marked as deprecated and will be removed in a future release. Be mindful that the command provides only statistics about reprovides (similar to `ipfs stats reprovide`) and not the new provide queue (this will be fixed as a part of wider refactor planned for a future release).
115
116
117
+
#### Pebble Database Format Config
118
+
119
+
This Kubo release provides node operators with more control over [Pebble's `FormatMajorVersion`](https://github.com/cockroachdb/pebble/tree/master?tab=readme-ov-file#format-major-versions). This allows testing a new Kubo release without automatically migrating Pebble datastores, keeping the ability to switch back to older Kubo.
120
+
121
+
When IPFS is initialized to use the pebbleds datastore (opt-in via `ipfs init --profile=pebbleds`), the latest pebble database format is configured in the pebble datastore config as `"formatMajorVersion"`. Setting this in the datastore config prevents automatically upgrading to the latest available version when Kubo is upgraded. If a later version becomes available, the Kubo daemon prints a startup message to indicate this. The user can them update the config to use the latest format when they are certain a downgrade will not be necessary.
122
+
123
+
Without the `"formatMajorVersion"` in the pebble datastore config, the database format is automatically upgraded to the latest version. If this happens, then it is possible a downgrade back to the previous version of Kubo will not work if new format is not compatible with the pebble datastore in the previous version of Kubo.
124
+
125
+
When installing a new version of Kubo when `"formatMajorVersion"` is configured, automatic repository migration (`ipfs daemon with --migrate=true`) does not upgrade this to the latest available version. This is done because a user may have reasons not to upgrade the pebble database format, and may want to be able to downgrade Kubo if something else is not working in the new version. If the configured pebble database format in the old Kubo is not supported in the new Kubo, then the configured version must be updated and the old Kubo run, before installing the new Kubo.
126
+
127
+
See other caveats and configuration options at [`kubo/docs/datastores.md#pebbleds`](https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds)
128
+
116
129
#### 📦️ Important dependency updates
117
130
118
131
- update `boxo` to [v0.30.0](https://github.com/ipfs/boxo/releases/tag/v0.30.0)
119
132
- update `ipfs-webui` to [v4.7.0](https://github.com/ipfs/ipfs-webui/releases/tag/v4.7.0)
133
+
- update `go-ds-pebble` to [v0.5.0](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.0)
134
+
- update `pebble` to [v2.0.3](https://github.com/cockroachdb/pebble/releases/tag/v2.0.3)
Copy file name to clipboardExpand all lines: docs/datastores.md
+15-3Lines changed: 15 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,9 +46,6 @@ Uses a leveldb database to store key value pairs.
46
46
47
47
## pebbleds
48
48
49
-
> [!WARNING]
50
-
> This is still **EXPERIMENTAL** opt-in. Datastore format can be set when first initializing the node via `ipfs init --profile pebbleds`.
51
-
52
49
Uses [pebble](https://github.com/cockroachdb/pebble) as a key value store.
53
50
54
51
```json
@@ -64,6 +61,7 @@ If they are not configured (or assigned their zero-valued), then default values
64
61
*`bytesPerSync`: int, Sync sstables periodically in order to smooth out writes to disk. (default: 512KB)
65
62
*`disableWAL`: true|false, Disable the write-ahead log (WAL) at expense of prohibiting crash recovery. (default: false)
66
63
*`cacheSize`: Size of pebble's shared block cache. (default: 8MB)
64
+
*`formatVersionMajor`: int, Sets the format of pebble on-disk files. If 0 or unset, automatically convert to latest format.
67
65
*`l0CompactionThreshold`: int, Count of L0 files necessary to trigger an L0 compaction.
68
66
*`l0StopWritesThreshold`: int, Limit on L0 read-amplification, computed as the number of L0 sublevels.
69
67
*`lBaseMaxBytes`: int, Maximum number of bytes for LBase. The base level is the level which L0 is compacted into.
@@ -76,6 +74,20 @@ If they are not configured (or assigned their zero-valued), then default values
76
74
> [!TIP]
77
75
> Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: `https://pkg.go.dev/github.com/cockroachdb/[email protected]#Options` (where `A.B.C` is pebble version from Kubo's `go.mod`).
78
76
77
+
Using a pebble datastore can be set when initializing kubo `ipfs init --profile pebbleds`.
78
+
79
+
#### Use of `formatMajorVersion`
80
+
81
+
[Pebble's `FormatMajorVersion`](https://github.com/cockroachdb/pebble/tree/master?tab=readme-ov-file#format-major-versions) is a constant controlling the format of persisted data. Backwards incompatible changes to durable formats are gated behind new format major versions.
82
+
83
+
At any point, a database's format major version may be bumped. However, once a database's format major version is increased, previous versions of Pebble will refuse to open the database.
84
+
85
+
When IPFS is initialized to use the pebbleds datastore (`ipfs init --profile=pebbleds`), the latest pebble database format is configured in the pebble datastore config as `"formatMajorVersion"`. Setting this in the datastore config prevents automatically upgrading to the latest available version when kubo is upgraded. If a later version becomes available, the kubo daemon prints a startup message to indicate this. The user can them update the config to use the latest format when they are certain a downgrade will not be necessary.
86
+
87
+
Without the `"formatMajorVersion"` in the pebble datastore config, the database format is automatically upgraded to the latest version. If this happens, then it is possible a downgrade back to the previous version of kubo will not work if new format is not compatible with the pebble datastore in the previous version of kubo.
88
+
89
+
When installing a new version of kubo when `"formatMajorVersion"` is configured, migration does not upgrade this to the latest available version. This is done because a user may have reasons not to upgrade the pebble database format, and may want to be able to downgrade kubo if something else is not working in the new version. If the configured pebble database format in the old kubo is not supported in the new kubo, then the configured version must be updated and the old kubo run, before installing the new kubo.
90
+
79
91
## badgerds
80
92
81
93
Uses [badger](https://github.com/dgraph-io/badger) as a key value store.
0 commit comments