Skip to content

Commit 7f2178a

Browse files
authored
fix(influxdb3): document exec-mem-pool-bytes usage for data persistence (#6394)
* fix(influxdb3): document exec-mem-pool-bytes usage for data persistence Updates documentation to reflect that exec-mem-pool-bytes is used for both query processing and parquet persistence operations, not just queries. Based on source code analysis showing the memory pool is used by: - Query executor for processing queries - Persister for converting WAL data to Parquet format Changes: - Updated config description to include "data operations" - Added memory usage info to durability docs for Parquet storage - Added troubleshooting section for memory-related write performance - Fixed capitalization of "object store" throughout Addresses DAR #499 Source analysis: influxdb3/src/commands/serve.rs:772-798 Shows separate executors for queries and write path operations, both using memory pools for data processing. * fix(influxdb3): broken links to no-sync
1 parent df06c64 commit 7f2178a

File tree

3 files changed

+48
-4
lines changed

3 files changed

+48
-4
lines changed

content/shared/influxdb3-cli/config-options.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1088,7 +1088,9 @@ Defines the address on which InfluxDB serves HTTP API requests.
10881088

10891089
#### exec-mem-pool-bytes
10901090

1091-
Specifies the size of memory pool used during query execution.
1091+
Specifies the size of the memory pool used for query processing and data operations.
1092+
This memory pool is used when {{% product-name %}} processes queries and performs
1093+
internal data management tasks.
10921094
Can be given as absolute value in bytes or as a percentage of the total available memory--for
10931095
example: `8000000000` or `10%`.
10941096

content/shared/influxdb3-internals-reference/durability.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,14 @@ As written data moves through {{% product-name %}}, it follows a structured path
1919

2020
- **Process**: InfluxDB validates incoming data before accepting it into the system.
2121
- **Impact**: Prevents malformed or unsupported data from entering the database.
22-
- **Details**: The database validates incoming data and stores it in the write buffer (in memory). If [`no_sync=true`](#no-sync-write-option), the server sends a response to acknowledge the write.
22+
- **Details**: The database validates incoming data and stores it in the write buffer (in memory). If `no_sync=true`, the server sends a response to acknowledge the write [without waiting for persistence](/influxdb3/version/reference/cli/influxdb3/write/#write-line-protocol-and-immediately-return-a-response).
2323

2424
### Write-ahead log (WAL) persistence
2525

2626
- **Process**: The database flushes the write buffer to the WAL every second (default).
2727
- **Impact**: Ensures durability by persisting data to object storage.
2828
- **Tradeoff**: More frequent flushing improves durability but increases I/O overhead.
29-
- **Details**: Every second (default), the database flushes the write buffer to the Write-Ahead Log (WAL) for persistence in the Object store. If [`no_sync=false`](#no-sync-write-option) (default), the server sends a response to acknowledge the write.
29+
- **Details**: Every second (default), the database flushes the write buffer to the Write-Ahead Log (WAL) for persistence in the object store. If `no_sync=false` (default), the server sends a response to acknowledge the write.
3030

3131
### Query availability
3232

@@ -40,7 +40,8 @@ As written data moves through {{% product-name %}}, it follows a structured path
4040
- **Process**: Every ten minutes (default), data is persisted to Parquet files in object storage.
4141
- **Impact**: Provides durable, long-term storage.
4242
- **Tradeoff**: More frequent persistence reduces reliance on the WAL but increases I/O costs.
43-
- **Details**: Every ten minutes (default), the {{% product-name %}} persists the oldest data from the queryable buffer to the Object store in Parquet format, and keeps the remaining data (the most recent 5 minutes) in memory.
43+
- **Memory usage**: The persistence process uses memory from the configured memory pool ([`exec-mem-pool-bytes`](/influxdb3/version/reference/config-options/#exec-mem-pool-bytes)) when converting data to Parquet format. For write-heavy workloads, ensure adequate memory is allocated.
44+
- **Details**: Every ten minutes (default), {{% product-name %}} persists the oldest data from the queryable buffer to the object store in Parquet format, and keeps the remaining data (the most recent 5 minutes) in memory.
4445

4546
### In-memory cache
4647

content/shared/influxdb3-write-guides/troubleshoot.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ Learn how to avoid unexpected results and recover from errors when writing to
66
- [Review HTTP status codes](#review-http-status-codes)
77
- [Troubleshoot failures](#troubleshoot-failures)
88
- [Troubleshoot rejected points](#troubleshoot-rejected-points)
9+
{{% show-in "core,enterprise" %}}- [Troubleshoot write performance issues](#troubleshoot-write-performance-issues){{% /show-in %}}
910

1011
## Handle write responses
1112

@@ -65,3 +66,43 @@ InfluxDB rejects points that don't match the schema of existing data.
6566
Check for [field data type](/influxdb3/version/reference/syntax/line-protocol/#data-types-and-format)
6667
differences between the rejected data point and points within the same
6768
database--for example, did you attempt to write `string` data to an `int` field?
69+
70+
{{% show-in "core,enterprise" %}}
71+
72+
## Troubleshoot write performance issues
73+
74+
If you experience slow write performance or timeouts during high-volume ingestion,
75+
consider the following:
76+
77+
### Memory configuration
78+
79+
{{% product-name %}} uses memory for both query processing and internal data operations,
80+
including converting data to Parquet format during persistence.
81+
For write-heavy workloads, insufficient memory allocation can cause performance issues.
82+
83+
**Symptoms of memory-related write issues:**
84+
- Slow write performance during data persistence (typically every 10 minutes)
85+
- Increased response times during high-volume ingestion
86+
- Memory-related errors in server logs
87+
88+
**Solutions:**
89+
- Increase the [`exec-mem-pool-bytes`](/influxdb3/version/reference/config-options/#exec-mem-pool-bytes)
90+
configuration to allocate more memory for data operations.
91+
For write-heavy workloads, consider setting this to 30-40% of available memory.
92+
- Monitor memory usage during peak write periods to identify bottlenecks.
93+
- Adjust the [`gen1-duration`](/influxdb3/version/reference/config-options/#gen1-duration)
94+
to control how frequently data is persisted to Parquet format.
95+
96+
### Example configuration for write-heavy workloads
97+
98+
```bash { placeholders="PERCENTAGE" }
99+
influxdb3 serve \
100+
--exec-mem-pool-bytes PERCENTAGE \
101+
--gen1-duration 15m \
102+
# ... other options
103+
```
104+
105+
Replace {{% code-placeholder-key %}}`PERCENTAGE`{{% /code-placeholder-key %}} with the percentage
106+
of available memory to allocate (for example, `35%` for write-heavy workloads).
107+
108+
{{% /show-in %}}

0 commit comments

Comments
 (0)