update buffers section for spectator-go

Author: copperlight

docs/spectator/lang/go/usage.md

@@ -200,16 +200,91 @@ the environment variable takes precedence over the passed config.
 
 The environment variable `SPECTATOR_OUTPUT_LOCATION` can be set to `none` to disable metrics collection.
 
-## Line Buffer
-
-The `NewConfigWithBuffer` factory function takes a `bufferSize` parameter that configures an optional
-`LineBuffer`, which caches protocol lines locally, before flushing them to `spectatord`. Flushes occur
-under two conditions: (1) the buffer size is exceeded, or (2) five seconds has elapsed. The buffer is
-available for the `UdpWriter` and the `UnixgramWriter`, where performance matters most. The `LineBuffer`
-is disabled by default (with size zero) in the standard `NewConfig` factory function, to ensure that the
-default operation of the library works under most circumstances. For high-performance scenarios, a 60KB
-buffer size is recommended. The maximum buffer size for udp sockets and unix domain sockets on Linux is
-64KB, so stay under this limit.
+## Buffers
+
+Three modes of operation are available, for applications that operate at different scales:
+
+* **Small.** No buffer (size 0 bytes). Write immediately to the socket upon every metric update, up to
+~150K lines/sec, with delays from 1 to 450 us, depending on thread count. No metrics are dropped, due
+to mutex locks.
+* **Medium.** LineBuffer (size <= 65536 bytes), which writes to the socket upon overflow, or upon a
+flush interval, up to ~1M lines/sec, with delays from 0.1 to 32 us, depending on thread count. No metrics
+are dropped. Status metrics are published to monitor usage.
+* **Large.** LowLatencyBuffer (size > 65536 bytes), which writes to the socket on a flush interval, up
+to ~1M lines/sec, with delays from 0.6 to 7 us, depending on thread count. The true minimum size is 2 *
+CPU * 60KB, or 122,880 bytes for 1 CPU. Metrics may be dropped. Status metrics are published to monitor
+usage.
+
+The buffers are available for the UdpWriter and the UnixWriter.
+
+### Line Buffer (bufferSize <= 65536)
+
+This is a single string buffer, protected by a mutex, that offers write performance up to ~1M lines/sec
+(spectatord maximum), with a latency per write ranging from 0.1 to 32 us, depending upon the number of
+threads in use.
+
+Metrics are flushed from the buffer when an overflow occurs, and periodically by a timer, according to
+the flush interval. Thus, if there are periods of time when metric publishing is slow, metrics will still
+be delivered from the buffer on time. Note that the spectatord publish interval is every 5 seconds, which
+is a good starting choice for this configuration. This buffer will block, and it will not drop lines.
+
+The LineBuffer reports two metrics, which can be used to monitor buffer performance:
+
+* `spectator-go.lineBuffer.bytesWritten` - A counter reporting bytes/sec written to spectatord.
+* `spectator-go.lineBuffer.overflows` - A counter reporting overflows/sec, which are flushes before the
+interval.
+
+Example configuration:
+
+```
+config, _ := NewConfigWithBuffer("udp", nil, nil, 61440, 5*time.Second)
+```
+
+### Low Latency Buffer (bufferSize > 65536)
+
+LowLatencyBuffer (bufferSize > 65536), which builds arrays of buffers that are optimized for introducing
+the least amount of latency in highly multithreaded applications that record many metrics. It offers write
+performance up to ~1 M lines/sec (spectatord maximum), with a latency per write ranging from 0.6 to 7 us,
+depending upon the number of threads in use.
+
+This is achieved by spreading data access across a number of different mutexes, and only writing buffers
+from a goroutine that runs periodically, according to the flushInterval. There is a front buffer and a back
+buffer, and these are rotated during the periodic flush. The inactive buffer is flushed, while the active
+buffer continues to receive metric writes from the application. Within each buffer, there are numCPU shards,
+and each buffer shard has N chunks, where a chunk is set to 60KB, to allow the data to fit within the
+spectatord socket buffers with room for one last protocol line. This buffer will not block, and it can drop
+lines, if it overflows.
+
+As a sizing example, if you have an 8 CPU system, and you want to allocate 5 MB to each buffer shard, and
+there are two buffers (front and back), then you need to configure a buffer size of 83,886,080 bytes. Each
+buffer shard will have 85 chunks, each of which is protected by a separate mutex.
+
+```
+2 buffers (front/back) * 8 CPU (shard count) * 5,242,880 bytes/shard *  = 83,886,080 bytes total
+```
+
+Pairing this with a 1-second flush interval will result in a configuration that can handle ~85K lines/sec
+writes to spectatord. Note that the spectatord publish interval is every 5 seconds, so you have some room to
+experiment with different buffer sizes and publish intervals.
+
+While the bufferSize can be set as low as 65537, it will guarantee a minimum size of 2 * CPU * 60KB, to
+ensure that there is always at least 1 chunk per shard. On a system with 1 CPU, this will be 122,880 bytes, 
+and on a system with 4 CPU, this will be 491,520 bytes.
+
+The LowLatencyBuffer reports metrics, which can be used to monitor buffer performance:
+
+* `spectator-go.lowLatencyBuffer.bytesWritten` - A counter reporting bytes/sec written to spectatord.
+* `spectator-go.lowLatencyBuffer.overflows` - A counter reporting overflows/sec, which are drops.
+* `spectator-go.lowLatencyBuffer.pctUsage` - A gauge reporting the percent usage of the buffers.
+
+When using the LowLatencyBuffer, it is recommended to watch the `spectatord.parsedCount` metric, to ensure
+that you have sufficient headroom against the maximum data ingestion rate of ~1M lines/sec for `spectatord`.
+
+Example configuration:
+
+```
+config, _ := NewConfigWithBuffer("udp", nil, nil, 83886080, 1*time.Second)
+```
 
 ## Batch Usage