Skip to content

Commit a05f52a

Browse files
authored
add multiprocessing caveats to spectator-py usage (#236)
Add section clarifying how to use the library with Python multiprocessing.
1 parent e6c97ea commit a05f52a

File tree

1 file changed

+30
-9
lines changed

1 file changed

+30
-9
lines changed

docs/spectator/lang/py/usage.md

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,36 @@ your job does not run this long, or you find you are missing metrics that were r
315315
of your job run, then add a five-second sleep before exiting: `time.sleep(5)`. This will allow time
316316
for the metrics to be sent.
317317

318+
## Design Considerations: High Volume Metrics
319+
320+
This client is stateless, and sends a UDP packet (or unixgram) to `spectatord` each time a meter is
321+
updated. If you are performing high-volume operations, on the order of tens-of-thousands or millions
322+
of operations per second, then you should pre-aggregate your metrics and report them at a cadence
323+
closer to the `spectatord` publish interval of 5 seconds. This will keep the CPU usage related to
324+
`spectator-py` and `spectatord` low (around 1% or less), as compared to up to 40% for high-volume
325+
scenarios.
326+
327+
## Caveats: multiprocessing
328+
329+
This library makes use of `threading.Lock()` to provide lazy acquisition of the socket that communicates
330+
with `spectatord`, and creates a background thread for flushing metrics periodically, if the Line Buffer
331+
is enabled. Therefore, if you want to use this library with `multiprocessing`, you have to make a design
332+
choice about how to use `spectator-py`, to ensure that you do not encounter deadlocks in your code.
333+
334+
* If you use a start method of `fork` or `forkserver` (Linux default), then you need to ensure that a new
335+
`Registry` is created after a process is started, which requires a few extra lines of code.
336+
* If you use the `spawn` start method (macOS and Windows default), then your code can use a "shared" instance
337+
of the `Registry` as written.
338+
339+
In both cases, the overhead cost will be maintaining a separate socket for communicating with `spectatord`, plus
340+
the memory associated with processing metrics for delivery.
341+
342+
See [multiprocessing - Contexts and start methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods)
343+
for more details from the official Python documentation.
344+
345+
See [fork-is-problematic](https://github.com/thatch/fork-is-problematic) for a detailed explanation of the
346+
challenges associated with forking.
347+
318348
## Debug Metrics Delivery to `spectatord`
319349

320350
In order to see debug log messages from `spectatord`, create an `/etc/default/spectatord` file with
@@ -327,15 +357,6 @@ SPECTATORD_OPTIONS="--verbose"
327357
This will report all metrics that are sent to the Atlas backend in the `spectatord` logs, which will
328358
provide an opportunity to correlate metrics publishing events from your client code.
329359

330-
## Design Considerations - Reporting Intervals
331-
332-
This client is stateless, and sends a UDP packet (or unixgram) to `spectatord` each time a meter is
333-
updated. If you are performing high-volume operations, on the order of tens-of-thousands or millions
334-
of operations per second, then you should pre-aggregate your metrics and report them at a cadence
335-
closer to the `spectatord` publish interval of 5 seconds. This will keep the CPU usage related to
336-
`spectator-py` and `spectatord` low (around 1% or less), as compared to up to 40% for high-volume
337-
scenarios.
338-
339360
## Writing Tests
340361

341362
To write tests against this library, instantiate an instance of the `Registry` and provide a `Config`

0 commit comments

Comments
 (0)