feat(inputs.statsd): Improve performance #17872

radykal-com · 2025-10-22T06:44:14Z

Summary

Avoid using the mutex in each metric name parse by passing each worker goroutine its own graphite.Parser.

Checklist

No AI generated code was used in this PR

Related issues

resolves #12313

Benchmark

There is a big difference in the overall throughput when using multiple threads

Old code

BenchmarkParser-10                               	  305120	      3939 ns/op
BenchmarkUDP-10                                  	       1	2014095375 ns/op
BenchmarkUDPThreads4-10                          	     128	   8014808 ns/op
BenchmarkUDPThreads8-10                          	     126	   7963413 ns/op
BenchmarkUDPThreads16-10                         	     123	   8457920 ns/op
BenchmarkTCP-10                                  	       1	1583591084 ns/op
BenchmarkParse-10                                	   29188	     41003 ns/op
BenchmarkParseWithTemplate-10                    	   32236	     36990 ns/op
BenchmarkParseWithTemplateAndFilter-10           	   31557	     37952 ns/op
BenchmarkParseWith2TemplatesAndFilter-10         	   31798	     37983 ns/op
BenchmarkParseWith2Templates3TagsAndFilter-10    	   25666	     46949 ns/op
PASS

New code

BenchmarkParser-10                               	  324426	      3748 ns/op
BenchmarkUDP-10                                  	       1	1545554250 ns/op
BenchmarkUDPThreads4-10                          	     235	   4815471 ns/op
BenchmarkUDPThreads8-10                          	     247	   4646433 ns/op
BenchmarkUDPThreads16-10                         	     220	   4568423 ns/op
BenchmarkTCP-10                                  	       1	1590451042 ns/op
BenchmarkParse-10                                	   31209	     38400 ns/op
BenchmarkParseWithTemplate-10                    	   31294	     38442 ns/op
BenchmarkParseWithTemplateAndFilter-10           	   30872	     38969 ns/op
BenchmarkParseWith2TemplatesAndFilter-10         	   30714	     39148 ns/op
BenchmarkParseWith2Templates3TagsAndFilter-10    	   22514	     53415 ns/op
PASS

telegraf-tiger · 2025-10-22T06:44:20Z

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

radykal-com · 2025-10-22T06:48:56Z

!signed-cla

Hipska · 2025-10-24T15:36:58Z

plugins/inputs/statsd/statsd_test.go

+	err := parser.Init()
+	require.NoError(b, err)


Same for all other occurrences

Suggested change

err := parser.Init()

require.NoError(b, err)

require.NoError(b, parser.Init())

Hipska · 2025-10-24T15:39:24Z

plugins/inputs/statsd/statsd.go

+	p := &graphite.Parser{Separator: s.MetricSeparator, Templates: s.Templates}
+	err := p.Init()
+	if err != nil {
+		return err
+	}


Maybe extract in a new func which returns *graphite.Parser, error?

I created it and used it also in tests

Perfect, that was the idea.

Hipska · 2025-10-24T21:12:12Z

plugins/inputs/statsd/statsd.go


+// newGraphiteParser initializes and returns a new graphite.Parser. graphite.Parser returned is not safe to be used in
+// multiple goroutines.
+func newGraphiteParser(s *Statsd) (*graphite.Parser, error) {


What about this?

Suggested change

func newGraphiteParser(s *Statsd) (*graphite.Parser, error) {

func (s *Statsd) newGraphiteParser() (*graphite.Parser, error) {

@Hipska one of the pipelines failed but I think it's not related to the changes

telegraf-tiger · 2025-10-25T06:53:42Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB	. RPM	. TAR . GZ	. ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

srebhan

Thanks for your contribution @radykal-com!

I think your code is racy for the case where another message is coming in via s.in in the parser function while we are currently using the parser. More specifically when setting p.DefaultTags in parseName().
If a second call (for another s.in message) arrives at that line before (or even worse while) the first call is executing the templates you will get mangled tags in your template result. We had this issue before and therefore introduced the lock.

What you could do though is to use a sync.Pool or parsers to be able to handle multiple messages in parallel while reusing the parser!

radykal-com · 2025-10-27T10:33:09Z

Thanks for your contribution @radykal-com!

I think your code is racy for the case where another message is coming in via s.in in the parser function while we are currently using the parser. More specifically when setting p.DefaultTags in parseName(). If a second call (for another s.in message) arrives at that line before (or even worse while) the first call is executing the templates you will get mangled tags in your template result. We had this issue before and therefore introduced the lock.

What you could do though is to use a sync.Pool or parsers to be able to handle multiple messages in parallel while reusing the parser!

That cannot happen because each parser call is in it's own goroutine (each goroutine with its own parser struct) and no further goroutines are created inside. What I mean is that every message received from s.in is executed in a single goroutine, so a second read cannot happen until the previous one is completely processed. What is running in parallel is that there are N goroutines running the parser() function, so there are N receiving from the s.in that's way there is a parser per goroutine created before the read loop in the parser method.

I'm pretty sure its working because the code is running in our high traffic metrics aggregator for the past week and is working fine (no panic) and the tests just pass OK with the -race flag

radykal-com · 2025-10-29T09:43:04Z

@Hipska any chance to rerun the pipeline to get the failing one pass?

srebhan

Thanks for your explanation @radykal-com! I agree with your assessment.

skartikey

@radykal-com Nice work! Thanks for the contribution!

create a graphite.Parser per worker goroutine to avoid the mutex

369418d

radykal-com changed the title ~~Fix issue 12313 bad statsD input performance~~ feat(inputs.statsd): Improve performance Oct 22, 2025

telegraf-tiger bot added area/statsd feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Oct 22, 2025

sort imports

12ef00d

radykal-com mentioned this pull request Oct 22, 2025

inputs.statsd - Increasing number of UDP pending messages dropped #12313

Closed

Hipska suggested changes Oct 24, 2025

View reviewed changes

extract initialization to new function

8513884

Hipska reviewed Oct 24, 2025

View reviewed changes

change function to method

cac689f

srebhan requested changes Oct 27, 2025

View reviewed changes

srebhan self-assigned this Oct 27, 2025

Hipska approved these changes Oct 27, 2025

View reviewed changes

radykal-com requested a review from srebhan October 27, 2025 11:41

srebhan approved these changes Oct 29, 2025

View reviewed changes

srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Oct 29, 2025

srebhan assigned skartikey and mstrandboge and unassigned srebhan Oct 29, 2025

skartikey unassigned mstrandboge Oct 29, 2025

skartikey approved these changes Oct 30, 2025

View reviewed changes

skartikey merged commit 6cc1353 into influxdata:master Oct 30, 2025
26 of 27 checks passed

github-actions bot added this to the v1.37.0 milestone Oct 30, 2025

	err := parser.Init()
	require.NoError(b, err)
	require.NoError(b, parser.Init())

	func newGraphiteParser(s Statsd) (graphite.Parser, error) {
	func (s Statsd) newGraphiteParser() (graphite.Parser, error) {

feat(inputs.statsd): Improve performance #17872

feat(inputs.statsd): Improve performance #17872

Conversation

radykal-com commented Oct 22, 2025

Summary

Checklist

Related issues

Benchmark

Old code

New code

Uh oh!

telegraf-tiger bot commented Oct 22, 2025

Uh oh!

radykal-com commented Oct 22, 2025

Uh oh!

Hipska Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Hipska Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

radykal-com Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Hipska Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Hipska Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

radykal-com Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

radykal-com Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

telegraf-tiger bot commented Oct 25, 2025

Artifact URLs

Uh oh!

srebhan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radykal-com commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radykal-com commented Oct 29, 2025

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

skartikey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

srebhan left a comment •

edited

Loading

radykal-com commented Oct 27, 2025 •

edited

Loading