TCP/TLS performance issue

[snarbles]

Any way to improve Event Per Second (EPS) using TCP/TLS? We seem to be throttled around 40k EPS which is less than our UDP which is about 80k EPS:

  source_palo_tls:
    type: syslog
    address: 0.0.0.0:10001
    mode: tcp
    receive_buffer_bytes: 268435456
    max_length: 409600
    tls:
      enabled: true
      crt_file: /etc/vector/cert/logs.pem
      key_file: /etc/vector/cert/logs.key
      ca_file: /etc/vector/cert/ca_chain.pem
      # if sending system presents a cert, set to true
      verify_certificate: false
      verify_hostname: false

We also set these settings in sysctl.d:

net.core.rmem_default = 268435456
net.core.wmem_default = 268435456
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.core.netdev_max_backlog = 2000

net.ipv4.tcp_rmem = 8192 2097152 268435456
net.ipv4.tcp_wmem = 4096 524288 268435456

[snarbles]

Currently we have this running and it looks like vector is having trouble keeping up with the Recv-Q from a high volume host when running "ss -t -a -n":
ESTAB 1560785438 0 10.X.X.X:10001 10.X.X.X:42499

[zhoufragon]

More comments on the issue.

ss -t -m -n|grep 175 -A1

ESTAB 1605607548 0 10.xxx.xxx.xxx:10001 10.xxx.xxx.xxx:37677
skmem:(r2147365632,rb2147483646,t0,tb2097152,f3328,w0,o0,bl0,d15066209)

This indicates that the received buffer has almost been fully used.

netstat -s

...
4454052 packets pruned from receive queue because of socket buffer overrun
399344 packets dropped from out-of-order queue because of socket buffer overrun
8206943950 packets collapsed in receive queue due to low socket buffer
truncated

Both outputs indicate that the vector's capability of handling data is far behind data received.

Notes that during these time, the CPU and memory and I/O are pretty good (util% < 5%)

which means that vector is hang on something, or waiting for something , it is not fully using OS resource.

[snarbles]

Performance begines to degrade after 22-35 TCP connections.

[snarbles]

Upgraded to .47 and performance decreased.

[snarbles]

This is related to:
#23230

[marnao]

just a thought here as a workaround (obviously not ideal).. could you set up multiple sources on different ports and load balance across them?

[snarbles]

Yes, we are doing that and trying multiple instances of vector on the same server.