etcd: leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk

[paulcsiki]

Configuration:
Proxmox on a physical server
6 VMs (3 CPs, 3 WNs)
VMs have disks on a ZFS pool comprised of 2 SSDs
CP VMs config file:

agent: 1
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
ide2: local:iso/talos-amd64-1.8.3-bb0ba48a52352c699781aeeb4aa1983b80ccc778c2eac94590fe6b4ab3c0fd00.iso,media=cdrom,size=118660K
memory: 8192
meta: creation-qemu=9.0.2,ctime=1732998340
name: tc1-cp2
net0: virtio=BC:24:11:CC:3E:81,bridge=vmbr0,firewall=1,tag=10
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-101-disk-0,discard=on,iothread=1,size=50G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=7882459c-15c3-46b7-9c2d-804e91b759de
sockets: 1
startup: order=1
vmgenid: dcb4b9d8-641e-4e2b-a842-cc8cdbd90986
args: -cpu kvm64,+cx16,+lahf_lm,+popcnt,+sse3,+ssse3,+sse4.1,+sse4.2

Problem:
etcd health checks keep failing and leader randomly changes multiple times during the day. Inside the logs I can see:
leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk
etcd logs from all CPs: etcd2.log

I don't see any IO delays on the Proxmox cluster and a ZFS pool of 2x SATA SSDs should be able to handle this use case, no?

[paulcsiki]

It seems to have been the disks. My SSDs have high latency.