cannot get pod from kubelet, err: call to /pods endpoint failed #1743
Description
Describe the bug
cloudwatch-agent keep crashing with the following error using latest version 1.3xxx in AWS EKS v1.32 with Auto-Mode enabled:
Image: cloudwatch-agent:1.300056.0b1123
2025-06-26T14:06:24Z I! {"caller":"host/ec2metadata.go:91","msg":"Fetch instance id and type from ec2 metadata","kind":"receiver","name":"awscontainerinsightreceiver","data_type":"metrics"}
2025-06-26T14:06:24Z E! {"caller":"graph/graph.go:426","msg":"Failed to start component","error":"cannot get pods from kubelet: call to /pods endpoint failed: Get \"https://10.20.20.227:10250/pods\": remote error: tls: internal error","type":"Receiver","id":"awscontainerinsightreceiver"}
2025-06-26T14:06:24Z I! {"caller":"[email protected]/service.go:303","msg":"Starting shutdown..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x215fec0]
goroutine 1 [running]:
github.com/aws/amazon-cloudwatch-agent/internal/k8sCommon/k8sclient.(*SafeChannel).Close(0x4000cfc7b8?)
github.com/aws/amazon-cloudwatch-agent/internal/k8sCommon/k8sclient/kubernetes_utils.go:207 +0x20
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals/internal/resolver.(*kubernetesResolver).Stop(0x4000cfc868?, {0x4000fd1648?, 0x4000cfc7d8?})
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals/internal/resolver/kubernetes.go:239 +0x20
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals/internal/resolver.(*attributesResolver).Stop(0x5423c40?, {0x5488b78, 0x8514ba0})
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals/internal/resolver/attributesresolver.go:85 +0x7c
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals.(*awsapplicationsignalsprocessor).Shutdown(0x4000ec7170, {0x5488b78, 0x8514ba0})
github.com/aws/amazon-cloudwatch-agent/plugins/processors/awsapplicationsignals/processor.go:102 +0x70
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
go.opentelemetry.io/collector/[email protected]/component.go:77
go.opentelemetry.io/collector/service/internal/graph.(*Graph).ShutdownAll(0x40015134a0, {0x5488b78, 0x8514ba0}, {0x543f880, 0x4000e54a80})
go.opentelemetry.io/collector/[email protected]/internal/graph/graph.go:465 +0x180
go.opentelemetry.io/collector/service.(*Service).Shutdown(0x4001040000, {0x5488b78, 0x8514ba0})
go.opentelemetry.io/collector/[email protected]/service.go:309 +0xd0
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0x40010c1d40, {0x5488b78, 0x8514ba0})
go.opentelemetry.io/collector/[email protected]/collector.go:229 +0x518
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0x40010c1d40, {0x5488b78, 0x8514ba0})
go.opentelemetry.io/collector/[email protected]/collector.go:282 +0x3c
go.opentelemetry.io/collector/otelcol.NewCommand.func1(0x4000951b08, {0x4c13507?, 0x7?, 0x4c03d41?})
go.opentelemetry.io/collector/[email protected]/command.go:36 +0x88
github.com/spf13/cobra.(*Command).execute(0x4000951b08, {0x40010c7c10, 0x1, 0x1})
github.com/spf13/[email protected]/command.go:985 +0x830
github.com/spf13/cobra.(*Command).ExecuteC(0x4000951b08)
github.com/spf13/[email protected]/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(0x0?)
github.com/spf13/[email protected]/command.go:1041 +0x1c
main.runAgent({0x5488be8, 0x4000cd8050}, {0x8514ba0, 0x0, 0x0}, {0x8514ba0, 0x0, 0x0})
github.com/aws/amazon-cloudwatch-agent/cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go:395 +0x1178
main.reloadLoop(0x40007f2000, {0x8514ba0, 0x0, 0x0}, {0x8514ba0, 0x0, 0x0}, {0x0?, 0x0?, 0x0?}, ...)
github.com/aws/amazon-cloudwatch-agent/cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go:180 +0x2b8
main.main()
github.com/aws/amazon-cloudwatch-agent/cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go:673 +0x9b4
I! imds retry client will retry 1 times
Steps to reproduce
Install Amazon CloudWatch Observability Add-On using the default configuration.
What did you expect to see?
CW Pods are not to keep crashing.
What version did you use?
cloudwatch-agent:1.300056.0b1123
What config did you use?
Default EKS Add-on configuration for the Amazon CloudWatch Observability
Environment
EKS AutoMode
Additional context
The same issue was reported 2 times but closed as issue resolved in newer version
#1100
#1417