How to configure pod scraped annotations

[DoyleRavPMPC]

Update: I worked out my issue (see below). See suggestions and >> updates below, hope this helps others.

Suggestions to improve docs:

1. Add a note to the docs near the links to the ama-metrics-settings-configmap.yaml explaining that the file should be renamed before applying the configmap. Perhaps a comment could be added to the file itself in the repo, aka: "apply this file as follows..."
2. Add a note in the troubleshooting section so folks verify the data section. I used the azure portal to browse my cluster config, and noticed this in the YAML view for the config map:

data:
  ama-metrics-settings-configmap.yaml: "kind: 

which needed to be this:

data:
  ama-metrics-settings-configmap: "kind: 

Hello,

We are using the managed prometheus service in our AKS cluster. I would like to add annotations to my pods to configure prometheus scraping. I have questions on how to do that and some follow ups on how to troubleshoot. I've read over the default/custom/troubleshooting topics.

My goal is to have the minimal ingestion targets enabled (cadvisor, kubelet, etc) and in addition, use annotations to indicate which pods in selected namespaces should also be scraped.

Here are the steps I'm following:

I have some nginx pods in the cluster that already have the prometheus.io annotations on them (scrape = true and port set to 10254). They are in namespace ingress-nginx. My deployed pods are in the namespace "dev" and do not have annotations (yet).

I copy the ama-metrics-settings-configmap.yaml in this repo, and I've editted these three fields:

prometheus-collector-settings: |-
cluster_alias = "DevCluster"
...
pod-annotation-based-scraping: |-
podannotationnamespaceregex = "|dev|ingress-nginx"
...
debug-mode: |-
enabled = true

I have ensured I do not have a configmap named ama-metrics-prometheus-config set (see Q2) and there is no other configmaps with the prefix ama-metrics. I am assuming these settings are all I need, and that I would use the prometheus config for static scrape config targets.

I upload the settings as follows:

Updated (IMPORTANT): I saw this tip in another topic and realized the file name is significant here - ensure your settings file is named ama-metrics-settings-configmap
If you copied the file from the repo, it will have a yaml extension, and that extension will be used for more than just locating the source file in the command. Here is an example of how to apply the file correctly:

copy ama-metrics-settings-configmap.yaml ama-metrics-settings-configmap
kubectl create configmap ama-metrics-settings --from-file=ama-metrics-settings-configmap -n kube-system

(original content with follow ups):
I don't see a pod restart, which seems odd. I don't see errors in logs, but I don't see obvious new activity on merging the config (as I would if I loaded a prometheus configmap). If I portforward 9090 to see Prometheus on my ama-metrics and ama-metrics-node pods, I see Configuration reload unsuccessful - that doesn't seem good. If I delete the configmap, I do not see the expected list of the minimal ingestion targets reappear in Prometheus as scrape targets. I only see kube-state-metrics on ama-metrics, and the four other default kube-system targets on ama-metrics-node.

Q1: any obvious issues with my configmap name, settings or commands above?

It was the yaml extension on the file name

Q2: Do I need to additionally add a job in prometheus configmap for service discovery? I wasn't clear if setting the podannotationsnamespace was enough (as long as annotations are present on pods in the namespace).

Yes on adding a job to prometheus-config and creating that configmap. here's an example of my prometheus-config in case it helps others (note my settings file specifies namespaces for pod annotation based scraping)

kubectl create configmap ama-metrics-prometheus-config --from-file=prometheus-config -n kube-system
contents of prometheus-config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: kubernetes_pods
    scheme: http
    kubernetes_sd_configs:
        - role: pod

Q3: Do you have a simple "hello world" example of pod based annotation scraping in managed prometheus?

This would still be useful to others for the docs, but I am past this issue

Q4: Can I expect to be able to combine pod annotation scraping with the scraping of the minimal ingestion kube-system targets? What configmaps are needed for that to work as expected?

This does appear to work, once I got the settings applied without the file extension. Note that if you use port forwarding to see th targets, you'll see kube-state scraping in the ama-metrics pod and you'll see the other "minimal ingestion" targets (cadvisor, etc) in the ama-metrics-node pods.

Troubleshooting questions:

1. Which logs should I look at for the load/parsing of the ama-metrics-settings-configmap? I'd like to troubleshoot the "configuration load unsuccessful" status reported in prometheus, I assume that indicates an issue.

I saw more info in the logs once I fixed the create configmap command for the settings file

5. Is there a way to validate the ama-metrics-settings-configmap after editting, similar to the promconfigvalidator process?

6. Should I see a pod restart when I create or delete the ama metrics settings configmap? (I do not see any restarts in kube-system, which seems odd). Should I be forcing nodes to restart after creating or deleting settings (and if so - which nodes)?

Perhaps the "no pod restart" was a key clue on the settings not being recognized

Thanks in advance,
Doyle

[vishiy]

Tracked by issue #613 that got resolved.

[Charlotte-br560]

Hi Doyle,

Glad to hear you worked out the issue! Your suggestions for improving the documentation sound helpful. If others face a similar problem, your detailed steps and updates might provide valuable insights. Feel free to share any additional findings or questions you may have.

Cheers!