Issues with DCGM Nvidia Dashboard

[gaby]

@chaitanya-sistla Just ran into the new blog showcases how to run DCGM to get metrics from Nvidia. Blog https://openobserve.ai/blog/how-to-monitor-nvidia-gpu/

Importing this same dashboard results in unusable metrics. The DCGM Exporter exports series in upper case while the dashboard is somehow using lowercase.

Changing the case makes visualizations appear and show data, but trying to edit them results in Y-Axis errors (could be related to v0.16.0).

The graphs are not very friendly for multi-GPU systems. For example, my test host has 8 GPUs.

Note:

• I'm using DCGM Exporter Container and Prometheus Container. I'm not using the Otel collector.

Related openobserve/openobserve#9008

[chaitanya-sistla]

@gaby thanks for getting this to my attention. You are right about the metrics to be in upper case but I believe this could be the version where I tested. I will do the same for stable versions and get back to you or update this repo by fixing the dashboard.

For multi-GPU - I have used this for multi-GPU initially but not like a cluster pool of GPU machines. I will test these scenarios though and get back to you! Also I guess you are writing to openobserve directly via prometheus write in this scenario which should not be any issue as otel here is doing exactly the same.

I will get back to you as soon as possible.

[gaby]

@chaitanya-sistla Yes, I'm using the latest release from DCGM Exporter: https://github.com/NVIDIA/dcgm-exporter/releases/tag/4.4.1-4.6.0

Another weird thing I notice, when I go to Metrics on the left menu, these series have 0 fields on them, but they still work. I don't know if you ran into that while making the dashboards.

I tested this on a bare metal server with 1 GPU, and also in one with 4 GPUs.

dcgm-exporter:

docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.4.1-4.6.0-ubuntu22.04

prometheus.yaml:

global:
  scrape_interval: 15s
  evaluation_interval: 30s

scrape_configs:
  - job_name: dcgm-exporter
    static_configs:
      - targets:
          - 'localhost:9400/metrics'

Several of the graphs work correctly for multiple GPUs, but they are not showing metrics for all 4, only for 2.

[gaby]

@chaitanya-sistla If you could integrate vLLM as a separate dashboard for AI, it would be awesome.

Related #29