metricCustomizers / attributesAsLabels + extraMetrics: Is hard to create a metric with a custom name and type

[polarnik]

Environment

Java 21
Development version (main branch)

docker-files

Dockerfile

FROM maven:3.9.9-eclipse-temurin-21-jammy

RUN apt-get update && apt-get install -y git
RUN git clone https://github.com/prometheus/jmx_exporter.git /tmp/jmx_exporter
WORKDIR /tmp/jmx_exporter
RUN mvn -DskipTests=true package
RUN mkdir /opt/app/
RUN cp /tmp/jmx_exporter/jmx_prometheus_javaagent/target/jmx_prometheus_javaagent-1.1.0-post.jar /opt/app/jmx_prometheus_javaagent-1.1.0-post.jar

FROM jetbrains/youtrack:2025.1.62967

ENV OTEL_JAVA_AGENT_VERSION=2.13.1
ENV JOLOKIA_JVM_AGENT_VERSION=2.2.1
ENV OTEL_JAVAAGENT_EXTENSIONS=/opt/app/opentelemetry-javaagent.jar

USER root

RUN mkdir /opt/app/
COPY --from=0 /opt/app/jmx_prometheus_javaagent-1.1.0-post.jar /opt/app/jmx_prometheus_javaagent-1.1.0-post.jar

RUN /bin/sh -c "yum -y install curl hostname"

ADD https://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-agent-jvm/$JOLOKIA_JVM_AGENT_VERSION/jolokia-agent-jvm-$JOLOKIA_JVM_AGENT_VERSION-javaagent.jar /opt/app/jolokia-agent-jvm-javaagent.jar
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$OTEL_JAVA_AGENT_VERSION/opentelemetry-javaagent.jar $OTEL_JAVAAGENT_EXTENSIONS
RUN chown -R jetbrains:jetbrains /opt/app/
USER jetbrains

Docker-compose:

services:
  youtrack:
    #image: jetbrains/youtrack:${YOUTRACK_VERSION:?}
    build:
      context: .
      dockerfile: docker/jetbrains-youtrack/youtrack-with-tools.Dockerfile
    hostname: youtrack
    restart: "no"
    ports:
      - '9404:9404'
      - '9010:9010'
      - '9011:9011'
      - "8080:8080"
    tmpfs:
      /tmp
    volumes:
      - ./volumes/youtrack_data:/opt/youtrack/data
      - ./volumes/youtrack_conf:/opt/youtrack/conf
      - ./volumes/youtrack_logs:/opt/youtrack/logs
      - ./volumes/youtrack_backups:/opt/youtrack/backups
      - ./volumes/youtrack_temp:/opt/youtrack/temp
      - "./configs/youtrack/youtrack.jvmoptions:/opt/youtrack/conf/youtrack.jvmoptions:ro"
      - "./configs/prometheus_jmx_exporter/jmx_exporter.yml:/opt/app/jmx_exporter.yml:ro"

youtrack.jvmoptions:

-XX:NativeMemoryTracking=summary
-XX:+PrintFlagsFinal
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9011
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1

-javaagent:/opt/app/jmx_prometheus_javaagent-1.1.0-post.jar=youtrack:9404:/opt/app/jmx_exporter.yml

Steps

1. Start a docker-compose
2. Copy a wizard_token from the docker logs
3. Open http://127.0.0.1:8080/
4. Add the wizard_token from the logs: http://127.0.0.1:8080/?wizard_token=pk191NZmHm8GGTOfpdVy
5. Start creating a new instance
6. Set a domain:

• http://127.0.0.1:8080

7. Set a password and tokens

• admin
• admin

8. Wait a few seconds
9. Install visualVM https://visualvm.github.io/
10. Install a visualVM plugin: Menu / Tools / Plugins / Available Plugins / VisualVM MBean
11. Connect from the VisualVM tool to the remote host: 127.0.0.1:9010
12. Open an MBean with numeric and String attributes, for example jetbrains.youtrack:type=Hub,name=HubIntegration

Description

I have an MBean with 2 string Attributes Build and Version and without any other attributes. I would like to convert two string attributes (Build and Version) into labels and get one prometheus metric with two labels.

My expected result is:

# HELP youtrack_version Version info jetbrains.youtrack:name=Global,type=Maintenance,attribute=info
# TYPE youtrack_version gauge
youtrack_version{Build="62967",Version="2025.1"} 1.0

I can get this output via the config:

startDelaySeconds: 10
lowercaseOutputName: false
lowercaseOutputLabelNames: false

whitelistObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

metricCustomizers:
  - mbeanFilter:
      domain: jetbrains.youtrack
      properties:
        type: Maintenance
        name: Global
    attributesAsLabels:
      - Build
      - Version
    extraMetrics:
      - name: info
        value: 1
        description: Version info

rules:
  - pattern: 'jetbrains.youtrack<type=Maintenance, name=Global>'
    name: youtrack_version
    type: GAUGE

We can use

includeObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

instead of or with

whitelistObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

but other combinations don't work clear

General problem

Documentation for the feature metricCustomizers has only one example.

Expected result — the documentation will have more examples

Example 2: A metric without type

The class extraMetrics doesn't have a type info, it has only a value. We will get an untyped metric

# HELP youtrack_version Version info jetbrains.youtrack:name=Global,type=Maintenance,attribute=info
# TYPE youtrack_version untyped
youtrack_version{Build="62967",Version="2025.1"} 1.0

with this config:

startDelaySeconds: 10
lowercaseOutputName: false
lowercaseOutputLabelNames: false

includeObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

metricCustomizers:
  - mbeanFilter:
      domain: jetbrains.youtrack
      properties:
        type: Maintenance
        name: Global
    attributesAsLabels:
      - Build
      - Version
    extraMetrics:
      - name: info
        value: 1
        description: Version info

whitelistObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

rules:
  - pattern: 'jetbrains.youtrack<type=Maintenance, name=Global><>'
    name: youtrack_version

Expected result — the section extraMetrics will have the type attribute

Example 2: We can move a value field from extraMetrics to rules, but we will get several metrics with different labels _objectname

It looks like this:

# TYPE youtrack_version gauge
youtrack_version{Build="62967",Version="2025.1",_objectname="jetbrains.youtrack<type=Maintenance, name=Global><>Build"} 1.0
youtrack_version{Build="62967",Version="2025.1",_objectname="jetbrains.youtrack<type=Maintenance, name=Global><>Version"} 1.0

via a config:

includeObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

metricCustomizers:
  - mbeanFilter:
      domain: jetbrains.youtrack
      properties:
        type: Maintenance
        name: Global
    attributesAsLabels:
      - Build
      - Version

rules:
  - pattern: 'jetbrains.youtrack<type=Maintenance, name=Global><>(\w+): (.+)'
    name: youtrack_version
    value: 1.0
    type: GAUGE
    help: Application version

Expected result — we will get one metric instead of a list of metrics. It may a wrong expected result

Example 3. Duplicates

An Exception occurred while scraping metrics: java.lang.IllegalArgumentException: Build: duplicate label name
	at io.prometheus.metrics.model.snapshots.Labels.validateNames(Labels.java:172)
	at io.prometheus.metrics.model.snapshots.Labels.sortAndValidate(Labels.java:162)
	at io.prometheus.metrics.model.snapshots.Labels.of(Labels.java:92)
	at io.prometheus.jmx.MatchedRuleToMetricSnapshotsConverter.isLabelsUnique(MatchedRuleToMetricSnapshotsConverter.java:167)
	at io.prometheus.jmx.MatchedRuleToMetricSnapshotsConverter.convertRulesWithSameName(MatchedRuleToMetricSnapshotsConverter.java:85)
	at io.prometheus.jmx.MatchedRuleToMetricSnapshotsConverter.convert(MatchedRuleToMetricSnapshotsConverter.java:79)
	at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:889)
	at io.prometheus.metrics.model.registry.MultiCollector.collect(MultiCollector.java:21)
	at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:84)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:135)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:56)
	at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:33)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:101)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:871)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:847)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

with a config

startDelaySeconds: 10
lowercaseOutputName: false
lowercaseOutputLabelNames: false

includeObjectNames:
  - "jetbrains.youtrack:type=Maintenance,name=Global"

metricCustomizers:
  - mbeanFilter:
      domain: jetbrains.youtrack
      properties:
        type: Maintenance
        name: Global
    attributesAsLabels:
      - Build
      - Version
    extraMetrics:
      - name: info
        value: true
        description: This is a boolean value indicating if the scenario is still active or is completed.

rules:
  - pattern: 'jetbrains.youtrack<type=Maintenance, name=Global><>(\w+): (.+)'
    name: youtrack_version_info
    labels:
      $1: $2
    value: 1.0
    type: GAUGE
    help: Application version info

Because the rule creates labels and the attributesAsLabels creates labels.

[dhoard]

@polarnik MetricCustomizers is a new feature that is very targeted and not expected to be commonly used.

Since it's so new, I consider it alpha. It's subject to change.

• More documentation is always better
• PRs are welcome.

Example 2 Issue - The metric type

Since you don't have a rule, the default type is untype.

Example 2 Issue - Several metrics with an _objectName label

A rule pattern that matches multiple MBeans and maps to a metric with the same name and labels would violate the supported metrics exposition formats. The _objectName label is added to the metric with the MBean object name value to create the required uniqueness.

The rule pattern you provided matches more than one MBean and tries to map to the same metric name + labels.

 - pattern: 'jetbrains.youtrack<type=Maintenance, name=Global><>(\w+): (.+)'

Example 3 Issue - java.lang.IllegalArgumentException: Build: duplicate label name

Have you tried to match Build and use a metric customizer to map in the version attribute?