Configure Custom Metric Labels

This page shows you how to configure custom metric labels that promote Kubernetes labels or annotations on ClusterQueues, LocalQueues, and Cohorts into additional Prometheus metric label dimensions.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

• A Kubernetes cluster is running.
• The kubectl command-line tool has communication with your cluster.
• Kueue is installed.
• Prometheus is set up to scrape Kueue metrics.

Configure custom labels

Custom metric labels require the CustomMetricLabels feature gate, which is alpha and disabled by default. See Installation for details on how to enable feature gates.

Add a metrics.customLabels section to the Kueue controller configuration. Each entry defines one extra Prometheus label dimension on ClusterQueue, LocalQueue, and Cohort metrics.

apiVersion: config.kueue.x-k8s.io/v1beta2
kind: Configuration
featureGates:
  CustomMetricLabels: true
metrics:
  customLabels:
    - name: team
    - name: cost_center
      sourceLabelKey: "billing/cost-center"
    - name: budget
      sourceAnnotationKey: "billing.example.com/budget"

Field reference

A maximum of 8 custom labels can be configured.

How values are resolved

For each ClusterQueue, LocalQueue, or Cohort, Kueue reads the value of the specified Kubernetes label or annotation. If the object does not have the label/annotation, an empty string is used as the value.

Example: label by team

Using the configuration above with name: team:

1. Create a ClusterQueue with a team label:

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: engineering-cq
     labels:
       team: platform
   spec:
     resourceGroups:
       - coveredResources: ["cpu"]
         flavors:
           - name: default
             resources:
               - name: cpu
                 nominalQuota: 10
   

2. Query metrics filtered by team:

   kueue_pending_workloads{custom_team="platform"}
   kueue_admitted_workloads_total{custom_team="platform"}
   

What’s next

• See Prometheus Metrics for the full list of Kueue metrics.
• See Common Grafana Queries for PromQL queries to monitor Kueue in Grafana.