Writing a Custom Prometheus Exporter in Go

by Bala

Building an exporter with client_golang - simple registered metrics vs the Collector interface for expensive or dynamic data

Introduction

The exporters guide on this site covers deploying existing exporters (node_exporter, mysqld_exporter, and similar). This one covers writing your own - the case for it is a system with no existing exporter, or internal application metrics that don’t fit an off-the-shelf tool. Prometheus’s official Go client library, client_golang, is the standard choice regardless of whether the thing being instrumented is itself written in Go.

The Simple Case: Registered Metrics

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "myapp_queue_depth",
		Help: "Current number of items in the processing queue.",
	})
	jobsProcessed = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "myapp_jobs_processed_total",
		Help: "Total jobs processed, labeled by outcome.",
	}, []string{"outcome"})
	jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
		Name:    "myapp_job_duration_seconds",
		Help:    "Job processing duration in seconds.",
		Buckets: prometheus.DefBuckets,
	})
)

func init() {
	prometheus.MustRegister(queueDepth, jobsProcessed, jobDuration)
}

func processJob() {
	timer := prometheus.NewTimer(jobDuration)
	defer timer.ObserveDuration()

	// ... actual work ...
	jobsProcessed.WithLabelValues("success").Inc()
}

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":9100", nil)
}

This pattern - a package-level metric, registered once via MustRegister, mutated wherever the relevant code path runs (.Inc(), .Set(), .Observe()) - covers the overwhelming majority of application instrumentation: metrics that live in memory and are cheap to update inline. promhttp.Handler() serves whatever’s currently registered in the default registry on every scrape; it does no work of its own beyond formatting the already-current values.

_total is a required naming convention (not just a style choice) for counters - Prometheus’s own tooling and community dashboards assume it, and client_golang’s CounterVec won’t stop you from omitting it, but doing so breaks the convention every consumer of the metric expects.

The Collector Interface: Metrics Computed at Scrape Time

Registered metrics work when your code updates them as things happen. Sometimes the data only makes sense to compute when scraped - querying an external system’s current state, for instance, rather than mirroring it into an always-updated gauge. That’s what the Collector interface is for:

type QueueStatsCollector struct {
	db *sql.DB
}

func (c *QueueStatsCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- oldestJobAgeDesc
	ch <- queuesByPriorityDesc
}

var (
	oldestJobAgeDesc = prometheus.NewDesc(
		"myapp_oldest_pending_job_age_seconds",
		"Age of the oldest pending job in seconds.",
		nil, nil,
	)
	queuesByPriorityDesc = prometheus.NewDesc(
		"myapp_pending_jobs",
		"Number of pending jobs by priority.",
		[]string{"priority"}, nil,
	)
)

func (c *QueueStatsCollector) Collect(ch chan<- prometheus.Metric) {
	var oldestAge float64
	c.db.QueryRow("SELECT EXTRACT(EPOCH FROM (now() - min(created_at))) FROM jobs WHERE status = 'pending'").Scan(&oldestAge)
	ch <- prometheus.MustNewConstMetric(oldestJobAgeDesc, prometheus.GaugeValue, oldestAge)

	rows, _ := c.db.Query("SELECT priority, count(*) FROM jobs WHERE status = 'pending' GROUP BY priority")
	defer rows.Close()
	for rows.Next() {
		var priority string
		var count float64
		rows.Scan(&priority, &count)
		ch <- prometheus.MustNewConstMetric(queuesByPriorityDesc, prometheus.GaugeValue, count, priority)
	}
}

func main() {
	prometheus.MustRegister(&QueueStatsCollector{db: db})
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":9100", nil)
}

Collect() runs the database queries on every scrape, not continuously in the background - this is the actual point of the interface: it queries the database’s current state fresh each time Prometheus scrapes, rather than maintaining a separate always-updated in-memory copy that could drift from reality between scrapes. It’s also the right pattern whenever gathering the data is itself the expensive part (an API call, a slow query) and doing it on every scrape interval, rather than continuously, is the acceptable tradeoff.

Making It Scrapable in Kubernetes

apiVersion: v1
kind: Service
metadata:
  name: myapp-exporter
  labels:
    app: myapp
spec:
  ports:
    - name: metrics
      port: 9100
  selector:
    app: myapp
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-exporter
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: metrics
      interval: 30s

With the Prometheus Operator already running, a ServiceMonitor is all that’s needed to get the exporter scraped - no manual edit to Prometheus’s own scrape config.

Best Practices

  1. Follow the naming conventions - a _total suffix on counters, a base unit in the name (_seconds, _bytes, not _ms or _kb), and a Help string describing exactly what the value means. These aren’t cosmetic; dashboards and alerting rules across the ecosystem assume them.
  2. Use the Collector interface for anything expensive or externally-sourced, and simple registered metrics for anything your own code already knows the current value of - don’t build a background goroutine polling a database into a gauge when Collect() computing it on scrape is simpler and avoids the staleness window.
  3. Keep label cardinality bounded - a label with a raw user ID, request ID, or timestamp turns one metric into effectively unbounded metrics, which is a real memory and query-performance cost on Prometheus. Labels should have a small, known set of values (outcome, priority, region), not an open-ended one.
  4. Never let Collect() block indefinitely - a slow or hung external call inside Collect() blocks that scrape (and can time out the whole /metrics endpoint); set a query timeout and skip that metric on failure rather than hanging the entire exporter.

Conclusion

client_golang covers both instrumentation styles a real application needs: cheap, in-process metrics registered once and updated inline, and expensive or externally-sourced metrics computed fresh via the Collector interface on each scrape - picking the right one for each metric is the actual design decision, not just wiring up the library.

Additional Resources