Prometheus Exporters in Kubernetes

This guide covers the setup and configuration of various Prometheus exporters in Kubernetes, enabling comprehensive monitoring of different system components and applications.

What are Prometheus Exporters?

Prometheus exporters are applications that collect metrics from various systems and expose them in a format that Prometheus can scrape. They act as bridges between Prometheus and systems that don’t expose metrics in Prometheus format natively.

Common Exporters

1. Node Exporter

Collects hardware and OS metrics.

# node-exporter.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        ports:
        - containerPort: 9100
          protocol: TCP
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

2. kube-state-metrics

Generates metrics about Kubernetes objects.

# kube-state-metrics.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0
        ports:
        - containerPort: 8080
          name: metrics
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

3. Blackbox Exporter

Probes endpoints over HTTP, HTTPS, DNS, TCP, and ICMP.

# blackbox-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blackbox-exporter
  template:
    metadata:
      labels:
        app: blackbox-exporter
    spec:
      containers:
      - name: blackbox-exporter
        image: prom/blackbox-exporter:v0.24.0
        args:
        - --config.file=/config/blackbox.yml
        ports:
        - containerPort: 9115
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: blackbox-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-config
  namespace: monitoring
data:
  blackbox.yml: |
    modules:
      http_2xx:
        prober: http
        timeout: 5s
        http:
          preferred_ip_protocol: "ip4"
      http_post_2xx:
        prober: http
        timeout: 5s
        http:
          method: POST
      tcp_connect:
        prober: tcp
        timeout: 5s
      icmp:
        prober: icmp
        timeout: 5s

4. MySQL Exporter

Collects MySQL server metrics.

# mysql-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-exporter
  template:
    metadata:
      labels:
        app: mysql-exporter
    spec:
      containers:
      - name: mysql-exporter
        image: prom/mysqld-exporter:v0.15.0
        env:
        - name: DATA_SOURCE_NAME
          valueFrom:
            secretKeyRef:
              name: mysql-credentials
              key: data-source-name
        ports:
        - containerPort: 9104
          name: metrics
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-credentials
  namespace: monitoring
type: Opaque
data:
  data-source-name: <base64-encoded-connection-string>

5. Redis Exporter

Monitors Redis instances.

# redis-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-exporter
  template:
    metadata:
      labels:
        app: redis-exporter
    spec:
      containers:
      - name: redis-exporter
        image: oliver006/redis_exporter:v1.55.0
        env:
        - name: REDIS_ADDR
          value: "redis:6379"
        ports:
        - containerPort: 9121
          name: metrics

6. MongoDB Exporter

# mongodb-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb-exporter
  template:
    metadata:
      labels:
        app: mongodb-exporter
    spec:
      containers:
      - name: mongodb-exporter
        image: percona/mongodb_exporter:0.40
        env:
        - name: MONGODB_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-credentials
              key: uri
        ports:
        - containerPort: 9216
          name: metrics

7. Elasticsearch Exporter

# elasticsearch-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch-exporter
  template:
    metadata:
      labels:
        app: elasticsearch-exporter
    spec:
      containers:
      - name: elasticsearch-exporter
        image: justwatch/elasticsearch_exporter:1.5.0
        args:
        - '--es.uri=http://elasticsearch:9200'
        - '--es.all'
        - '--es.indices'
        - '--es.timeout=30s'
        ports:
        - containerPort: 9114
          name: metrics

8. RabbitMQ Exporter

# rabbitmq-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq-exporter
  template:
    metadata:
      labels:
        app: rabbitmq-exporter
    spec:
      containers:
      - name: rabbitmq-exporter
        image: kbudde/rabbitmq-exporter:v1.0.0-RC9
        env:
        - name: RABBIT_URL
          value: "http://rabbitmq:15672"
        - name: RABBIT_USER
          valueFrom:
            secretKeyRef:
              name: rabbitmq-credentials
              key: username
        - name: RABBIT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: rabbitmq-credentials
              key: password
        ports:
        - containerPort: 9419
          name: metrics

9. Kafka Exporter

# kafka-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-exporter
  template:
    metadata:
      labels:
        app: kafka-exporter
    spec:
      containers:
      - name: kafka-exporter
        image: danielqsj/kafka-exporter:v1.7.0
        args:
        - '--kafka.server=kafka:9092'
        - '--kafka.version=2.8.0'
        ports:
        - containerPort: 9308
          name: metrics

10. PostgreSQL Exporter

# postgres-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-exporter
  template:
    metadata:
      labels:
        app: postgres-exporter
    spec:
      containers:
      - name: postgres-exporter
        image: wrouesnel/postgres_exporter:v0.10.1
        env:
        - name: DATA_SOURCE_NAME
          valueFrom:
            secretKeyRef:
              name: postgres-credentials
              key: data-source-name
        ports:
        - containerPort: 9187
          name: metrics

11. NGINX Exporter

# nginx-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-exporter
  template:
    metadata:
      labels:
        app: nginx-exporter
    spec:
      containers:
      - name: nginx-exporter
        image: nginx/nginx-prometheus-exporter:0.11.0
        args:
        - '-nginx.scrape-uri=http://nginx:8080/stub_status'
        ports:
        - containerPort: 9113
          name: metrics

12. HAProxy Exporter

# haproxy-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: haproxy-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: haproxy-exporter
  template:
    metadata:
      labels:
        app: haproxy-exporter
    spec:
      containers:
      - name: haproxy-exporter
        image: prom/haproxy-exporter:v0.13.0
        args:
        - '--haproxy.scrape-uri=http://haproxy:8404/stats;csv'
        ports:
        - containerPort: 9101
          name: metrics

13. JMX Exporter

# jmx-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jmx-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jmx-exporter
  template:
    metadata:
      labels:
        app: jmx-exporter
    spec:
      containers:
      - name: jmx-exporter
        image: bitnami/jmx-exporter:0.17.2
        args:
        - '--config.file=/etc/jmx-exporter/config.yaml'
        ports:
        - containerPort: 5556
          name: metrics
        volumeMounts:
        - name: config
          mountPath: /etc/jmx-exporter
      volumes:
      - name: config
        configMap:
          name: jmx-exporter-config

14. Windows Exporter

# windows-exporter.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: windows-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: windows-exporter
  template:
    metadata:
      labels:
        app: windows-exporter
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      containers:
      - name: windows-exporter
        image: prom/windows-exporter:v0.20.0
        args:
        - '--collectors.enabled=cpu,cs,logical_disk,memory,net,os,service,system'
        ports:
        - containerPort: 9182
          name: metrics

15. Memcached Exporter

# memcached-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memcached-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memcached-exporter
  template:
    metadata:
      labels:
        app: memcached-exporter
    spec:
      containers:
      - name: memcached-exporter
        image: prom/memcached-exporter:v0.10.0
        args:
        - '--memcached.address=memcached:11211'
        ports:
        - containerPort: 9150
          name: metrics

ServiceMonitor Configurations

1. Node Exporter ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  endpoints:
  - port: metrics
    interval: 15s

2. Blackbox Exporter ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blackbox
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: blackbox-exporter
  endpoints:
  - port: http
    interval: 30s
    scrapeTimeout: 10s
    metricRelabelings:
    - sourceLabels: [__name__]
      regex: probe_(dns|duration|http|ssl|success).*
      action: keep

Custom Exporters

1. Application Metrics Exporter

# app_exporter.py
from prometheus_client import start_http_server, Counter, Gauge
import time

# Create metrics
requests_total = Counter('app_requests_total', 'Total requests processed')
request_duration = Gauge('app_request_duration_seconds', 'Request duration')

# Start server
start_http_server(8000)

# Example metric collection
def process_request():
    start = time.time()
    # Process request
    duration = time.time() - start
    requests_total.inc()
    request_duration.set(duration)

2. Custom Exporter Deployment

# custom-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-exporter
  template:
    metadata:
      labels:
        app: custom-exporter
    spec:
      containers:
      - name: custom-exporter
        image: custom-exporter:latest
        ports:
        - containerPort: 8000
          name: metrics

Best Practices

1. Resource Management

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

2. Security Configuration

securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  readOnlyRootFilesystem: true

3. Health Checks

livenessProbe:
  httpGet:
    path: /health
    port: metrics
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health
    port: metrics
  initialDelaySeconds: 5
  periodSeconds: 10

Troubleshooting

Common Issues and Solutions

Exporter Not Scraping

# Check if exporter is running
kubectl get pods -n monitoring

# Check exporter logs
kubectl logs -n monitoring <exporter-pod>

# Verify ServiceMonitor
kubectl get servicemonitor -n monitoring

Metric Collection Issues

# Check metrics endpoint
kubectl port-forward -n monitoring <exporter-pod> 9090:9090
curl localhost:9090/metrics

# Check Prometheus targets
kubectl port-forward -n monitoring prometheus-0 9090:9090
# Visit http://localhost:9090/targets

Prometheus Exporters in Kubernetes

Prometheus Exporters in Kubernetes

What are Prometheus Exporters?

Common Exporters

1. Node Exporter

2. kube-state-metrics

3. Blackbox Exporter

4. MySQL Exporter

5. Redis Exporter

6. MongoDB Exporter

7. Elasticsearch Exporter

8. RabbitMQ Exporter

9. Kafka Exporter

10. PostgreSQL Exporter

11. NGINX Exporter

12. HAProxy Exporter

13. JMX Exporter

14. Windows Exporter

15. Memcached Exporter

ServiceMonitor Configurations

1. Node Exporter ServiceMonitor

2. Blackbox Exporter ServiceMonitor

Custom Exporters

1. Application Metrics Exporter

2. Custom Exporter Deployment

Best Practices

1. Resource Management

2. Security Configuration

3. Health Checks

Troubleshooting

Common Issues and Solutions

Additional Resources