Building Kubernetes Operators - CustomResourceDefinitions and Controllers

by Bala

How CRDs extend the Kubernetes API and how controllers implement the operator pattern, with a real CRD and reconcile-loop example

Introduction

A CustomResourceDefinition (CRD) extends the Kubernetes API with your own resource types - kubectl get postgresqldatabases becomes a real command, backed by etcd storage, validation, and kubectl support, the same as any built-in resource. A CRD alone is just a schema, though; the operator pattern pairs it with a controller that watches instances of the custom resource and reconciles real-world state to match what’s declared - the same control-loop idea that drives every built-in controller (a Deployment’s controller reconciling ReplicaSets, for instance), applied to a domain you define.

Defining a CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqldatabases.db.example.com
spec:
  group: db.example.com
  names:
    kind: PostgreSQLDatabase
    plural: postgresqldatabases
    singular: postgresqldatabase
    shortNames: ["pgdb"]
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["databaseName", "storageSize"]
              properties:
                databaseName:
                  type: string
                  pattern: "^[a-z][a-z0-9_]*$"
                storageSize:
                  type: string
                  pattern: "^[0-9]+Gi$"
                replicas:
                  type: integer
                  minimum: 1
                  default: 1
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ["Pending", "Provisioning", "Ready", "Failed"]
                connectionSecret:
                  type: string
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Phase
          type: string
          jsonPath: .status.phase
        - name: Storage
          type: string
          jsonPath: .spec.storageSize
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

apiextensions.k8s.io/v1 is the only supported CRD API version - v1beta1 was removed in Kubernetes 1.22, so a CRD manifest written against the older version fails outright on any current cluster.

subresources.status: {} splits status into its own subresource, meaning kubectl edit/apply against the main resource can’t accidentally overwrite status fields the controller manages, and RBAC can grant write access to spec without also granting write access to status - the pattern every built-in resource with a status field (Deployments, Pods) already uses. additionalPrinterColumns is what makes kubectl get postgresqldatabases show useful columns instead of just name and age.

An Instance of the Custom Resource

apiVersion: db.example.com/v1
kind: PostgreSQLDatabase
metadata:
  name: app-db
  namespace: production
spec:
  databaseName: appdb
  storageSize: 20Gi
  replicas: 2

At this point, kubectl apply -f app-db.yaml succeeds and kubectl get pgdb shows the object - but nothing actually happens beyond storing it, since a CRD alone has no behavior. That’s the controller’s job.

The Reconcile Loop

A controller for this CRD, written with controller-runtime (the library both Kubebuilder and the Operator SDK are built on) in Go, follows the same shape every Kubernetes controller follows - watch for changes, then reconcile observed state toward desired state:

func (r *PostgreSQLDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var db dbv1.PostgreSQLDatabase
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Desired state: a StatefulSet and Service for this database
	desired := buildStatefulSet(&db)
	var existing appsv1.StatefulSet
	err := r.Get(ctx, req.NamespacedName, &existing)

	switch {
	case apierrors.IsNotFound(err):
		if err := r.Create(ctx, desired); err != nil {
			return ctrl.Result{}, err
		}
	case err != nil:
		return ctrl.Result{}, err
	case !reflect.DeepEqual(existing.Spec, desired.Spec):
		existing.Spec = desired.Spec
		if err := r.Update(ctx, &existing); err != nil {
			return ctrl.Result{}, err
		}
	}

	db.Status.Phase = "Ready"
	if err := r.Status().Update(ctx, &db); err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Every reconcile call is expected to be idempotent - it might run because the PostgreSQLDatabase changed, because the StatefulSet it owns changed, or just because the periodic resync fired, and it has to produce the same correct end state regardless of which triggered it. RequeueAfter: 5 * time.Minute schedules a periodic re-check even with no changes, catching drift (someone manually editing the StatefulSet) that the watch-based triggers alone wouldn’t.

Scaffolding with Kubebuilder

Writing the CRD YAML, Go types, RBAC manifests, and controller boilerplate by hand is real but repetitive work - Kubebuilder generates all of it from a single command and keeps the CRD YAML in sync with your Go type definitions via code generation:

kubebuilder init --domain example.com --repo github.com/example/postgres-operator
kubebuilder create api --group db --version v1 --kind PostgreSQLDatabase
# Edit the generated api/v1/postgresqldatabase_types.go, then:
make manifests   # regenerates the CRD YAML from your Go types
make install     # applies the CRD to your current kubeconfig context
make run         # runs the controller locally against the cluster

Best Practices

  1. Split status into a subresource (subresources.status: {}) on every CRD with a status field - it’s the difference between a controller’s status updates and a user’s spec edits ever racing each other.
  2. Make reconcile idempotent and safe to call repeatedly - never assume it only runs once per actual change; design for “what if this runs twice in a row with nothing different.”
  3. Version your CRD’s API deliberately (v1alpha1 → v1beta1 → v1) as its schema stabilizes, the same maturity signal Kubernetes itself uses for its own APIs - don’t ship v1 on day one if the schema is still likely to change in breaking ways.
  4. Set RBAC scoped to exactly what the controller needs - a controller with cluster-admin because “it’s easier” defeats the purpose of Kubernetes’ own RBAC model for anything it touches.

Conclusion

The operator pattern is how most complex, stateful systems get Kubernetes-native lifecycle management (the Prometheus Operator and ArgoCD on this site are both real-world examples) - the CRD defines what users declare, the controller’s reconcile loop defines how the cluster actually gets there.

Additional Resources