Managing Cloud Pub/Sub with Terraform

In this guide, we’ll explore how to manage Google Cloud Pub/Sub using Terraform.

Video Tutorial

Prerequisites

• Google Cloud SDK installed and configured
• Terraform installed (version 1.0.0 or later)
• A GCP project with billing enabled

Project Structure

.
├── main.tf                   # Main Terraform configuration file
├── variables.tf              # Variable definitions
├── outputs.tf               # Output definitions
├── terraform.tfvars         # Variable values
└── modules/
    └── pubsub/
        ├── main.tf          # Cloud Pub/Sub specific configurations
        ├── variables.tf      # Module variables
        ├── topics.tf        # Topic configurations
        └── outputs.tf       # Module outputs

Provider Configuration

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

Variables

variable "project_id" {
  description = "The ID of the GCP project"
  type        = string
}

variable "region" {
  description = "The region to deploy resources to"
  type        = string
  default     = "us-central1"
}

variable "topic_name" {
  description = "Name of the Pub/Sub topic"
  type        = string
}

variable "subscription_name" {
  description = "Name of the Pub/Sub subscription"
  type        = string
}

Topic Configuration

resource "google_pubsub_topic" "topic" {
  name = var.topic_name

  labels = {
    environment = "production"
  }

  message_retention_duration = "86600s"  # 24 hours

  schema_settings {
    schema = "NONE"
    encoding = "JSON"
  }
}

Subscription Configuration

resource "google_pubsub_subscription" "subscription" {
  name  = var.subscription_name
  topic = google_pubsub_topic.topic.name

  labels = {
    environment = "production"
  }

  message_retention_duration = "604800s"  # 7 days
  retain_acked_messages      = true

  ack_deadline_seconds = 20

  expiration_policy {
    ttl = "2592000s"  # 30 days
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"  # 10 minutes
  }

  enable_message_ordering    = false
  enable_exactly_once_delivery = true

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter.id
    max_delivery_attempts = 5
  }
}

resource "google_pubsub_topic" "dead_letter" {
  name = "${var.topic_name}-dead-letter"
}

IAM Configuration

resource "google_pubsub_topic_iam_binding" "topic_publisher" {
  topic   = google_pubsub_topic.topic.name
  role    = "roles/pubsub.publisher"
  members = ["serviceAccount:publisher@${var.project_id}.iam.gserviceaccount.com"]
}

resource "google_pubsub_subscription_iam_binding" "subscription_subscriber" {
  subscription = google_pubsub_subscription.subscription.name
  role         = "roles/pubsub.subscriber"
  members      = ["serviceAccount:subscriber@${var.project_id}.iam.gserviceaccount.com"]
}

Outputs

output "topic_id" {
  value       = google_pubsub_topic.topic.id
  description = "The ID of the Pub/Sub topic"
}

output "subscription_id" {
  value       = google_pubsub_subscription.subscription.id
  description = "The ID of the Pub/Sub subscription"
}

output "dead_letter_topic_id" {
  value       = google_pubsub_topic.dead_letter.id
  description = "The ID of the dead letter topic"
}

Best Practices

1. Message Handling:

   • Implement proper error handling
   • Use dead letter queues
   • Configure retry policies
   • Monitor message flow

2. Security:

   • Use appropriate IAM roles
   • Implement encryption
   • Monitor access patterns
   • Regular security audits

3. Performance:

   • Configure appropriate timeouts
   • Set proper message sizes
   • Monitor throughput
   • Optimize subscriptions

4. Cost Optimization:

   • Monitor message volume
   • Clean up unused resources
   • Configure retention policies
   • Use appropriate pricing tiers

Common Operations

Creating Resources

terraform init
terraform plan
terraform apply

Publishing Messages

gcloud pubsub topics publish $TOPIC_ID --message="Hello World"

Pulling Messages

gcloud pubsub subscriptions pull $SUBSCRIPTION_ID --auto-ack

Best Practices and Tips

1. Topic Management:

   • Use meaningful names
   • Implement proper organization
   • Monitor message flow
   • Regular cleanup

2. Subscription Management:

   • Configure proper timeouts
   • Implement error handling
   • Monitor performance
   • Use dead letter queues

3. Operations:

   • Regular monitoring
   • Set up alerts
   • Track metrics
   • Maintain documentation

Conclusion

You’ve learned how to set up and manage Google Cloud Pub/Sub using Terraform. This setup provides:

• Reliable message delivery
• Scalable messaging infrastructure
• Best practices implementation
• Easy management and monitoring