Managing Cloud Scheduler with Terraform

In this guide, we’ll explore how to manage Google Cloud Scheduler using Terraform.

Video Tutorial

Learn more about managing Google Cloud Scheduler with Terraform in this comprehensive video tutorial:

Prerequisites

• Google Cloud SDK installed and configured
• Terraform installed (version 1.0.0 or later)
• A GCP project with billing enabled

Project Structure

.
├── main.tf                   # Main Terraform configuration file
├── variables.tf              # Variable definitions
├── outputs.tf               # Output definitions
├── terraform.tfvars         # Variable values
└── modules/
    └── scheduler/
        ├── main.tf          # Cloud Scheduler specific configurations
        ├── variables.tf      # Module variables
        ├── jobs.tf          # Job configurations
        └── outputs.tf       # Module outputs

Provider Configuration

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

Variables

variable "project_id" {
  description = "The ID of the GCP project"
  type        = string
}

variable "region" {
  description = "The region to deploy resources to"
  type        = string
  default     = "us-central1"
}

HTTP Job Configuration

resource "google_cloud_scheduler_job" "http_job" {
  name             = "http-job"
  description      = "HTTP job to trigger an endpoint"
  schedule         = "*/10 * * * *"  # Every 10 minutes
  time_zone        = "America/New_York"
  attempt_deadline = "320s"

  http_target {
    uri         = "https://example.com/api/endpoint"
    http_method = "POST"
    
    headers = {
      "Content-Type" = "application/json"
    }

    body = base64encode(jsonencode({
      key = "value"
    }))

    oauth_token {
      service_account_email = google_service_account.scheduler_sa.email
    }
  }

  retry_config {
    retry_count          = 3
    max_retry_duration   = "0s"
    min_backoff_duration = "5s"
    max_backoff_duration = "3600s"
    max_doublings       = 5
  }
}

Pub/Sub Job Configuration

resource "google_cloud_scheduler_job" "pubsub_job" {
  name             = "pubsub-job"
  description      = "Job to publish to Pub/Sub"
  schedule         = "0 */1 * * *"  # Every hour
  time_zone        = "UTC"
  attempt_deadline = "60s"

  pubsub_target {
    topic_name = google_pubsub_topic.scheduler_topic.id
    data       = base64encode("Hello World!")
    attributes = {
      source = "scheduler"
    }
  }
}

resource "google_pubsub_topic" "scheduler_topic" {
  name = "scheduler-topic"
}

App Engine Job Configuration

resource "google_cloud_scheduler_job" "appengine_job" {
  name             = "appengine-job"
  description      = "Job to trigger App Engine endpoint"
  schedule         = "0 0 * * *"  # Daily at midnight
  time_zone        = "UTC"
  attempt_deadline = "600s"

  app_engine_http_target {
    http_method = "POST"
    app_engine_routing {
      service  = "default"
      version  = "v1"
      instance = "instances"
    }
    relative_uri = "/tasks/process"
    body         = base64encode("Process daily tasks")
    headers = {
      "Content-Type" = "application/x-www-form-urlencoded"
    }
  }
}

Service Account Configuration

resource "google_service_account" "scheduler_sa" {
  account_id   = "scheduler-sa"
  display_name = "Cloud Scheduler Service Account"
}

resource "google_project_iam_member" "scheduler_sa_roles" {
  for_each = toset([
    "roles/cloudscheduler.jobRunner",
    "roles/pubsub.publisher"
  ])
  
  project = var.project_id
  role    = each.key
  member  = "serviceAccount:${google_service_account.scheduler_sa.email}"
}

OIDC Configuration

resource "google_cloud_scheduler_job" "oidc_job" {
  name             = "oidc-job"
  description      = "Job with OIDC authentication"
  schedule         = "*/15 * * * *"  # Every 15 minutes
  time_zone        = "UTC"
  attempt_deadline = "180s"

  http_target {
    uri         = "https://secure-api.example.com/endpoint"
    http_method = "GET"

    oidc_token {
      service_account_email = google_service_account.scheduler_sa.email
      audience             = "https://secure-api.example.com"
    }
  }
}

Monitoring Configuration

resource "google_monitoring_alert_policy" "job_failures" {
  display_name = "Cloud Scheduler Job Failures"
  combiner     = "OR"

  conditions {
    display_name = "Job Failure Rate"
    condition_threshold {
      filter          = "metric.type=\"cloudscheduler.googleapis.com/job/execution_count\" AND resource.type=\"cloud_scheduler_job\" AND metric.labels.status=\"error\""
      duration        = "300s"
      comparison     = "COMPARISON_GT"
      threshold_value = 5

      trigger {
        count = 1
      }

      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_RATE"
      }
    }
  }

  notification_channels = [google_monitoring_notification_channel.email.name]
}

Outputs

output "http_job_id" {
  value       = google_cloud_scheduler_job.http_job.id
  description = "The ID of the HTTP job"
}

output "pubsub_job_id" {
  value       = google_cloud_scheduler_job.pubsub_job.id
  description = "The ID of the Pub/Sub job"
}

output "service_account_email" {
  value       = google_service_account.scheduler_sa.email
  description = "The email of the scheduler service account"
}

Best Practices

1. Job Configuration:

   • Use retry logic
   • Set deadlines
   • Monitor execution
   • Regular review

2. Security:

   • Use service accounts
   • Limit permissions
   • Regular audits
   • Monitor access

3. Performance:

   • Optimize schedules
   • Monitor latency
   • Track failures
   • Regular testing

4. Cost Optimization:

   • Monitor usage
   • Optimize frequency
   • Clean up unused
   • Regular review

Common Operations

Creating Resources

terraform init
terraform plan
terraform apply

Job Operations

# List jobs
gcloud scheduler jobs list

# Run job
gcloud scheduler jobs run JOB_NAME

# Update job
gcloud scheduler jobs update http JOB_NAME --schedule="0 * * * *"

Best Practices and Tips

1. Job Management:

   • Plan schedules
   • Use retry logic
   • Regular review
   • Monitor execution

2. Security:

   • Use OIDC/OAuth
   • Limit access
   • Regular audits
   • Monitor usage

3. Operations:

   • Monitor metrics
   • Track failures
   • Set up alerts
   • Regular maintenance

Conclusion

You’ve learned how to set up and manage Google Cloud Scheduler using Terraform. This setup provides:

• Automated job scheduling
• Multiple target types
• Secure authentication
• Best practices implementation