Part 2: Building a Full CI Pipeline with GitHub Actions, Terraform Cloud, and Automated Imports

If Part 1 introduced the Terraform tooling in this repo, Part 2 puts it into production. We’ll build a full CI pipeline that:

• uses GitHub Actions as the orchestrator
• stores state and variables in Terraform Cloud
• and performs automated imports for pre-existing resources

This walkthrough is written as a practical playbook: copy, adapt, and ship.

---

1) Pipeline Architecture at a Glance

Here’s the flow you’re building:

1. Pre-flight checks: lint, format, validate.
2. Terraform Cloud bootstrap: ensure variables and workspace settings are up to date.
3. Import stage (optional but powerful): auto-import existing resources into state.
4. Plan stage: a speculative plan or run in Terraform Cloud.
5. Approval gate: manual or policy-based.
6. Apply: safe, tracked, and auditable.

In production pipelines, each stage should be idempotent, minimal-permission, and observable.

---

2) Prerequisites (Tokens, Org, and Workspace Settings)

You need:

• Terraform Cloud org and workspace
• Terraform Cloud API token
• GitHub Actions secrets
• A Terraform backend (Terraform Cloud handles state, so no S3/GCS backend needed here)

Export locally to test scripts:

export TERRAFORM_TOKEN="..."
export TERRAFORM_ORGANIZATION="my-org"
export TERRAFORM_WORKSPACE="my-workspace"

This repo’s scripts let you enumerate and validate state:

terraform/terraform_cloud_organizations.sh
terraform/terraform_cloud_workspaces.sh
terraform/terraform_cloud_workspace_vars.sh <workspace_id>

---

3) Terraform Cloud Workspace Wiring (Automation-First)

Before CI starts, wire your workspace so that GitHub Actions can fully drive it.

3.1 Set environment variables in Terraform Cloud

If you store secrets in Terraform Cloud (recommended), set them via the workspace helper:

terraform/terraform_cloud_workspace_set_vars.sh <workspace_id> \
  AWS_ACCESS_KEY_ID=... \
  AWS_SECRET_ACCESS_KEY=... \
  TF_LOG=ERROR

You can also set Terraform variables instead of env vars:

export TERRAFORM_VARIABLES=1
terraform/terraform_cloud_workspace_set_vars.sh <workspace_id> region=us-east-1

3.2 Manage shared org variables with varsets

If you have shared credentials or org-wide settings:

terraform/terraform_cloud_varsets.sh
terraform/terraform_cloud_varset_set_vars.sh <varset_id> LOG_LEVEL=info

3.3 CI trust: use Terraform Cloud for state and run execution

In Terraform Cloud, set:

• Execution mode: remote
• Terraform version: pinned
• Workspace VCS: GitHub repo (optional, if you prefer TFC-driven runs)

For GitHub Actions-driven runs, you’ll typically use CLI-driven workflow but still store state in Terraform Cloud.

---

4) GitHub Actions Pipeline: Structure and Stages

Here’s a complete workflow example. It includes linting, a Terraform Cloud plan, optional import automation, and apply with manual approval. Adjust names as needed.

name: terraform-ci

on:
  pull_request:
  push:
    branches: [ main ]

env:
  TF_IN_AUTOMATION: "true"
  TF_INPUT: "false"
  TERRAFORM_ORGANIZATION: my-org
  TERRAFORM_WORKSPACE: my-workspace

jobs:
  lint-and-validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Terraform
        run: install/install_terraform.sh
      - name: Terraform fmt
        run: terraform fmt -check -recursive
      - name: Terraform validate
        run: terraform validate

  plan:
    runs-on: ubuntu-latest
    needs: lint-and-validate
    steps:
      - uses: actions/checkout@v4
      - name: Install Terraform
        run: install/install_terraform.sh
      - name: Terraform login (Terraform Cloud)
        env:
          TERRAFORM_TOKEN: ${{ secrets.TERRAFORM_TOKEN }}
        run: |
          mkdir -p ~/.terraform.d
          cat <<EOF > ~/.terraform.d/credentials.tfrc.json
          {
            "credentials": {
              "app.terraform.io": {
                "token": "${TERRAFORM_TOKEN}"
              }
            }
          }
          EOF
      - name: Terraform init
        run: terraform init
      - name: Terraform plan
        run: terraform plan -out=tfplan

  import-existing:
    runs-on: ubuntu-latest
    needs: plan
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Install Terraform
        run: install/install_terraform.sh
      - name: Terraform login (Terraform Cloud)
        env:
          TERRAFORM_TOKEN: ${{ secrets.TERRAFORM_TOKEN }}
        run: |
          mkdir -p ~/.terraform.d
          cat <<EOF > ~/.terraform.d/credentials.tfrc.json
          {
            "credentials": {
              "app.terraform.io": {
                "token": "${TERRAFORM_TOKEN}"
              }
            }
          }
          EOF
      - name: Terraform init
        run: terraform init
      - name: Import missing resources
        run: |
          export TERRAFORM_PRINT_ONLY=0
          terraform/terraform_import_foreach.sh github_repository

  apply:
    runs-on: ubuntu-latest
    needs: [plan, import-existing]
    if: github.ref == 'refs/heads/main'
    environment:
      name: production
    steps:
      - uses: actions/checkout@v4
      - name: Install Terraform
        run: install/install_terraform.sh
      - name: Terraform login (Terraform Cloud)
        env:
          TERRAFORM_TOKEN: ${{ secrets.TERRAFORM_TOKEN }}
        run: |
          mkdir -p ~/.terraform.d
          cat <<EOF > ~/.terraform.d/credentials.tfrc.json
          {
            "credentials": {
              "app.terraform.io": {
                "token": "${TERRAFORM_TOKEN}"
              }
            }
          }
          EOF
      - name: Terraform init
        run: terraform init
      - name: Terraform apply
        run: terraform apply -auto-approve tfplan

What this pipeline does

• Lint & validate: standard hygiene before any plan.
• Plan: remote state in Terraform Cloud.
• Import: optional stage that pulls existing resources into state.
• Apply: protected by GitHub environment approvals.

If you want full Terraform Cloud-driven runs, you can skip local

plan/apply

and trigger runs in TFC via

terraform_cloud_api.sh

. But the above gives you end-to-end control from GitHub Actions while still using Terraform Cloud as the state + execution backend.

---

5) Automated Imports: Make Drift Disappear

When onboarding existing resources, import is the biggest friction point. This repo’s tooling makes it pipeline-friendly.

5.1 Import based on for_each plan output

Use:

terraform/terraform_import_foreach.sh github_repository

This parses missing resources and issues the correct

terraform import

commands for any

for_each

-generated resources.

5.2 Provider-specific helpers

For common services:

terraform/terraform_import_aws_iam_users.sh
terraform/terraform_import_aws_iam_groups.sh
terraform/terraform_import_github_repos.sh
terraform/terraform_import_github_teams.sh

Tip: Run imports in a separate job to avoid mixing plan/apply output. That makes rollback easier and prevents partial state drift in a failed run.

---

6) Policy & Approval Gates

You have two options:

1. GitHub environment approvals (simple and visible)
2. Terraform Cloud policies (Sentinel or OPA for enterprise-grade guardrails)

For GitHub approvals, use:

environment:
  name: production

For Terraform Cloud policies, evaluate policies before apply via TFC-run-based workflows or use policy checks in TFC workspaces.

---

7) Secrets & Security Posture

Recommended setup:

• Store provider creds in Terraform Cloud workspace vars.
• Store Terraform Cloud token in GitHub Actions secrets.
• Keep import scripts in CI logs minimal; avoid echoing secrets.

CI environment hardening:

• Use least-privilege provider credentials.
• Split read-only plan workflows from apply permissions.
• Use Terraform Cloud variable sets to avoid repeating secrets across workspaces.

---

8) Observability & Diagnostics

If your pipeline fails:

• Terraform Cloud run logs show remote execution details.
• GitHub Actions logs show local command output.

Useful helpers:

terraform/terraform_cloud_workspaces.sh
terraform/terraform_cloud_workspace_vars.sh <workspace_id>
terraform/terraform_cloud_api.sh /runs

---

9) A Production-Grade Flow (Summary Checklist)

✅ CI-ready Terraform code
✅ Terraform Cloud configured
✅ Workspace variables wired
✅ Import automation ready
✅ Plan + apply gated
✅ Logging and auditability

---

Final Thoughts

This pipeline structure gives you a repeatable, auditable Terraform CI flow using GitHub Actions, Terraform Cloud, and automated imports. It’s designed to scale: you can add more workspaces, more repos, and more provider imports without rethinking the core workflow.

If you want, the next addition can cover multi-environment promotion, drift detection schedules, or policy-as-code enforcement in Terraform Cloud.

Credits

Special thanks to Hari Sekhon https://github.com/HariSekhon for creating and maintaining the repository, and to all contributors for their valuable contributions.