The Complete Terraform
Deep Dive

🖱️ Manual Provisioning (ClickOps)

Engineers would log into the AWS Console, Azure Portal, or physical data-center tools and click through wizards to create servers, networks, and databases. This approach had catastrophic failures at scale:

• No reproducibility: "Works on my region" — no one could recreate the same environment twice. Staging drifted from prod within weeks.
• Zero audit trail: Who opened port 22 to 0.0.0.0/0 in production? The console didn't tell you.
• Human error at scale: One misclick in a production firewall could trigger a three-hour outage. In 2013, a misconfigured security group at a major US exchange caused a 45-minute trading halt.
• No version control: Infrastructure had no Git history, no rollback, no peer review. You couldn't git revert a broken datacenter.
• Knowledge silos: Only "the guy who built it" knew how it worked. When he left, the company lost institutional memory entirely.

📜 Imperative Scripting (Bash / Python / PowerShell)

The natural evolution was automation scripts. Teams wrote Bash wrappers around AWS CLI commands or Python using boto2. These solved reproducibility but introduced a new class of failures:

• Idempotency failures: aws ec2 create-vpc would succeed the first time and crash the second with "VPC already exists." Scripts had to manually check current state before every action — a logic nightmare.
• State blindness: Scripts had no memory. They didn't know what existed in the cloud. Every run was a guess.
• Order dependency: Creating a subnet before the VPC exists causes an error. Engineers had to manually encode the correct order of 50+ API calls.
• Error handling hell: A half-run script that crashed after creating 3 of 10 resources left the environment in an unknown partial state — the worst possible scenario.
• No parallelism: Scripts ran sequentially. Deploying 20 independent S3 buckets took 20× as long as deploying 1.
• Provider lock-in: An AWS Bash script was 100% AWS-specific. A multi-cloud strategy meant maintaining separate toolchains.

⚙️ Imperative Model ("How")

You describe the steps to achieve a goal. The execution engine follows your instructions literally.

# You must check IF it exists first
if ! aws ec2 describe-vpcs \
    --filters "Name=tag:Name,Values=my-vpc" \
    | grep -q VpcId; then
  aws ec2 create-vpc \
    --cidr-block 10.0.0.0/16
fi
# Then create the subnet — manually ordered
aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.1.0/24

Problems: manual state checks, order management, error handling, no parallelism.

📋 Declarative Model ("What")

You describe the desired end state. The engine computes how to get there — and what to change if reality differs.

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "public" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.0.1.0/24"
}

Terraform infers order, parallelizes independent resources, and checks current state automatically.

📁 Terraform Workspaces

Workspaces allow multiple state files within the same backend configuration. Each workspace maps to a separate terraform.tfstate object (e.g., env:/dev/terraform.tfstate).

terraform workspace new dev
terraform workspace new staging
terraform workspace select prod

# Reference in config:

locals {
  env_config = {
    dev     = { instance_type = "t3.small"  }
    prod    = { instance_type = "m5.4xlarge" }
  }
  cfg = local.env_config[terraform.workspace]
}

Limitations

• All environments share the same code — no isolation of backend config
• Cannot have different providers per workspace easily
• Easy to accidentally apply to wrong workspace (dangerous)
• Best for simple, ephemeral environments (feature branches, testing)

🌲 Terragrunt (Recommended for Teams)

Terragrunt is a thin wrapper around Terraform that adds DRY configuration, remote state management, and cross-module dependencies. Created by Gruntwork.

infrastructure/
├── _base/
│   └── vpc/          # Reusable Terraform module
├── dev/
│   └── vpc/
│       └── terragrunt.hcl
├── staging/
│   └── vpc/
│       └── terragrunt.hcl
└── prod/
    └── vpc/
        └── terragrunt.hcl

terraform {
  source = "../../_base/vpc"
}

remote_state {
  backend = "s3"
  config  = {
    bucket = "my-prod-tfstate"
    key    = "vpc/terraform.tfstate"
  }
}

inputs = {
  env          = "prod"
  cidr_block   = "10.0.0.0/16"
  az_count     = 3
}

Terragrunt advantages: separate backend per environment, run-all apply across stacks, dependency blocks between modules, DRY backend configuration via root.hcl.

🔑 HashiCorp Vault

Best for: Dynamic secrets, database credentials, PKI, SSH signing. Vault generates short-lived credentials on demand — no static secrets in config.

data "vault_generic_secret" "db" {
  path = "secret/prod/db"
}

resource "aws_db_instance" "db" {
  password = data.
    vault_generic_secret.
    db.data["password"]
}

🏭 AWS KMS + SSM / Secrets Manager

Best for: AWS-native secrets. Store encrypted values in Secrets Manager, reference via Terraform data source. KMS encrypts state at rest.

data "aws_secretsmanager_secret_version"
    "db_pass" {
  secret_id = "prod/db/password"
}

locals {
  db_pass = jsondecode(
    data.aws_secretsmanager
    _secret_version.db_pass
    .secret_string)["password"]
}

💻 Environment Variables

Best for: CI/CD pipeline credentials injected at runtime. Never store secrets in .tf files or commit them to git. Use TF_VAR_ prefix.

# CI/CD pipeline injects these
export TF_VAR_db_password="$SECRET"
export AWS_ACCESS_KEY_ID="$KEY"
export AWS_SECRET_ACCESS_KEY="$SID"

# NEVER do this in .tf files:
# password = "hardcoded123" ← DANGER

# Always mark sensitive variables

🔄 Trap 1: Cycle Errors

Symptom: Error: Cycle: aws_security_group.a, aws_security_group.b

Occurs when resource A references resource B which references resource A — creating a circular dependency in the DAG. Terraform cannot determine which to create first.

Common Causes

• Two security groups that reference each other's ID in their ingress rules
• A module that outputs its own input
• Misuse of depends_on creating an indirect cycle

The Fix

# WRONG: creates a cycle
resource "aws_security_group" "a" {
  ingress {
    security_groups = [aws_security_group.b.id]
  }
}

# FIX: use aws_security_group_rule
resource "aws_security_group" "a" {}
resource "aws_security_group" "b" {}
resource "aws_security_group_rule" "a_to_b" {
  type                     = "ingress"
  source_security_group_id = aws_security_group.a.id
  security_group_id        = aws_security_group.b.id
}

📉 Trap 2: State Drift

Symptom: Resources exist in the cloud but not in state, or vice versa. Plan shows unexpected changes.

Occurs when humans manually modify cloud resources (console changes, emergency hotfixes) without updating Terraform configuration. This is the most dangerous operational failure mode.

Diagnosis & Remediation

# 1. Detect drift (refresh from real APIs)
terraform plan -refresh-only

# 2. If resource exists in cloud but not state:
terraform import aws_s3_bucket.my_bucket my-bucket-name

# 3. If resource in state but deleted in cloud:
terraform state rm aws_s3_bucket.my_bucket

# 4. If resource config drifted:
terraform apply -refresh-only # accept real state

# 5. View raw state for investigation:
terraform state list
terraform state show aws_s3_bucket.my_bucket

🔒 Trap 3: State Lock Stuck

Symptom: Error acquiring the state lock: ConditionalCheckFailedException

Occurs when a previous Terraform process was killed mid-apply (Ctrl+C, CI timeout, network loss) and failed to release the DynamoDB lock. The state is safe but no one can apply.

# Get lock ID from error message, then:
terraform force-unlock <LOCK_ID>

# Or directly in DynamoDB (emergency):
aws dynamodb delete-item \
  --table-name terraform-state-locks \
  --key '{"LockID": {"S": "path/to/state"}}'

# ⚠️ Verify no other process is running first!

🔄 Trap 4: Resource Replace on Change

Symptom: Plan shows -/+ destroy and then create replacement for an in-place change you expected.

Some resource attributes are "ForceNew" — changing them requires destroying the old resource and creating a new one (e.g., changing an EC2 instance's AMI ID or an RDS engine version requires replacement). This can cause production downtime.

resource "aws_instance" "web" {
  ami           = var.ami_id
  instance_type = "t3.medium"

  # Create new instance before destroying old
  lifecycle {
    create_before_destroy = true
  }
}

Use create_before_destroy = true for zero-downtime replacements. Combine with a load balancer to drain traffic from the old instance before destruction.