Production deploys and infrastructure

Hold terraform apply, kubectl and production deploys until a person says yes, whether they run from CI, a script or a coding agent's hook.

Updated

Agents now run terraform apply, kubectl and deploy scripts, in pipelines and from coding agents on a laptop. One bad change takes production down for everyone, and some of them, such as a destroyed database, cannot be taken back.

The risk

  • A plan that does more than intended. A renamed resource in Terraform becomes a delete and a create. The plan says so; nobody read it.
  • The wrong cluster. A coding agent with a production context in its kubeconfig runs the command it tested in staging.
  • Fixing forward at speed. An agent retrying a failed deploy can make an incident worse faster than a person can.

Name what rules need to tell apart

A rule matches the action name, never the command, so put the environment and the kind of change in the name. A wildcard only works at the end, so the part you want to group by comes first:

  • deploy.staging and deploy.production for deploy scripts.
  • infra.apply.production for a Terraform plan that only adds and changes, infra.destroy.production for one that deletes or replaces anything.
  • k8s.change.production and k8s.delete.production for kubectl. Reads do not ask at all.

The rules

  1. Always allow deploy.staging, for ci-deployerStaging deploys from the pipeline never wait. Only the pipeline's key gets this.
  2. Always ask about deploy.production, for Ana and Ben, anyone after 10 minutesProduction deploys go to the people on call. The wait is shorter than the 15 minutes a request lives by default, so it can escalate.
  3. Always ask about infra.destroy.*, two peopleA Terraform plan that deletes or replaces anything needs two different people. One denial stops it.
  4. Never allow k8s.delete.*, for claude-code-laptopThe coding agent may change a deployment with a yes, but never delete in the cluster. A person runs those.
The same rules as API bodies
# POST /v1/workspaces/{workspace_id}/rules, one call per rule
{
  "action": "deploy.staging",
  "agent_key_id": "CI_DEPLOYER_KEY_ID",
  "effect": "allow"
}

{
  "action": "deploy.production",
  "deciders": [
    "USER_ID_OF_ANA",
    "USER_ID_OF_BEN"
  ],
  "effect": "ask",
  "escalate_after_seconds": 600
}

{
  "action": "infra.destroy.*",
  "approvals": 2,
  "effect": "ask"
}

{
  "action": "k8s.delete.*",
  "agent_key_id": "CLAUDE_CODE_KEY_ID",
  "effect": "deny"
}

Owners write rules under Rules in the app, or post these bodies while signed in. Replace the ids in capitals with your own.

Everything else, such as infra.apply.production or k8s.change.production, waits for any member, as it would with no rules at all.

Scripts, pipelines and ops agents

Wrap the command that changes production, so it asks whoever runs it: a pipeline, a person, or an agent. variantgate run runs the command only after a yes, passes on its exit code and reports the outcome.

#!/bin/sh
# apply-prod.sh: plan, ask with what the plan does, apply exactly that plan.
set -eu
terraform plan -input=false -out=tfplan
COUNTS=$(terraform show -json tfplan | jq -c '{
  create:  [.resource_changes[] | select(.change.actions | index("create"))] | length,
  update:  [.resource_changes[] | select(.change.actions == ["update"])] | length,
  destroy: [.resource_changes[] | select(.change.actions | index("delete"))] | length
}')
if [ "$(echo "$COUNTS" | jq .destroy)" -gt 0 ]; then
  ACTION=infra.destroy.production
else
  ACTION=infra.apply.production
fi
WHAT=$(echo "$COUNTS" | jq -r '"\(.create) to add, \(.update) to change, \(.destroy) to destroy"')
variantgate run --agent terraform-ci --action "$ACTION" \
  --summary "Apply the production plan for $(git rev-parse --short HEAD): $WHAT" \
  --details "$COUNTS" \
  -- terraform apply -input=false tfplan
  • The Terraform script applies the saved plan file, so what runs is what was approved. Terraform refuses a saved plan that has gone stale.
  • variantgate run needs VARIANTGATE_KEY in the environment. Install it with the TypeScript or Python package; both include the same command.
  • The command waits up to 15 minutes, the life of a request, then stops with exit code 1. A changed approval exits 3 and runs nothing.

Coding agents: Claude Code and Codex

A hook runs before every shell command the agent starts and lets it through only on a yes. Install the command once with npm install --global (see Claude Code or Codex), then add:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "variantgate hook claude-code", "timeout": 180 }
        ]
      }
    ]
  }
}

Every command then asks as shell.run, with the command itself in the details. The hook waits two minutes for a person, then denies. Claude Code falls back to its own prompt if the service cannot be reached; Codex denies.

Because rules see shell.run, not the command, no rule can allow ls and hold back terraform apply. Turn the hook on where the agent can reach production credentials, and rely on the wrapped scripts where it cannot. With both, a production change asks twice: once for the command, once for the change.

What the person approving sees

terraform-ci

infra.destroy.production · asked just now

14 min left

Apply the production plan for 3f2a1c9: 2 to add, 1 to change, 1 to destroy

0 of 2 approvals

create

2

destroy

1

update

1

How this request looks in the inbox, with example values.
  • The summary carries the plan’s counts, and the details show them. A non-zero destroy is the line to read first.
  • This one needs two people. The first approval shows on the card as “1 of 2 approvals”, with who gave it.
  • Deny with a note such as “The replace on aws_db_instance is not intended”. The script stops with exit code 1 and prints it.

Rolling it out

  1. Wrap the production commands, write no rules. Every production deploy, apply and cluster change waits for a person.
  2. Turn the hook on for coding agents that work where production credentials are.
  3. After a week or two, allow what you never refuse. Staging deploys are the usual first rule.
  4. Route production to on-call, and require two people for destroys.
  5. Keep production deploys with a person. While the ask rule covers deploy.production, remembering an approval from the inbox is refused rather than ignored: a more cautious rule always wins.

When something goes wrong

Pause the agent or the whole workspace from the emergency stop. Wrapped commands and hooks get allowed: false until an owner resumes. For schema changes in the same pipeline, see database migrations.