CI/CD in 10 Minutes

A pipeline is a sequence of evidence gates. Put formatting, static checks, and focused unit tests first because they fail cheaply. Run independent jobs in parallel. Put integration tests and packaging later, then deploy the exact package that passed. If developers wait forty minutes to learn about a typo, they will batch changes and stop trusting the pipeline.

# Fail cheap, then spend more confidence
lint ───────────────┐
unit tests ─────────┼─→ integration → build once → deploy staging → production
secret scan ────────┘

# Every command must also work on a clean developer machine
./ci/lint
./ci/test-unit
./ci/test-integration
./ci/build

Error: Process completed with exit code 1.

The same commit and declared inputs should produce the same functional artifact. Pin the runtime, commit the lockfile, start from a clean workspace, and keep build-time network choices out of your control path. Record the source commit and tool versions inside the artifact. “Latest” is not a version; it is a delayed incident.

# Pin base image by immutable digest
FROM node:22.17.0-alpine@sha256:0123456789abcdef...
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm test && npm run build
ARG GIT_SHA
LABEL org.opencontainers.image.revision=$GIT_SHA

Reproducible does not mean every archive byte is automatically identical; timestamps and file ordering can vary unless normalized. It means nobody can quietly choose a newer dependency while rebuilding yesterday's commit. Prefer deploying the artifact already built by CI instead of rebuilding it in each environment.

A cache is an optimization that may disappear at any moment. Key dependency caches by operating system, architecture, runtime version, and lockfile hash. Never cache test results without including every input that changes them. Never put secrets or built release artifacts into a general shared cache.

# Good cache identity
key: npm-${{ runner.os }}-node-22-${{ hashFiles('package-lock.json') }}
path: ~/.npm

# Prove the pipeline works without yesterday's state
weekly-clean-build: restore-keys: []

# Artifact storage is not a cache
release/app-8f31c2a.tar.gz  retention: 90 days  immutable: true

npm error code EINTEGRITY
npm error sha512 checksum failed when using sha512: wanted ... but got ...

Do not store credentials in the repository, workflow file, image layer, artifact, cache, or log. Prefer workload identity: the CI job proves who it is and receives a short-lived cloud credential. Restrict production deployment to protected branches and environments. Pull requests from forks must not receive privileged secrets.

# Prefer identity exchange over a permanent cloud key
permissions:
  contents: read
  id-token: write

environment: production
# Production environment adds approval and branch restrictions.

Resource not accessible by integration

Build once, identify the result by digest, and promote that same artifact through test, staging, and production. Environment-specific configuration arrives at runtime. If staging rebuilds from source and production rebuilds again, you tested two cousins of the deployed software, not the deployed software.

# CI records an immutable identity
IMAGE=registry.example/app@sha256:4c3f...9a10

# Staging and production differ in configuration, not application bytes
deploy staging    $IMAGE --config=config/staging
verify staging    $IMAGE
approve production
deploy production $IMAGE --config=config/production

Keep test data out of production and production credentials out of staging. Make environments similar in architecture, but do not pretend they are interchangeable. Attach the commit, build run, dependency manifest, checksums, and release notes to the artifact so an operator can answer exactly what is running.

Application rollback is easy only when the database still accepts the old application. Use expand and contract. First add a nullable column, new table, or compatible index. Deploy code that understands old and new shapes. Backfill in bounded batches. Switch reads. Remove the old shape in a later release after rollback risk has passed.

# Release A: expand
ALTER TABLE users ADD COLUMN display_name text;
# App writes both old name and display_name.

# Background job: small batches, observable progress
UPDATE users SET display_name = name
WHERE display_name IS NULL AND id > $1 AND id <= $2;

# Later release: contract after every reader moved
ALTER TABLE users DROP COLUMN name;

A rolling deployment gradually replaces instances and costs little extra capacity, but old and new versions overlap. Blue-green keeps two complete environments and switches traffic quickly, which makes reversal fast but costs more. A canary sends a small percentage to the new version and expands only when real metrics remain healthy.

# Canary progression with explicit gates
1%  for 10 min → 10% for 20 min → 50% for 20 min → 100%

stop if:
  error_rate > baseline + 0.5 percentage points
  p95_latency > baseline * 1.20
  checkout_success < agreed floor

# Health means dependency-ready, not merely process-alive.

Do not choose Kubernetes, canaries, or blue-green to look mature. A small service with instant rollback may need a careful rolling release. Use canaries when production behavior supplies evidence staging cannot, and make the gate automatic. Watching a dashboard while traffic rises is theater unless someone owns a stop rule.

Rollback is a normal control, not an admission of defeat. Keep the previous artifact deployable, store the active version, and make reversal one tested command. Roll back application code when errors rise. Roll forward when data has already been transformed incompatibly or a security fix cannot be reintroduced safely. Feature flags can disable behavior faster than either, but stale flags become a second configuration system.

# A release record gives the operator concrete choices
current:  app@sha256:4c3f...9a10  commit: 8f31c2a
previous: app@sha256:119d...7b42  commit: 60ab911

# Reversal changes the desired digest; it does not rebuild.
./deploy production app@sha256:119d...7b42
./smoke-test https://app.example
./verify-metrics --compare-to=pre-release

Practice this path. A rollback script that has not run in six months is documentation, not a capability. Define who can trigger it, what metric crosses the threshold, how database compatibility is checked, and how customers are told. Preserve the failed artifact and logs for diagnosis after service is safe.

Separate pipeline failures into source, runner, dependency, credential, artifact, deployment, and runtime layers. Start with the first failed command, the runner image and version, the commit, and whether a clean rerun changes the result. Compare facts with the last successful run. Randomly editing YAML destroys evidence.

/bin/sh: ./ci/build: Permission denied

# Inspect the repository mode, not just local filesystem behavior
git ls-files --stage ci/build
# wanted: 100755 ... ci/build

# Fix and commit the executable bit
git update-index --chmod=+x ci/build
# Also verify the first line, for example: #!/bin/sh

Use this as a release review, not as a badge collection.

# commit path
fast checks in parallel → integration → build once → checksum and attest
→ deploy same digest to staging → verify → approve → production

# trust boundaries
untrusted fork: no privileged secrets
CI identity: short-lived credential, least privilege
cache: disposable speed-up · artifact: immutable release input

# database
expand → deploy compatible code → backfill → switch reads → contract later

# release
rolling: simple, versions overlap
blue-green: quick switch, double capacity
canary: production evidence, automatic stop rules

# failure order
first failed command → first concrete error → clean local reproduction
→ compare runner, versions, permissions, cache, artifact digest

# rollback
keep previous digest · test the command · preserve database compatibility

My opinionated version: optimize for boring releases. A ten-minute pipeline that produces one traceable artifact is better than a two-minute pipeline that occasionally deploys stale cache contents. A one-command rollback is better than an elaborate strategy nobody has rehearsed. CI/CD is successful when shipping a small change feels routine and stopping a bad one feels immediate.