Webhooks in 10 Minutes

A webhook is an API call you do not initiate. That reversal creates three jobs: prove who sent it, make repeat delivery harmless, and acknowledge it before your useful work begins. Get those right and webhooks become boring infrastructure, which is exactly what you want.

🎙️ Published & recorded:

01Webhook vs API: who starts the call?

With a normal API, your code asks a question when it wants an answer. With a webhook, you publish an address and somebody else's system calls it when an event occurs. Polling asks, “Did the payment change yet?” every minute. A webhook says, “Tell me when it changes.” Use webhooks for timely notification, then use the provider's API when you need the current authoritative record.

# API: your app initiates
GET https://api.example.com/payments/pay_42

# webhook: provider initiates
POST https://your-app.com/webhooks/payments
Content-Type: application/json

{"id":"evt_91","type":"payment.succeeded","data":{"payment_id":"pay_42"}}
A useful boundary

Treat the webhook as a notification, not a magical command. If the consequence is expensive or sensitive, fetch the object from the provider's API after verifying the event. That protects you from stale fields and keeps the provider's record authoritative.

02Design a narrow endpoint contract

Give each provider its own route, accept only POST, require HTTPS in production, and keep the handler deliberately small. Read the headers, preserve the original bytes, authenticate the message, record the event, enqueue work, and return. Do not funnel five providers through one clever generic route. Their signatures, retry rules, and payload versions will diverge.

POST /webhooks/acme

# Keep the envelope. It is your audit trail and deduplication key.
{
  "id": "evt_91",
  "type": "invoice.paid",
  "created_at": "2026-07-24T12:30:00Z",
  "data": { "invoice_id": "inv_7" }
}

Pin the provider's webhook API version when that option exists. Store the event type, provider event ID, receipt time, signature result, and processing state. Payloads that only live in application logs become impossible to replay cleanly.

03Verify the signature before trusting JSON

A public endpoint can be called by anyone. A secret URL is not authentication; URLs leak into logs and screenshots. Good providers sign the exact request bytes with a shared secret. Recompute that signature, compare it in constant time, and reject old timestamps so a captured request cannot be replayed forever.

import { createHmac, timingSafeEqual } from "node:crypto";

function validSignature(rawBody, timestamp, suppliedHex, secret) {
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
  const expected = createHmac("sha256", secret)
    .update(String(timestamp)).update(".").update(rawBody).digest();
  const supplied = Buffer.from(suppliedHex, "hex");
  return supplied.length === expected.length && timingSafeEqual(supplied, expected);
}
Do not improvise the format

Use the provider's documented signing string and official library when one is maintained. Some sign the body alone; others sign a timestamp, a dot, and the body. “HMAC SHA-256” does not tell you the byte format. Rotate secrets with a short overlap where both old and new keys verify.

04The raw-body trap

Signature verification operates on bytes, not on the JavaScript object your JSON parser creates. Parsing and serializing can change whitespace, escapes, or key formatting. The data still looks identical to a person, but the signature no longer matches. Capture the raw request body first, verify it, and parse JSON only after verification succeeds.

Webhook signature verification failed.
Error: No signatures found matching the expected signature for payload.

# Express: raw middleware must run before express.json()
app.post("/webhooks/acme", express.raw({ type: "application/json" }), handler);
# Put app.use(express.json()) after the webhook route.
Fix it in this order

One: log the body type and confirm it is a Buffer or byte array, not an object. Two: move the raw-body middleware before every JSON parser that can touch this route. Three: compute the signature over those untouched bytes with the endpoint's current secret. Four: confirm the timestamp and signature header names match the provider docs. Never “fix” this by skipping verification.

05Retries mean duplicates; order is not promised

Webhook delivery is usually at least once. If the provider misses your response, it cannot know whether you processed the event, so it sends the same event again. Duplicates are normal, not an edge case. Put a unique constraint on the provider's event ID and make the resulting business operation idempotent too. Also assume events can arrive out of order.

CREATE TABLE webhook_events (
  provider text NOT NULL, event_id text NOT NULL,
  event_type text NOT NULL, payload jsonb NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (provider, event_id)
);

INSERT INTO webhook_events(provider,event_id,event_type,payload)
VALUES ($1,$2,$3,$4) ON CONFLICT DO NOTHING;
A real duplicate symptom

If PostgreSQL says duplicate key value violates unique constraint "webhook_events_pkey", the constraint did its job but your insert path did not. One: change the insert to ON CONFLICT DO NOTHING. Two: check whether a row was inserted. Three: enqueue only when it was new. For ordering, compare provider object versions or event timestamps and ignore stale transitions; never infer sequence from arrival time.

06Acknowledge 2xx before async processing

Your synchronous path should verify, atomically record the event plus work to do, and answer with a 2xx response. Then a worker sends email, updates search, calls slow APIs, and performs business logic. “Finish business processing, then return 200” is a bad default. It couples provider delivery to every dependency you own and manufactures retries during ordinary slowness.

async function webhook(req, res) {
  const event = verifyAndParse(req.rawBody, req.headers);
  // One database transaction writes both the inbox event and
  // an outbox job. A dispatcher publishes unsent outbox rows.
  await db.transaction(tx => tx.storeEventAndOutboxIfNew(event));
  res.status(204).end();
}
Timeout failure, fixed

A provider dashboard may report 504 Gateway Timeout while your logs show the work completed seconds later. One: time the handler and identify calls after verification. Two: move each slow call to a worker. Three: commit the inbox row and outbox job in one transaction, or use a queue with an equivalent atomic handoff. Four: return 204 immediately afterward. A dispatcher can retry unsent outbox rows after a crash, so an acknowledged event cannot get stranded between database and queue.

07Test localhost without pretending it is public

A provider on the internet cannot call localhost on your laptop. Use the provider's official webhook forwarding CLI when available; otherwise use a reputable HTTPS tunnel. Point the provider at the temporary HTTPS address, but keep signature verification enabled and use a test-mode secret. A tunnel exposes your machine to real traffic.

# First prove the local route itself works
curl -i -X POST http://127.0.0.1:3000/webhooks/acme \
  -H "Content-Type: application/json" \
  --data-binary @fixture.json

# Then start the provider CLI/tunnel and register its HTTPS URL:
https://temporary-host.example/webhooks/acme
Connection refused, fixed

If curl prints curl: (7) Failed to connect to localhost port 3000: Connection refused, no process is listening there. One: start the app. Two: confirm its actual port in the startup log. Three: bind to the interface required by your tunnel or container. Four: rerun local curl before debugging the tunnel. Copy a real test delivery into a fixture so you can replay the same bytes repeatedly.

08Operate an inbox, not a mystery endpoint

A useful webhook system has an inbox table, queue attempts, and a dead-letter path. Log the event ID, type, receipt latency, verification result, HTTP response, processing attempt, and final state. Never log signing secrets or authorization headers. Your dashboard should answer three questions quickly: did we receive it, did we accept it, and what happened afterward?

# One searchable structured record
{
  "provider":"acme", "event_id":"evt_91", "type":"invoice.paid",
  "verified":true, "response_status":204, "queue_id":"job_6",
  "processing_state":"succeeded", "attempt":1
}

Keep a controlled replay tool that re-enqueues a stored, already-verified event by ID. Do not replay by forging a new inbound HTTP request unless you intentionally want to test verification too. Alert on growing queue age and repeated failures, not on every single retry.

09Troubleshooting: read the failure literally

Start at the boundary and move inward. Provider delivery log, edge proxy, application receipt, signature verification, inbox insert, queue publish, worker result. Guessing from the final business symptom wastes time because a webhook can fail at six distinct layers.

404 Not Found                         # wrong public path or deployment
405 Method Not Allowed                  # route does not accept POST
415 Unsupported Media Type              # body parser rejects Content-Type
Webhook signature verification failed.  # bytes/secret/header mismatch
504 Gateway Timeout                     # handler answered too slowly
Step-by-step triage

For 404, copy the exact delivery URL, compare its path with the deployed route, and curl that public URL. For 405, register POST on that route and check whether a proxy rewrites the method. For 415, inspect the received Content-Type, configure raw bytes for that media type, and do not parse first. For a signature failure, verify raw bytes, endpoint-specific secret, signing format, and clock. For 504, move business work behind a durable queue and acknowledge sooner.

10Webhook cheat sheet

The production checklist is short because the design should be short.

# inbound path
POST only → capture raw bytes → verify signature + timestamp
→ parse JSON → insert unique event ID → durable queue → 204

# worker path
load stored event → idempotent business action → mark succeeded
retry transient failures with backoff → dead-letter permanent failures

# assumptions
duplicates: yes · delayed delivery: yes · out of order: yes
one delivery only: no · secret URL is auth: no · arrival order is truth: no

# debug order
provider log → public URL/status → app receipt → signature
→ inbox row → queue job → worker result

# localhost
local curl first → official forwarder or HTTPS tunnel → test secret

My opinionated version: keep the receiver dull and make the worker capable. If the receiver knows how to verify, persist, enqueue, and answer, it will survive traffic spikes and bad dependencies. If it also sends receipts, updates five tables, and calls three APIs, it will eventually turn a five-second outage into a duplicate-delivery incident.