Progressive delivery with traffic mirrors

Traffic mirroring lets you validate behavior with production-shaped load while shielding users from regressions. The patterns below are the ones we reach for when a change is risky but reversible.

Start with a shadow pool

Create a mirror target that points to the new build. Keep it stateless and isolated so failures are contained.

apiVersion: v1
kind: Service
metadata:
  name: api-shadow
spec:
  selector:
    app: api
    version: shadow
  ports:
    - port: 80
      targetPort: http

Send a slice of traffic to the shadow pool using your ingress controller. Here is an NGINX example:

location /api {
  proxy_pass http://api-primary;
  mirror /_mirror;
}

location /_mirror {
  internal;
  proxy_pass http://api-shadow;
}

Observe at two layers

We log two sets of signals: application traces and network deltas. Anything unstable in either view stops rollout.

# capture deltas in response codes between primary and shadow
mirrorctl compare \
  --primary "loki-query rate(http_requests_total{version='primary'}[5m])" \
  --shadow  "loki-query rate(http_requests_total{version='shadow'}[5m])"

-- surface request skew by endpoint
select
  route,
  percentile_cont(0.95) within group (order by shadow_latency_ms) as p95_shadow,
  percentile_cont(0.95) within group (order by primary_latency_ms) as p95_primary,
  (p95_shadow - p95_primary) as drift_ms
from mirror_latency
where ts > now() - interval '10 minutes'
group by route
order by drift_ms desc;

Promote by header, not by hope

Avoid global switches. Promote traffic via opt-in headers from trusted clients (e.g., QA, observability hooks).

# curl through the ingress with an opt-in header
curl -H "x-rollout: shadow" https://api.example.com/payments/health

// envoy filter to route header-tagged requests
{
  "route_config": {
    "virtual_hosts": [
      {
        "routes": [
          {
            "match": { "prefix": "/payments", "headers": [{"name": "x-rollout", "exact_match": "shadow"}] },
            "route": { "cluster": "api-shadow" }
          }
        ]
      }
    ]
  }
}

Cut over with a tight loop

When metrics and traces stay flat, flip the default and leave the mirror in place for fast rollback.

# promote shadow to primary
kubectl patch service api -p '{"spec":{"selector":{"app":"api","version":"shadow"}}}'

# keep the mirror alive for 24h to catch late regressions
schedule "mirror-drain" --after "24h" --action "kubectl scale deploy/api-shadow --replicas=0"

Shadow traffic gives teams room to experiment safely. Use it whenever a change is hard to predict from unit tests alone.