The 3am packet ❦ that bit every other Tuesday.

On Tuesday May 19 I came in at 02:30 UTC with a thermos and a Cat6 console cable. The capture I had planned was deliberately wide — broader than I usually allow myself, because I'd rather scroll a gigabyte of pcap than re-run a missed event. I started two tcpdumps, one on each side of the peering.

# on the cluster side, 10.40.0.0/16 internal
sudo tcpdump -i ens5 -s 0 -w /var/cap/fra-05-19.pcap \
  'host 10.42.7.5 or net 192.0.2.0/24 or arp' \
  -G 1800 -W 4 -Z root # 30-min rotating, 2h retained

# on the gateway facing the NorthRing peering, ens6 is the 10G LACP
sudo tcpdump -i ens6 -nn -s 96 -tt -w /var/cap/peer-05-19.pcap \
  '(vlan 412) and (tcp port 4443 or arp or icmp)'

The capture filter on ens6 matters more than you might think. NorthRing tags the private peering on VLAN 412; without the vlan 412 predicate, BPF on Linux misses the dot1q-tagged ARP entirely, because the kernel had already de-tagged for the rest of the stack. This is the kind of thing you learn once, painfully, and never forget. I had learned it once, painfully, in 2019, in a different city, debugging a different problem.

At 02:58:41 UTC the first sign appeared. A single ARP request from 192.0.2.1 asking for 192.0.2.5 — our side of the peering — broadcast on VLAN 412. Nothing unusual yet; ARP requests happen. What was unusual was that we had answered the same request from the same MAC fourteen times in the previous hour. Our gateway always replied within 40 microseconds. NorthRing's distribution switch, for some reason, was asking again every four to six minutes.

Then, at 03:04:09 UTC, two ARP replies arrived almost on top of each other for the same address. One came from our gateway's ec:0d:9a:73:1c:bf. The other came from 00:50:56:91:a0:42 — a VMware OUI. We do not run VMware on this network. We have not run VMware on this network since 2022. There it was, in the trace, calmly replying as if it owned the address. The L2 was forked.

Wireshark is a kind of dictionary you have to know how to read aloud. I spent the first two hours of May 20 going down filter rabbit holes that looked productive but weren't. Here, for the public record, are the ones that did not help, followed by the four that did.

The duplicate-address-detected expander told the story in one window. Wireshark had been quietly noticing, on every Tuesday capture, that 192.0.2.5 was being answered by two MACs within a few seconds of each other. We had two captures from previous Tuesdays — March 24 and April 7 — that I had filed and not opened. Both, when re-examined, contained the same two replies. The ghost MAC 00:50:56:91:a0:42 appeared, did its damage, and disappeared within a window of forty to sixty seconds. The distribution switch downstream, after this brief seizure, settled on our MAC again and the world looked normal.

tshark -r peer-05-19.pcap \
  -Y 'arp.opcode==2 and arp.src.proto_ipv4==192.0.2.5' \
  -T fields -e frame.time -e arp.src.hw_mac -e arp.src.proto_ipv4

# 03:04:09.412  ec:0d:9a:73:1c:bf  192.0.2.5   ← our gateway, expected
# 03:04:09.488  00:50:56:91:a0:42  192.0.2.5   ← who are you?
# 03:04:21.117  ec:0d:9a:73:1c:bf  192.0.2.5   ← gratuitous, ours

By Thursday morning I had a packet-level diagnosis but no why. The why is always the harder thing. A timeline helped — I sat down with the previous eleven incidents and the partner's public maintenance window page and worked through them in chronological order.

Once you see it, you cannot unsee it. The fortnightly cadence wasn't mysterious — it was a cron job. The 03:04 onset wasn't poetic — it was the third minute of a four-minute maintenance script. The ghost MAC was a literal ghost: an Ansible inventory entry from a server that had been decommissioned thirty-eight months earlier but never removed from a group_vars/static_arp.yml file. It returned, faithfully, every fourteen days, like a sleepwalker.

A polite, evidence-laden e-mail to NorthRing's NOC at 03:00 UTC on Sunday May 3 was the single most leveraged action of this entire investigation. I attached three pcap excerpts, a timeline of incidents, and the exact filter that surfaced the ghost ARP. I did not speculate about cause. I described, in flat declarative prose, what I had observed.

# Run on a payments gateway, watches for ghost MAC reappearance.
# Lives in /usr/local/sbin/arp-ghost-watch — runs from systemd timer.
#!/usr/bin/env bash
target="192.0.2.5"
expected_mac="ec:0d:9a:73:1c:bf"
ts="$(date -u +%FT%TZ)"
seen="$(arping -c 3 -I ens6 -f $target | awk '/reply/ {print $5; exit}')"
if [[ "$seen" != "$expected_mac" ]]; then
  logger -t arp-ghost-watch "GHOST $ts saw=$seen expected=$expected_mac"
  curl -sS -X POST "$ALERT_URL" -d "ghost-arp $ts $seen"
fi

The watcher has fired exactly zero true positives since June 2. It has fired twice during planned maintenance on our side and both alerts were correctly suppressed by the silencing label. I don't trust the silence. I trust the watcher. I will keep it running for at least a year.

A postmortem isn't worth filing if it doesn't change the rooms it touches. Here are the seven things this incident changed, ordered by usefulness rather than chronology.

The thing I keep returning to, the thing I want every junior engineer at our cage to take from this story, is the cadence of the work itself. I did not discover the ghost ARP. I let the network discover it for me, by waiting until 02:30 UTC on a Tuesday with a thermos and two tcpdumps. The patient hours of capture were the entire investigation. Wireshark filters are tools, not searches; they help you read a thing you already have rather than fetch a thing you don't.

Frequently asked, often badly.