At 7:00 AM, Linda called. "Why are the morning graphs showing record throughput?"
Or rather, two of the WAPs did the heavy lifting. The third one, wap-03.internal.stratus.com , was the problem child.
It was 2:00 AM on a Tuesday. I was on call, nursing a cold brew and watching the dashboards for Stratus Finance , a global payment processor. Our web cluster was pristine: six origin servers humming behind three Web Application Proxy (WAP) servers. The WAPs handled SSL offloading, pre-authentication, and acted as a reverse proxy for our customer-facing APIs. remove web application proxy server from cluster
"Removed a bad actor from the team," I said, sipping my cold brew.
And always, always check your health checks. At 7:00 AM, Linda called
No alerts. No 500 errors. No angry emails from the night shift fraud team.
She paused. "The WAP server?"
Tonight was the night. I had a change ticket: CHG-0421 – Remove wap-03 from cluster and decommission.
But here's the terrifying part. Because wap-03 was "alive" according to basic ICMP pings, the cluster's consensus protocol had been treating it as a voting member. For six months, every time wap-03 choked on a null byte, it would delay the cluster's session replication by 400ms. It was 2:00 AM on a Tuesday
I ran the stop command: Stop-WebApplicationProxy -Node wap-03