The in-process rollback of a node configuration change appears to have introduced enough pressure on autoscaling that we were seeing increased delays for cold boots and scale outs. Once we deleted many of the crashing pods, we saw queues rapidly drop.
Resolved
The in-process rollback of a node configuration change appears to have introduced enough pressure on autoscaling that we were seeing increased delays for cold boots and scale outs. Once we deleted many of the crashing pods, we saw queues rapidly drop.
Identified
We are seeing a large number of pods in Pending while waiting for a node refresh to complete. Clearing the pods stuck in CrashLoopBackOff appears to have helped clear the blockage.
