Incorrectly disabled users have been re-enabled at this time.

All services have returned to normal.

Things seem to have been stable for a while now, marking as resolved.

All monitoring is showing normal behavior over the past 30 minutes. Thank you for your patience!

All telemetry is reporting normal at this point and the rollback is complete. Thanks for your patience! :sparkling_heart:

The acute problem appears to be resolved based on observations over the past hour. We don't have a clear explanation for why the errors began and ended, so further investigation is needed.

The in-process rollback of a node configuration change appears to have introduced enough pressure on autoscaling that we were seeing increased delays for cold boots and scale outs. Once we deleted many of the crashing pods, we saw queues rapidly drop.

L40S and CPU instances are properly pulling images again. All nodes are functioning as expected.