We've been below capacity for A100 hardware for the past 30 minutes and all queues have been drained. Thanks for your patience

All affected models have been scaling correctly for more than 30 minutes at this point, and we see no residual prediction queues. Thank you for your patience!

We have been monitoring for a while and services seem to be back. Thank you for your patience

We have restarted the affected component and service should be back to normal. Thank you for your patience

We're looking stable now, and with caching back on. Thanks again for your patience!

This issue is resolved

We have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared

H100 capacity has returned to normal levels

We are back under max capacity for H100 hardware. Thank you for your patience!

The H100 capacity is back in good health. Thanks for your patience!