Resolved
Incident is resolved.
Investigating
All models have returned to normal processing timeframes. Thank you for your patience.
Investigating
We have shifted traffic for the Flux Schnell model to a region with additional capacity. While we work through the backlog of requests, new requests should begin to be served within a normal timeframe. This continues to impact only the Flux Schnell official model.
Investigating
We have identified and remediated a data store related to queues that was wedged, preventing traffic from reaching it within one of our regions. Most inference and API usage as returned to normal with the exception of the Flux Schnell model. We are working on restoring service to the flux schnell model. The impact is large delays on predictions and elevated errors when submitting preditions.
Investigating
We are aware of elevated errors for predictions being sent to one of our regions. This is impacting an subset of models targeting H100 and L40S Hardware types. We are investigating and will provide an update as soon as information is available.
