Resolved
We've added mitigations and been monitoring for a bit, and the service is looking healthy again. Thanks for your patience!
Investigating
We are seeing predictions sometimes fail to create on T4s. We believe we have found part of the problem and are continuing to look at more potential issues.