Jobs stuck in pending state

This issue has now been resolved.

A fix has been identified and implemented. We are observing a reduction in the job reconciliation backlog to normal levels, and customer jobs are now being submitted as expected.

Update: Team is continuing to investigate. Anyscale Jobs continue to be in status "Pending" for 5-10 minutes before being reconciled.

We are currently investigating this issue impacting the Anyscale platform.

Downtime on Platform, UI and SDK/CLI is now resolved. Thank you for your patience as we worked to resolve the issue

Anyscale Platform, UI and SDK/CLI have started to recover. At this point, the root cause appears to be related to an outage with AWS STS affecting the Anyscale Control Plane. The AWS outage appears to be mitigated at this time.

We are currently experiencing downtime on Anyscale Platform, UI and SDK/CLI. Submitted jobs are stuck in pending state. We are investing the issue.