We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC).

If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

We've applied a candidate fix and are seeing some services begin to recover, and are continuing to monitor

We're continuing to monitor affected gRPC APIs as well. We're seeing some reports of StatusCode.UNAVAILABLE being returned to SDKs from the API

We're monitoring some degradation in performance on the Hatchet dashboard and REST APIs, and the scheduler, and are working on a fix.

We believe about 30% of tenants are affected