Incidents reported on status page for Hatchet https://status.hatchet.run/ https://d1lppblt9t2x15.cloudfront.net/logos/0f7bb28a2e044dec7ac360534525bc0a.png https://status.hatchet.run/ en https://status.hatchet.run/incident/940638 Wed, 01 Jul 2026 15:50:00 -0000 https://status.hatchet.run/incident/940638#39f2e6f9d257a4ef037f62b146c40a96fcb7fc24761d276b557cff0420f4f6b6 We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC). If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again. https://status.hatchet.run/incident/940638 Wed, 01 Jul 2026 15:15:00 -0000 https://status.hatchet.run/incident/940638#9196d89c02e0b37467d17b814f0d81fc803e270a0629606235c3a27bfbac4567 We've applied a candidate fix and are seeing some services begin to recover, and are continuing to monitor https://status.hatchet.run/incident/940638 Wed, 01 Jul 2026 15:00:00 -0000 https://status.hatchet.run/incident/940638#93b3a65314b620c50c0895cb3d0e1d9c669a84f573adae7426f559cff9476433 We're continuing to monitor affected gRPC APIs as well. We're seeing some reports of `StatusCode.UNAVAILABLE` being returned to SDKs from the API https://status.hatchet.run/incident/940638 Wed, 01 Jul 2026 14:43:00 -0000 https://status.hatchet.run/incident/940638#2343c0e2a109ee5636acda13936104e596a18bc07edfd6a614359b6faa8fb105 We're monitoring some degradation in performance on the Hatchet dashboard and REST APIs, and the scheduler, and are working on a fix. We believe about 30% of tenants are affected https://status.hatchet.run/incident/940181 Wed, 01 Jul 2026 05:42:00 -0000 https://status.hatchet.run/incident/940181#2209dfbd39fb066b12a449babc5905f0bd91d7256eb6a2723d516b129fd8c7f8 We have identified the source of the latency and have pushed initial configuration changes to mitigate. We will continue to monitor for degraded performance. https://status.hatchet.run/incident/940181 Wed, 01 Jul 2026 04:14:00 -0000 https://status.hatchet.run/incident/940181#25f9d1a569dbe22b92bdf1c6319b7e862363f13a8b769b769b1fd4f80e558e0c We are investigating reports of elevated latency on our OLAP database, which powers parts of the Hatchet UI and some API requests. The core database is not affected. Workflow execution and workloads are continuing to run normally. https://status.hatchet.run/incident/930584 Sun, 21 Jun 2026 20:00:00 -0000 https://status.hatchet.run/incident/930584#7a67f703b17016500daefb14e4ece69a3b203de12040cc9f371e9639c8dd4e24 We’re back! Hatchet Cloud region us-west-2 should be up and running. Thanks for bearing with us. https://status.hatchet.run/incident/930584 Fri, 19 Jun 2026 23:00:00 -0000 https://status.hatchet.run/incident/930584#68c6ac68f2dd1100f45fe48848b6b0c087939b53f291589a5a818a4be7fdfbdf Hatchet's shared cloud (free, team, and scale plans) in us-west-2 experienced a partial outage between 6/19/26 23:07:00 UTC and 6/20 and 00:34:00 UTC caused by crashes on an internal queue. We were alerted and responded immediately, which brought the service online. Task processing (e.g. dispatching tasks to workers) was affected, but ingestion was not. You may see stuck DAGs, which likely will need to be cancelled and replayed or retriggered from scratch to reconcile. We'll continue to update as we investiage the root cause of the message queue crash. https://status.hatchet.run/incident/912076 Wed, 03 Jun 2026 14:30:52 -0000 https://status.hatchet.run/incident/912076#063d85cf42684f9e863cc8259690c4d3dc94d855d35669e2c0e1185f12762a7b Self-Service billing on cloud.hatchet.run will be unavailable as we work on organization scoped plans. Please contact us should you need to make immediate changes https://hatchet.run/office-hours https://status.hatchet.run/incident/905642 Wed, 13 May 2026 18:20:00 -0000 https://status.hatchet.run/incident/905642#cbb6ea887d441dae31e1e9e9bad3ad3469cd5e4b0b76900dd99ca53627204537 Hatchet experienced ~15 minutes of partial, unexpected downtime on some parts of the system on May 13th, 2026 between 17:56 and 18:20 UTC, causing delayed task runs, the dashboard to appear frozen, and errors on ingestion in certain cases. Cron runs were also impacted, and will need to be replayed manually (which we can help with, if needed). This was an isolated incident, which should not happen again. Root Cause: At approximately 17:56, we began a routine deploy included a database migration. This migration ran an ALTER TABLE statement before a VALIDATE CONSTRAINT in the same transaction, which resulted in the ACCESS EXCLUSIVE lock on the table that was acquired during the original ALTER TABLE to be held for the duration of the transaction. This lock was then held for the duration. of the VALIDATE CONSTRAINT (which would otherwise not require an ACCESS EXCLUSIVE lock). This kept the internal table locked for an extended period of time, causing significant backpressure on our internal queue, and some upstream effects on publishers. We took steps to mitigate the effects of the issue as soon as we were aware of it, at which point full recovery took approximately ten minutes. At this time, we believe that task ingestion was continuing to progress as normal, albeit with delays on execution, as long as the task was accepted by Hatchet. Action Items: We’re actively working on an improved way of running migrations under load on large databases, and are planning to have that ready to use within the upcoming weeks, to help safeguard against this kind of unexpected failure scenario in our deploy process. Additionally, we’re continuously working on improving our monitoring and alerting to help recover more quickly from incidents such as this one. https://status.hatchet.run/incident/857459 Wed, 25 Mar 2026 16:24:00 -0000 https://status.hatchet.run/incident/857459#05801fc85571130ca1950efcb801949adcbdeebb9a09155cfc967e7a36326717 Between 5:00 UTC and 16:25 UTC, we received reports of increased latency and error rates impacting some tenants in us-west-2. We have scaled capacity and all tenants are recovered. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 15:04:00 -0000 https://status.hatchet.run/incident/779664#05942827d32f3cca4d2987630edc7ddcc3a4d0d272ad7b2b7c6beb39b4d5572b This issue is now resolved and runs should be available again over the UI. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:57:00 -0000 https://status.hatchet.run/incident/779664#59f2c69a359bbafdee222f2c13671f64ef9ddd8bcedfdef03d1c5fc9ee7f5339 Update: we are rolling out a hotfix to get around the DNS resolution failures. Services are starting to recover. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:43:00 -0000 https://status.hatchet.run/incident/779664#04c6206ae5b6a52f429dc6cfe92302007d8b79fb4544ffefa5a13d375fb20b64 We are seeing DNS issues with our upstream database provider which is preventing runs from loading in the UI or over the API. Core queueing operations are not impacted, but endpoints utilizing the REST API will fail to respond. https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:55:00 -0000 https://status.hatchet.run/incident/769126#ba1ff2ecd03a17634ad04b42dc22413716adfb3cd02da11c8a431a51a931b6a7 We have taken steps to mitigate affected DNS services and have fully restored all Hatchet services https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:33:00 -0000 https://status.hatchet.run/incident/769126#3784ae48737fd6fa833c78c80dea956aad93b31283bde6cdb5b3c3a3dbebec83 We are currently tracking a global outage of Cloudflare's DNS. This is currently impacting access to cloud.onhatchet.run and dashboards for dedicated tenants as well as REST API calls. Workers, which connect via gRPC and do not proxy through Cloudflare, should continue to run and should not be impacted. You can track this issue directly here: https://www.cloudflarestatus.com/ https://status.hatchet.run/incident/764298 Fri, 14 Nov 2025 20:06:00 -0000 https://status.hatchet.run/incident/764298#341ab1ba2066b50b0003225d59846c05db63df346b61abb7fc8f49a7a892622c On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:55:00 -0000 https://status.hatchet.run/incident/764298#433c81879cfb98383606513487f5c2ebf4043b97dfe3c2b828feccd4bb4886eb We have identified the issue and deployed a patch, services should be fully restored. We are continuing to monitor and will provide a detailed update shortly. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:39:00 -0000 https://status.hatchet.run/incident/764298#5c840bcf60aec75bf94da83e7fcc1c32cb883e3dfc77a3a569babfcb1d271186 We have received reports for issues on enqueuing and assigning runs, we are currently investigating and working on a resolution. https://status.hatchet.run/incident/703206 Sat, 09 Aug 2025 01:55:00 -0000 https://status.hatchet.run/incident/703206#931db1054e0d92c56d5400402e559c1bc4e990cf91b812733a37e266634e5678 We saw an increased number of 503 error rates between 1:40-1:50 UTC. We've deployed a fix and are monitoring now. https://status.hatchet.run/incident/490423 Sat, 04 Jan 2025 11:00:19 -0000 https://status.hatchet.run/incident/490423#d246ae4a3477bfd4aa12d3788211d52048aa9f182a592cb7e89af60162308ef7 We are performing essential database maintenance affecting shared and dedicated cloud infrastructure. Expected Impact: Increased latency and possible brief interruptions for shared and dedicated cloud tenants https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 23:02:06 -0000 https://status.hatchet.run/incident/490008#0fdef3f93aa108f33d4c9ee33b4a3ca2bd65036c3dc7de9dd2c290a222fd6557 Maintenance completed https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 21:00:30 -0000 https://status.hatchet.run/incident/490008#f61997ab0ea3ce64a87ab24b6673bbd8e63aee4bb5abe39b7104e0c0cd2feaeb We are performing essential database maintenance affecting shared cloud infrastructure. Dedicated instances will not be impacted. Expected Impact: Increased latency and possible brief interruptions for shared cloud tenants Update: Completed first phase https://status.hatchet.run/incident/476151 Wed, 11 Dec 2024 19:11:00 -0000 https://status.hatchet.run/incident/476151#3ec30ce98ea7423fe6e39f3fb5f968b959225ee1207da6a51bb25526de4fccbd The Hatchet API is currently experiencing increased API error rates. https://status.hatchet.run/incident/473127 Thu, 05 Dec 2024 21:30:00 -0000 https://status.hatchet.run/incident/473127#f56e0f284de50977cdca5d6a13cdb032f2b512ee14599a8e8a7150317109e37a Our API experienced elevated error rates from 21:30 to 21:33 UTC, caused by an automatic infrastructure upgrade.