Incidents reported on status page for Hatchet https://status.hatchet.run/ https://d1lppblt9t2x15.cloudfront.net/logos/e543620a36be1d0303e6f92860250dc8.png https://status.hatchet.run/ en https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 15:04:00 -0000 https://status.hatchet.run/incident/779664#05942827d32f3cca4d2987630edc7ddcc3a4d0d272ad7b2b7c6beb39b4d5572b This issue is now resolved and runs should be available again over the UI. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 15:04:00 -0000 https://status.hatchet.run/incident/779664#05942827d32f3cca4d2987630edc7ddcc3a4d0d272ad7b2b7c6beb39b4d5572b This issue is now resolved and runs should be available again over the UI. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:57:00 -0000 https://status.hatchet.run/incident/779664#59f2c69a359bbafdee222f2c13671f64ef9ddd8bcedfdef03d1c5fc9ee7f5339 Update: we are rolling out a hotfix to get around the DNS resolution failures. Services are starting to recover. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:57:00 -0000 https://status.hatchet.run/incident/779664#59f2c69a359bbafdee222f2c13671f64ef9ddd8bcedfdef03d1c5fc9ee7f5339 Update: we are rolling out a hotfix to get around the DNS resolution failures. Services are starting to recover. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:43:00 -0000 https://status.hatchet.run/incident/779664#04c6206ae5b6a52f429dc6cfe92302007d8b79fb4544ffefa5a13d375fb20b64 We are seeing DNS issues with our upstream database provider which is preventing runs from loading in the UI or over the API. Core queueing operations are not impacted, but endpoints utilizing the REST API will fail to respond. https://status.hatchet.run/incident/779664 Thu, 04 Dec 2025 14:43:00 -0000 https://status.hatchet.run/incident/779664#04c6206ae5b6a52f429dc6cfe92302007d8b79fb4544ffefa5a13d375fb20b64 We are seeing DNS issues with our upstream database provider which is preventing runs from loading in the UI or over the API. Core queueing operations are not impacted, but endpoints utilizing the REST API will fail to respond. https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:55:00 -0000 https://status.hatchet.run/incident/769126#ba1ff2ecd03a17634ad04b42dc22413716adfb3cd02da11c8a431a51a931b6a7 We have taken steps to mitigate affected DNS services and have fully restored all Hatchet services https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:55:00 -0000 https://status.hatchet.run/incident/769126#ba1ff2ecd03a17634ad04b42dc22413716adfb3cd02da11c8a431a51a931b6a7 We have taken steps to mitigate affected DNS services and have fully restored all Hatchet services https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:33:00 -0000 https://status.hatchet.run/incident/769126#3784ae48737fd6fa833c78c80dea956aad93b31283bde6cdb5b3c3a3dbebec83 We are currently tracking a global outage of Cloudflare's DNS. This is currently impacting access to cloud.onhatchet.run and dashboards for dedicated tenants as well as REST API calls. Workers, which connect via gRPC and do not proxy through Cloudflare, should continue to run and should not be impacted. You can track this issue directly here: https://www.cloudflarestatus.com/ https://status.hatchet.run/incident/769126 Tue, 18 Nov 2025 13:33:00 -0000 https://status.hatchet.run/incident/769126#3784ae48737fd6fa833c78c80dea956aad93b31283bde6cdb5b3c3a3dbebec83 We are currently tracking a global outage of Cloudflare's DNS. This is currently impacting access to cloud.onhatchet.run and dashboards for dedicated tenants as well as REST API calls. Workers, which connect via gRPC and do not proxy through Cloudflare, should continue to run and should not be impacted. You can track this issue directly here: https://www.cloudflarestatus.com/ https://status.hatchet.run/incident/764298 Fri, 14 Nov 2025 20:06:00 -0000 https://status.hatchet.run/incident/764298#341ab1ba2066b50b0003225d59846c05db63df346b61abb7fc8f49a7a892622c On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans. https://status.hatchet.run/incident/764298 Fri, 14 Nov 2025 20:06:00 -0000 https://status.hatchet.run/incident/764298#341ab1ba2066b50b0003225d59846c05db63df346b61abb7fc8f49a7a892622c On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:55:00 -0000 https://status.hatchet.run/incident/764298#433c81879cfb98383606513487f5c2ebf4043b97dfe3c2b828feccd4bb4886eb We have identified the issue and deployed a patch, services should be fully restored. We are continuing to monitor and will provide a detailed update shortly. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:55:00 -0000 https://status.hatchet.run/incident/764298#433c81879cfb98383606513487f5c2ebf4043b97dfe3c2b828feccd4bb4886eb We have identified the issue and deployed a patch, services should be fully restored. We are continuing to monitor and will provide a detailed update shortly. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:39:00 -0000 https://status.hatchet.run/incident/764298#5c840bcf60aec75bf94da83e7fcc1c32cb883e3dfc77a3a569babfcb1d271186 We have received reports for issues on enqueuing and assigning runs, we are currently investigating and working on a resolution. https://status.hatchet.run/incident/764298 Thu, 13 Nov 2025 22:39:00 -0000 https://status.hatchet.run/incident/764298#5c840bcf60aec75bf94da83e7fcc1c32cb883e3dfc77a3a569babfcb1d271186 We have received reports for issues on enqueuing and assigning runs, we are currently investigating and working on a resolution. https://status.hatchet.run/incident/703206 Sat, 09 Aug 2025 01:55:00 -0000 https://status.hatchet.run/incident/703206#931db1054e0d92c56d5400402e559c1bc4e990cf91b812733a37e266634e5678 We saw an increased number of 503 error rates between 1:40-1:50 UTC. We've deployed a fix and are monitoring now. https://status.hatchet.run/incident/703206 Sat, 09 Aug 2025 01:55:00 -0000 https://status.hatchet.run/incident/703206#931db1054e0d92c56d5400402e559c1bc4e990cf91b812733a37e266634e5678 We saw an increased number of 503 error rates between 1:40-1:50 UTC. We've deployed a fix and are monitoring now. https://status.hatchet.run/incident/490423 Sat, 04 Jan 2025 17:00:19 -0000 https://status.hatchet.run/incident/490423#4a2b4d7abf0725bbc9e980233bded3402a70ac665615a523dcbaf16c23957210 Maintenance completed https://status.hatchet.run/incident/490423 Sat, 04 Jan 2025 17:00:19 -0000 https://status.hatchet.run/incident/490423#4a2b4d7abf0725bbc9e980233bded3402a70ac665615a523dcbaf16c23957210 Maintenance completed https://status.hatchet.run/incident/490423 Sat, 04 Jan 2025 11:00:19 -0000 https://status.hatchet.run/incident/490423#d246ae4a3477bfd4aa12d3788211d52048aa9f182a592cb7e89af60162308ef7 We are performing essential database maintenance affecting shared and dedicated cloud infrastructure. Expected Impact: Increased latency and possible brief interruptions for shared and dedicated cloud tenants https://status.hatchet.run/incident/490423 Sat, 04 Jan 2025 11:00:19 -0000 https://status.hatchet.run/incident/490423#d246ae4a3477bfd4aa12d3788211d52048aa9f182a592cb7e89af60162308ef7 We are performing essential database maintenance affecting shared and dedicated cloud infrastructure. Expected Impact: Increased latency and possible brief interruptions for shared and dedicated cloud tenants https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 23:02:06 -0000 https://status.hatchet.run/incident/490008#0fdef3f93aa108f33d4c9ee33b4a3ca2bd65036c3dc7de9dd2c290a222fd6557 Maintenance completed https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 23:02:06 -0000 https://status.hatchet.run/incident/490008#0fdef3f93aa108f33d4c9ee33b4a3ca2bd65036c3dc7de9dd2c290a222fd6557 Maintenance completed https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 21:00:30 -0000 https://status.hatchet.run/incident/490008#f61997ab0ea3ce64a87ab24b6673bbd8e63aee4bb5abe39b7104e0c0cd2feaeb We are performing essential database maintenance affecting shared cloud infrastructure. Dedicated instances will not be impacted. Expected Impact: Increased latency and possible brief interruptions for shared cloud tenants Update: Completed first phase https://status.hatchet.run/incident/490008 Fri, 03 Jan 2025 21:00:30 -0000 https://status.hatchet.run/incident/490008#f61997ab0ea3ce64a87ab24b6673bbd8e63aee4bb5abe39b7104e0c0cd2feaeb We are performing essential database maintenance affecting shared cloud infrastructure. Dedicated instances will not be impacted. Expected Impact: Increased latency and possible brief interruptions for shared cloud tenants Update: Completed first phase https://status.hatchet.run/incident/476151 Wed, 11 Dec 2024 19:11:00 -0000 https://status.hatchet.run/incident/476151#3ec30ce98ea7423fe6e39f3fb5f968b959225ee1207da6a51bb25526de4fccbd The Hatchet API is currently experiencing increased API error rates. https://status.hatchet.run/incident/473127 Thu, 05 Dec 2024 21:30:00 -0000 https://status.hatchet.run/incident/473127#f56e0f284de50977cdca5d6a13cdb032f2b512ee14599a8e8a7150317109e37a Our API experienced elevated error rates from 21:30 to 21:33 UTC, caused by an automatic infrastructure upgrade. https://status.hatchet.run/incident/473127 Thu, 05 Dec 2024 21:30:00 -0000 https://status.hatchet.run/incident/473127#f56e0f284de50977cdca5d6a13cdb032f2b512ee14599a8e8a7150317109e37a Our API experienced elevated error rates from 21:30 to 21:33 UTC, caused by an automatic infrastructure upgrade.