Alert: Asana is experiencing app issues and delays today (9/12) which may affect API usage (Update: resolved)

Asana is currently experiencing an issue due to a bug in a background component of Asana’s infrastructure. A recent deploy has caused a very small percentage of sessions (about 1 percent) to write data to our databases in an inconsistent way. Our engineers have rolled back to the last good deployment of Asana, but have stopped some of the background processes which handle things like mobile push notifications and webhooks. Additionally, a small percentage of API calls may show missing data or return 5XX error codes from server crashes for the next few hours until we’re fully recovered.

You can keep track of the progress of our stability team in fixing what data is affected by keeping an eye on trust.asana.com for updates, but our stability team does not anticipate turning our background job workers on until they manage to fix the data that is in an inconsistent state. They have to do this manually, which means their goal is to accomplish this by the end of the day.

We’re sorry if this disrupts your workflow, but we’re working hard on it. We expect to be able to fully recover. Thanks for your patience, and I’ll be sure to come back when we’re recovered with updates.


Edit:

We believe that this should be resolved for the majority of the data (with a small subset of data still being chased down) and webhooks and push notification behavior should be back to normal. Thanks for you patience, everyone!

4 Likes

Thanks for the heads up, this must feel very uncomfortable for your team as well when that happens :sweat:

Thanks @Matt_Bramlage! Marking this thread resolved as we’re fully operational again!