@AndrewWong@Matt_Bramlage
No webhook events from Asana for 6h+.
We have multiple organizations, users, API keys, and OAuth.
None have received any events in webhooks since 05 Oct 03.13 UTC+1.
Asana has acknowledged the issue. Received a text about 15min ago. You can subscribe to these status updates using this link: Asana Status - Issue with webhooks
I think you were just one of the first to experience it @JFrentz
Hi all, thanks for reporting this issue! We are investigating and will update this thread with more information as we get it. As @LEGGO mentioned, you can subscribe to our status page to get alerts as soon as the issue is solved too.
As of this writing, the status page hasn’t changed, still reflects the issue. @JFrentz I don’t know about you but I am getting some webhooks for Flowsana - but can’t say that I’m getting all of the ones it’s supposed to be getting.
Volume-wise it looks like a regular Tuesday. We have a test indicating that some events have still not been received. Will have to dig a bit deeper to verify tomorrow.
Just wanted to drop by and say thank you for your patience today. The status page has been updated as @Phil_Seeman mentioned, but at the moment we’re still actively looking into the issue.
We’ll reply back when we can be more thorough about the issue and any next steps. In the meantime, feel free to continue sharing your experiences directly in this thread. We hope to have a more complete picture of the incident for you very soon.
Thanks again for your support this week. Our team has completed an investigation of Tuesday’s incident, and I just wanted to make sure we close the loop for everyone here.
Incident: An operation created a large backlog of events that was unable to be processed within our timeout. The event processing job was rescheduled in the failing state, causing corresponding workers to be stuck and resulting in severe delays for all event distribution. Full recovery to expected conditions took ~11 hours.
Impact: Events associated with this database were delayed. A subset of these events failed to be delivered because they aged out due to the delay. Of the events that were not delivered, only ~5-10% were customer events. No customer data was lost.
Movingforward: As a result of this incident, Asana is implementing changes to make our event distribution systems more resilient to cascading failures and high event volume.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We’re looking to have about 30 minutes of scheduled maintenance at 9:00am PDT on Saturday, October 16th. I’m happy to share that we’re continuing to make progress on what we’ve committed, and just need a brief period of down time as we improve our webhooks infrastructure. For more details, check out this post.