No webhooks from Asana

@AndrewWong @Matt_Bramlage
No webhook events from Asana for 6h+.
We have multiple organizations, users, API keys, and OAuth.
None have received any events in webhooks since 05 Oct 03.13 UTC+1.

No information on https://status.asana.com/.

Related to major DNS and SSL issues on the internet? Still would love confirmation on the Asana status page.

Will we receive all missing events?

3 Likes

same here, was about to raise ticket

3 Likes

Same here, No webhooks from Asana

2 Likes

Same here. Chaos reigns…

2 Likes

Any response from asana anyone? I got a confirmation that they recieved my ticket an hour ago but no response… Any responses to your tickets?

1 Like

@Phil_Seeman any information?

Asana has acknowledged the issue. Received a text about 15min ago. You can subscribe to these status updates using this link: Asana Status - Potential issue with webhooks

I think you were just one of the first to experience it @JFrentz :wink:

3 Likes

Hi all, thanks for reporting this issue! We are investigating and will update this thread with more information as we get it. As @Jerod_Hillard mentioned, you can subscribe to our status page to get alerts as soon as the issue is solved too.

2 Likes

As of this writing, the status page hasn’t changed, still reflects the issue. @JFrentz I don’t know about you but I am getting some webhooks for Flowsana - but can’t say that I’m getting all of the ones it’s supposed to be getting.

1 Like

Volume-wise it looks like a regular Tuesday. :+1: We have a test indicating that some events have still not been received. Will have to dig a bit deeper to verify tomorrow.

1 Like

Just to close the loop here in this thread, the webook status has been changed to Resolved. :partying_face:

2 Likes

Hi all,

Just wanted to drop by and say thank you for your patience today. The status page has been updated as @Phil_Seeman mentioned, but at the moment we’re still actively looking into the issue.

We’ll reply back when we can be more thorough about the issue and any next steps. In the meantime, feel free to continue sharing your experiences directly in this thread. We hope to have a more complete picture of the incident for you very soon.

Best,

Andrew

3 Likes

Hi everyone,

Thanks again for your support this week. Our team has completed an investigation of Tuesday’s incident, and I just wanted to make sure we close the loop for everyone here.

We’ve updated our status page with the following postmortem report:

Incident: An operation created a large backlog of events that was unable to be processed within our timeout. The event processing job was rescheduled in the failing state, causing corresponding workers to be stuck and resulting in severe delays for all event distribution. Full recovery to expected conditions took ~11 hours.

Impact: Events associated with this database were delayed. A subset of these events failed to be delivered because they aged out due to the delay. Of the events that were not delivered, only ~5-10% were customer events. No customer data was lost.

Moving forward: As a result of this incident, Asana is implementing changes to make our event distribution systems more resilient to cascading failures and high event volume.

Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

Feel free to reply here directly if you have more questions about the incident. Again, we appreciate your understanding as we continue to optimize the durability and stability of webhooks.

We’re looking to have about 30 minutes of scheduled maintenance at 9:00am PDT on Saturday, October 16th. I’m happy to share that we’re continuing to make progress on what we’ve committed, and just need a brief period of down time as we improve our webhooks infrastructure. For more details, check out this post.

Thank you and have a great weekend, everyone!

Andrew

5 Likes