During these times, we will be migrating event storage to MemoryDB. As such, no webhooks will be sent within these windows (i.e., they will be delayed, as will events in the event stream). We anticipate that these improvements will allow us to achieve durability guarantees for webhooks.
As always, thank you for your support, and feel free to check back here for any updates along the way!
It looks like some existing webhook connections have broken. I’ve deleted and recreated a few webhooks that had stopped working and they appear to be working again. Try doing that with your webhooks and see if that gets them going again. Let us know the outcome.
@JFrentz are you having any webhook issues today like this?
It’s been confirmed by asana that there is an issue since yesterday.
I have a few thousand project subscriptions and would love a response from asana before I recreate them.
This is the most critical issue we have had with Asana since we started using it 6 years ago. On top of that very limited information and response from Asana and nothing on the status page.
Hello everyone, and thank you for your patience as we investigate this incident with webhooks.
As predicted, the issue is related to the recent migration of event storage to MemoryDB discussed at the top of this thread. An error during the migration caused webhooks subscriptions for resources created prior to the migration to start failing on Saturday. We have identified the root cause and we’re working to fix it as soon as possible.
In the meantime, I can confirm recreating webhook subscriptions that stopped working as a workaround as suggested by @Phil_Seemanhere and by @JFrentzhere.
We have updated our status page to reflect the current state of webhooks, and you can track progress on the incident here.
We sincerely apologize for all the inconvenience this incident has caused for you and your users. We hope to have it resolved soon.
We’ve deployed a fix and we believe this issue is now resolved; webhooks distribution should be back to normal. We updated the status page to reflect this. Please let me know if you continue to experience any issues with webhooks / events.
