They worked in my brief testing but I also just had a report from a Flowsana customer that he is not working, so I suspect there is still an issue. I’m contacting Asana.
EDIT:
I’ve notified my Asana contacts and they are having the Asana engineers look into things now.
Hey @Kem_Ozbek, thanks for your updates.
We receive webhooks on our new subscriptions from yesterday, but not on the old ones.
We estimate that we have about 30 000 events that are lost. Will require a lot to fetch and processes them from the API. No sync token available. @Kem_Ozbek can you provide an option to send those retroactive?
My webhooks did not start receiving events. I had to do workaround - recreate webhooks.
Hi everyone. Just wanted to let you know that we’re continuing our investigation into the lingering issues with webhook delivery. At this time, we recommend that you recreate any webhooks that you suspect might not be receiving events since that’s been demonstrated to resolve the issue.
If you’ve identified a webhook subscription that’s not working as expected, I’d appreciate if you can let us know the webhook id before deleting it to help with our investigation.
Unfortunately we’re not able to resend webhooks that would have fired during the incident.
I’ll post another update on our investigation as soon as we have more to share.
As of this writing, webhooks appear to be completely down - I am getting nothing at all. I’ve sent a message to Asana API support.
Note that the Asana status page indicates “degraded performance” but I’d say it’s worse than that. Also the status page lists my workaround of recreating webhooks as a solution, but I just tried it and it’s not helping n this case - so don’t bother trying that.
Yes unfortunately the problem seems to grow, no webhooks.
FWIW just got confirmation from API support that engineers are working on the outage.
I just started getting some, can’t say yet if its 100% back up or not.
How about you, @JFrentz , are you getting some now?
Thank you @Phil_Seeman! Yes, we have a few now.
Still looks broken though.
Hi everyone,
I wanted to close the loop on webhook delivery issue you’ve been experiencing. We believe this issue has now been resolved, but please do let me know if you notice any continuing problems with webhooks.
At around 5 pm PST yesterday, our engineering team identified the cause of the issue as bad data unintentionally introduced during the migration that took place last weekend. After developing a fix to correct instances of bad data, we paused webhook distribution between 8 pm - 9 pm PST and then again between 10:45 pm - 12:00 am PST to apply this fix in two stages. Note that during these periods events were delayed but not dropped. We unfortunately neglected to provide this information on our status page, which led to further confusion about the state of webhooks. Making matters worse, we also experienced a major outage on asana.com shortly after we resumed webhook delivery, adding to the confusion.
We will provide a full post-mortem report for both this incident as well as the incident earlier this week soon.
I want to once again apologize for the impact these issues have caused for you and your users. I know that providing timely and high-fidelity updates on our status page regarding the health of webhook delivery, in particular, is an area we’ve been falling short in, and this continues to cause confusion and frustration among many of you. I’ll personally see that we do better in this regard in the new year.
With the migration behind us, we are a major step closer to providing you with a more reliable and performant webhooks system. We appreciate your patience and partnership on this journey.
We appreciate the more detailed insight on the issue @Kem_Ozbek. Hopefully, the worst is behind us and we can have a smooth rest of the week prior to the holiday.
Thanks very much for this info, @Kem_Ozbek; excited for the improved infrastructure that’s now in place.