Webhooks no longer work

[Garth]

@Stephan_Hodges here are more details on the issue and resolution.

@Kem_Ozbek

Hi everyone,

I wanted to close the loop on webhook delivery issue you’ve been experiencing. We believe this issue has now been resolved, but please do let me know if you notice any continuing problems with webhooks.

At around 5 pm PST yesterday, our engineering team identified the cause of the issue as bad data unintentionally introduced during the migration that took place last weekend. After developing a fix to correct instances of bad data, we paused webhook distribution between 8 pm - 9 pm PST and then again between 10:45 pm - 12:00 am PST to apply this fix in two stages. Note that during these periods events were delayed but not dropped. We unfortunately neglected to provide this information on our status page, which led to further confusion about the state of webhooks. Making matters worse, we also experienced a major outage on asana.com shortly after we resumed webhook delivery, adding to the confusion.

We will provide a full post-mortem report for both this incident as well as the incident earlier this week soon.

I want to once again apologize for the impact these issues have caused for you and your users. I know that providing timely and high-fidelity updates on our status page regarding the health of webhook delivery, in particular, is an area we’ve been falling short in, and this continues to cause confusion and frustration among many of you. I’ll personally see that we do better in this regard in the new year.

With the migration behind us, we are a major step closer to providing you with a more reliable and performant webhooks system. We appreciate your patience and partnership on this journey.