Webhooks no longer work

[Axel_Marciano]

Hi, we are not receiving webhooks events since Friday with the config :

{
action: ‘added’,
resource_type: ‘story’,
resource_subtype: ‘section_changed’,
}

[Mikolaj_Kowal]

same situation here, both of our webhooks stopped receiving events. last event was received on 2021-12-20T00:38:48.866Z

[Rebecca_McGrath]

Hi folks, thanks for flagging this! This has been raised with our Developer team - I’ll keep you updated!

[avinash4]

Asana Webhooks are not working for Tasks older than 3 days. I have double checked that the web hook exists and points at my server. Below is the response to the “get webhook” API:

Webhooks are working fine for tasks not older than 3days

[JFrentz]

@Axel_Marciano @Mikolaj_Kowal @avinash4

We can confirm that @Phil_Seeman solution to create a new subscription is a workaround that enables you to retrieve all new events.

You can have multiple subscriptions on the same project (using parameter on target eg. /target/:projectId?v=2), but it might result in duplicate events when Asana has resolved the issue on their side. On our side we are working on handling duplicate events if the old subscription would start working again.

We have raised the same issue here: Webhooks: Scheduled Downtime in December - #8 by Jonas_O as we see that it’s likely to be related.

[Kem_Ozbek]

We believe this issue is now resolved.

[Garth]

Is there a certain amount of time we should expect to wait for the fix to take effect before the webhooks distribution is back to normal? I am still seeing the issue on my end.

[Stephan_Hodges]

Thanks for this information. It was useful.

[Garth]

@Stephan_Hodges here are more details on the issue and resolution.

@Kem_Ozbek

Hi everyone,

I wanted to close the loop on webhook delivery issue you’ve been experiencing. We believe this issue has now been resolved, but please do let me know if you notice any continuing problems with webhooks.

At around 5 pm PST yesterday, our engineering team identified the cause of the issue as bad data unintentionally introduced during the migration that took place last weekend. After developing a fix to correct instances of bad data, we paused webhook distribution between 8 pm - 9 pm PST and then again between 10:45 pm - 12:00 am PST to apply this fix in two stages. Note that during these periods events were delayed but not dropped. We unfortunately neglected to provide this information on our status page, which led to further confusion about the state of webhooks. Making matters worse, we also experienced a major outage on asana.com shortly after we resumed webhook delivery, adding to the confusion.

We will provide a full post-mortem report for both this incident as well as the incident earlier this week soon.

I want to once again apologize for the impact these issues have caused for you and your users. I know that providing timely and high-fidelity updates on our status page regarding the health of webhook delivery, in particular, is an area we’ve been falling short in, and this continues to cause confusion and frustration among many of you. I’ll personally see that we do better in this regard in the new year.

With the migration behind us, we are a major step closer to providing you with a more reliable and performant webhooks system. We appreciate your patience and partnership on this journey.