Incident affecting events and webhooks on March 7th

Last night, our infrastructure team observed intermittent delays and decreased remote job completion due to a bug.

To fix the issue, they started new jobs to start processing events going forward and kept the old jobs and data to be able to deliver these events after the bug is fixed. You may see some old events delivered intermittently as our job system recovers. Most of the delayed events won’t trigger webhooks as they were triggered by deletion jobs, not user action.

As explained in the incident report, the bug prevented our infrastructure from processing some webhooks events between 2024-03-07 00:20 and 2024-03-07 04:10. All events after this time were processed. We are working to deliver events from the timeframe when the infrastructure was experiencing the issue.

Thank you for your patience while any delayed events are being delivered. We apologize for the inconvenience.

Asana DevRel


Hey @Jeff_Schneider ! Great job and thanks for the updates on and here. I guess you are asleep but hopefully your team is aware that the problem seems to be back today. We don’t get all the events from Asana. Some of the events that we have measured have 30-50min delay. Also the web app just crashed and needed reloading for me now. :pray:

1 Like

Yeah its still slow. Confirming the same @JFrentz is experiencing.

1 Like

@JFrentz Did you report it to Suggest you do that if you haven’t yet.

1 Like

Thank you @Phil_Seeman! FYI see attached image. :slight_smile:

Oh, I guess that support email has been merged into their regular support email!

@Jeff_Schneider @John_Baldo @Kem_Ozbek

1 Like

Thank you @Phil_Seeman! We still miss events and have delays. I was able to create a support ticket using the chat they referred to in the email but no response yet, 4h ago. We have have experienced this problem for 15 hours now.

It seems like events are not sent for a long time then some come in a batch even if there are many minutes in between. Some event where delayed 45min now others just 12min but they come at the same time.

Still no info on the status page:
Screenshot 2024-03-08 at 16.17.52

@JFrentz, I’ve also escalated your report internally, our team will be in touch as soon as possible!


@JFrentz, @Jonas_O and any others seeing anomalies with event delivery: I would love to help investigate if you can provide any or all of:

  • webhook ID
  • resource ID
  • event ID

If you happen to have the entire event payload that would be fantastic, but just those IDs will give us enough to trace through our pipeline. Thanks for your help as we investigate this issue

1 Like

Thank you @Marie and @Aaron_Gutierrez! I’ve sent additional details in a DM to @Aaron_Gutierrez . :slight_smile: