Last night, our infrastructure team observed intermittent delays and decreased remote job completion due to a bug.
To fix the issue, they started new jobs to start processing events going forward and kept the old jobs and data to be able to deliver these events after the bug is fixed. You may see some old events delivered intermittently as our job system recovers. Most of the delayed events won’t trigger webhooks as they were triggered by deletion jobs, not user action.
As explained in the incident report, the bug prevented our infrastructure from processing some webhooks events between 2024-03-07 00:20 and 2024-03-07 04:10. All events after this time were processed. We are working to deliver events from the timeframe when the infrastructure was experiencing the issue.
Thank you for your patience while any delayed events are being delivered. We apologize for the inconvenience.
Hey @Jeff_Schneider ! Great job and thanks for the updates on status.asana.com and here. I guess you are asleep but hopefully your team is aware that the problem seems to be back today. We don’t get all the events from Asana. Some of the events that we have measured have 30-50min delay. Also the web app just crashed and needed reloading for me now.
Thank you @Phil_Seeman! We still miss events and have delays. I was able to create a support ticket using the chat they referred to in the email but no response yet, 4h ago. We have have experienced this problem for 15 hours now.
It seems like events are not sent for a long time then some come in a batch even if there are many minutes in between. Some event where delayed 45min now others just 12min but they come at the same time.
@JFrentz, @Jonas_O and any others seeing anomalies with event delivery: I would love to help investigate if you can provide any or all of:
webhook ID
resource ID
event ID
If you happen to have the entire event payload that would be fantastic, but just those IDs will give us enough to trace through our pipeline. Thanks for your help as we investigate this issue
