Missing events and no "sync_error" recieved on that webhook endpoint.

Thank you for the update @Matt_Bramlage, it’s really valuable to us. :handshake:

We just experienced a 33 min absence of incoming webhooks. :frowning:
First time in 5 days.

Events happening in Asana during this time have not been sent to Cloudmate webhook endpoint yet, but newer events have. This feels like a lost event to us. From experience, these can take up to 48h to be delivered.

I’m sharing our perspective and insight in hope of resolving this issue faster. Let us know if there is anything else we can do to be of service here.

During this post, the event stream stopped again and is currently down.

!Update: Now we seem to be back on track, longest recorded delay so far today is 82 min.

1 Like

Hi @JFrentz,

So that I might see if/how that corresponds to my incoming webhooks today, can you tell me what time zone is on the X axis?

Also, don’t know if it’s related but I’ve gotten 9 sync_errors in the past 24 hours or so, after not getting any in the few days prior to that.

1 Like

Hi @Phil_Seeman , we are GMT +2, Sweden. :+1:
We don’t see any correlation between sync_errors and the total absence/delay of webhooks. Sync_error count today is one, and unusually close to the time when delay happened today.

We only receive sync_error on one subscription type(many projects). We are evaluating if it’s related to us subscribing to projects while they are being generated(filled with tasks and so on). Do you see any correlation to that assumption?

1 Like

I’m not seeing any gaps today - this is last 4 hours, in UTC time:

I don’t think so since that would be happening daily for me so I would then expect to see sync errors daily.

1 Like

@JFrentz,

Oh well, so much for that burst of optimism on our part…

1 Like

Hey @Phil_Seeman and team!

We’ve recently experienced performance issues with webhooks and want to share a quick update.

Recent issues with the infrastructure that supports our webhooks and events systems has resulted in delays or losses of events over the last 2 days. We believe we’ve identified the root cause of the most recent incidents, we have recovered, and are working on ensuring they don’t happen again. We plan to share more information about the recent outage, and our plans to improve webhooks and events, in the near future.

In the meantime, we recommend that you subscribe to our status page to be alerted to incidents as they happen, which now includes a separate line for webhooks and events. We also recommend that you implement a fallback of scanning resources periodically to catch up with any changes missed from lost events.

Thanks for your patience as we work through this. We’ll get back to you shortly with more information!

7 Likes

Sorry to just write when it is bad. Most of the time Asana is awesome!

We have not received any webhooks for the past 3 hours.

Asana support has been responsive but the escalation of priority is slow. No info at status.asana.com but support have confirmed resolving some issue. Problem remains 3h+ now

Confirmed @JFrentz, I’m seeing webhook issues in my app as well. I think they may be coming through but very delayed, not sure. I did one test last night - I got the event but 5 hours late. I did two more tests this morning, have not received the events yet, will see if I get them delayed as well.

cc: @Jeff_Schneider @Matt_Bramlage @Ross_Grambo

1 Like

Thanks for confirmation @Phil_Seeman :raised_hands:
We had a 4h delay this morning. Delay has been shorter and shorter during the day, just recorded a 45 min delay on some event. Impact organization’s performance for the day.

Hey @JFrentz,

I just ran a few tests and did not see any delays - how is your current experience?

1 Like

Looking good so far this morning. :+1:

2 Likes

Hi folks,

My name is Kem and I’m the newest member of the Developer Relations team here at Asana. I’m writing to provide some additional information on the webhooks issue from yesterday. As some of you have noticed, we experienced delays yesterday in sending out webhook notifications. In some cases, these delays were significant. The issue was resolved, and the webhooks started firing on a timely fashion again, by about 8:30 am PDT on Thursday.

First, I want to acknowledge and sincerely apologize for the disruptions incidents like this cause for your apps and users. Rest assured that we take reliability of webhooks very seriously, and working diligently to resolve the underlying issues. Please see this post for more information about what we’re doing in this regard.

Second, I want to make a commitment to providing regular updates as we hit major milestones in this remediation effort. We value your feedback in this process, so please feel free to drop a comment with additional perspective on how these issues have been affecting you.

Finally, I want to address the fact that our status page (status.asana.com) showed no disruptions to service while these delays took place. This was simply because no system experienced an outage: our jobs queue, which works on tasks in the background to keep Asana responsive and is shared by many Asana systems including webhooks, got backed up. So while it was processing jobs, we had delays on some jobs like webhooks. We’ve identified the root cause, and are working with our infrastructure teams to mitigate this going forward. That being said, we recognize that we can do better to inform you about the status of webhooks – down, delayed, or operating as expected – and we’re looking into ways we might be able to do just that on status.asana.com.

Thank you for being a valued part of our developer community. We appreciate your patience as we work to empower your apps with top-notch webhooks performance, and welcome all feedback in the meantime.

5 Likes

Hi @Kem_Ozbek,

Welcome and thanks for this message! This is great to hear, on all of the aspects you address…

I know you’ve already heard some of my input on the impact these webhook outages and delays have on my Flowsana app and its customers, so I won’t beat that into the ground here. :slight_smile:

It will be really valuable to be kept informed about the progress of the webhook rebuild process; much appreciated for that commitment.

And I know you’ve already heard my feelings about status.asana.com, so it’s good to know you’re looking at how to provide info on delays. I get and appreciate it’s not simple since delays don’t generate an actual error condition (at what point of event backup do you call it an issue worth reporting on the status page - 5 minutes, an hour, …?), but it’s good to know you’re looking into how you might provide some feedback there. Again, thanks!

1 Like

Hi! We still had some delayed events last Friday 2021-05-27T19:00:00Z. It strongly affects our working processes cause lags makes from hours to days.

Hi and welcome @Kem_Ozbek!
We don’t receive any webhooks this morning. 1 hour now. Nothing on status.asana.com

2 Likes

Still, 0 webhooks are sent to us.
Issue confirmed by Asana Support. Dev team troubleshooting.
Nothing is shown on the status page.

We are up and running again, ~2,5h downtime today.

1 Like

@JFrentz,

This is a tough question to know the answer to, since you don’t really know what you may have missed, but can you tell if you lost events during the outage, or if you’re receiving all the events but delayed?

1 Like

Just dropping in to confirm that we had delays with event distribution early this morning. The issue was resolved and webhooks returned to normal latencies for event delivery by about 6:25 am PDT.

I’m afraid I don’t have much more to add beyond that, but I will reiterate the commitments from my earlier post, including being clearer about latency issues on status.asana.com. Please stay tuned; we greatly appreciate your patience.

3 Likes

Today we experienced two periods of webhook silence.

UTC +2

  1. 07.04 - 07.35
  2. 08.25 - 09.16

All event seems to have been delivered since. No information on the status page.