Missing webhook events for tasks

Yep. That is correct.

And we got 5 more customers in the past 18 hrs complaining about things not syncing w/ Asana.

@Ross_Grambo,

I’m not sure if it’s related but I also have an occurrence where there was a change made to a task and my app didn’t seem to receive an event for it.

Here is the info if it helps:

Task gid = 1195951499091229
Webhook gid = 1192893260436310
Story gid representing the section-change event that I didn’t get notified about = 1195963942967398

Another quick update: 2 more customers complained about it since the last time I posted.

@Ross_Gambo,

We’ve had several of our clients experiencing similar issues recently as well. If it helps, I can provide some sample webhook gids. However, we have since deleted and re-created the hooks and that seems to have resolved the issue.

One interesting thing we noticed is that the last success date and the last error date are both not updated, even though there are items that should have generated a notification. It appears to have started for us on September 21st.

1 Like

I have the same experience as @Maggie_Reddi - regarding the problematic webhook I mentioned above, I deleted and recreated it and it seems to be working now.

@Raul_Popadineti, have you tried that to see if it gets your webhooks working again?

Our API team was able to find the events and see that they were not sent via webhook. They’re looking into the cause now.

Again, sorry that this is happening. Has anyone noticed this issue happening for the /events endpoint? /events and webhooks use a lot of the same underlying infrastructure, so I’m curious if this is only happening for webhooks.

@Ross_Grambo I just hit another instance of this issue. I recreated the webhook and that solved it.

I think I’m gonna have to delete/recreate all of my Flowsana user webhooks tonight…

Sorry Phil :frowning: it sounds like they made progress on finding where the issue occurred but it doesn’t sound like they’ve landed on an easy fix.

Hey! Thanks for the suggestion. I did not. I’ll try to do that. We have almost 1k Asana integrations at Hubstaff :sweat_smile:. Some of them have a lot of projects. Might take a while until we get to recreate them all.

Odd though that the status page doesn’t mention anything about this.

The /events endpoint seems to work fine, as we fallback to this now until the webhooks are fixed.

1 Like

Is a fix possible with existing webhooks, or are we looking at taking the approach Phil suggested which is to re-create all webhooks. As you can imagine with thousands of customers, and each customer having hundreds of hooks, it’s not ideal to be re-creating them all. Especially if we think the issue might still affect newly created hooks.

That’s frustrating :frowning: Sorry @Maggie_Reddi. It seems like re-creating webhooks only temporarily resolves the issue. If you want something working asap, you’re probably best off using /events instead.

I’ll keep posting here as I get more info.

Hate to be a bother, but any update on this? Seems like more and more of our users are experiencing this issue. Using the events endpoint does not line up with our current architecture.

Hey @Maggie_Reddi,

Could you send me your webhook gid and if possible the resource that missed an event?

Here is some information for you. Hope this helps:

Hook Id: 1181137944038193
Project Id: 1181105591860870

If I retrieve the hook, I see the following details:

“created_at”: “2020-06-19T07:26:55.013Z”,
“last_failure_at”: null,
“last_failure_content”: “”,
“last_success_at”: “2020-10-13T23:20:19.211Z”,

Notice that there are no failure details. The last success date is also well before the creation of new tasks within this project. Here are a few tasks, and their creation date.

Task 1
“gid”: “1198211115781435”,
“created_at”: “2020-10-14T02:36:48.010Z”

Task 2
“gid”: “1198211840394755”,
“created_at”: “2020-10-14T03:46:39.485Z”

Let me know if there is anything else that will help with this.

1 Like

Thank you everyone! We found the cause and I’ll be doing a write-up in the near future about what happened. If you want your webhooks fixed ASAP, then as said before, you can re-create them.

We’re looking into reviving broken webhooks on our side of things. I’m waiting for the answer to this question before posting the write-up. If we do revive webhooks, it’s unlikely that they will catch events that they have missed during this timeframe.

Let me know if you have any additional questions in the meantime.

4 Likes

Excellent news!

Here’s a link to that forum post: A Problem with Webhooks & Events. We were unable to find a quick way to revive webhooks, so the solution so far is to recreate any webhook you think was affected.

Thanks Ross. Any chance we could get a list of affected webhooks by our client id? If not, we would likely need to go through and re-create all of our webhooks as we are not aware of which ones are working or not.

That’s exactly was I was hoping to get from this investigation but it looking like a no. The nodes push data to the webhook and the way to find out would be scanning all of our nodes and cross referencing our webhooks to see which ones didn’t have data pushed to them.

While this sounds feasible, it seems like the AWS infrastructure handling these nodes doesn’t offer an easy way to do this. The API team is still looking into it and I’ll let you know if they come up with a way to list affected ids. Sadly, it wouldn’t surprise me if that investigation wasn’t complete until late next week and/or the outcome of the investigation might simply be “no we can’t get a list”.

1 Like