@Ross_Grambo I just hit another instance of this issue. I recreated the webhook and that solved it.
I think I’m gonna have to delete/recreate all of my Flowsana user webhooks tonight…
Sorry Phil 
it sounds like they made progress on finding where the issue occurred but it doesn’t sound like they’ve landed on an easy fix.
Hey! Thanks for the suggestion. I did not. I’ll try to do that. We have almost 1k Asana integrations at Hubstaff 
. Some of them have a lot of projects. Might take a while until we get to recreate them all.
Odd though that the status page doesn’t mention anything about this.
The /events endpoint seems to work fine, as we fallback to this now until the webhooks are fixed.
1 Like
Is a fix possible with existing webhooks, or are we looking at taking the approach Phil suggested which is to re-create all webhooks. As you can imagine with thousands of customers, and each customer having hundreds of hooks, it’s not ideal to be re-creating them all. Especially if we think the issue might still affect newly created hooks.
That’s frustrating Sorry @Maggie_Reddi. It seems like re-creating webhooks only temporarily resolves the issue. If you want something working asap, you’re probably best off using /events instead.
I’ll keep posting here as I get more info.
Hate to be a bother, but any update on this? Seems like more and more of our users are experiencing this issue. Using the events endpoint does not line up with our current architecture.
Hey @Maggie_Reddi,
Could you send me your webhook gid and if possible the resource that missed an event?
Here is some information for you. Hope this helps:
Hook Id: 1181137944038193
Project Id: 1181105591860870
If I retrieve the hook, I see the following details:
“created_at”: “2020-06-19T07:26:55.013Z”,
“last_failure_at”: null,
“last_failure_content”: “”,
“last_success_at”: “2020-10-13T23:20:19.211Z”,
Notice that there are no failure details. The last success date is also well before the creation of new tasks within this project. Here are a few tasks, and their creation date.
Task 1
“gid”: “1198211115781435”,
“created_at”: “2020-10-14T02:36:48.010Z”
Task 2
“gid”: “1198211840394755”,
“created_at”: “2020-10-14T03:46:39.485Z”
Let me know if there is anything else that will help with this.
1 Like
Thank you everyone! We found the cause and I’ll be doing a write-up in the near future about what happened. If you want your webhooks fixed ASAP, then as said before, you can re-create them.
We’re looking into reviving broken webhooks on our side of things. I’m waiting for the answer to this question before posting the write-up. If we do revive webhooks, it’s unlikely that they will catch events that they have missed during this timeframe.
Let me know if you have any additional questions in the meantime.
4 Likes
Excellent news!
Here’s a link to that forum post: A Problem with Webhooks & Events. We were unable to find a quick way to revive webhooks, so the solution so far is to recreate any webhook you think was affected.
Thanks Ross. Any chance we could get a list of affected webhooks by our client id? If not, we would likely need to go through and re-create all of our webhooks as we are not aware of which ones are working or not.
That’s exactly was I was hoping to get from this investigation but it looking like a no. The nodes push data to the webhook and the way to find out would be scanning all of our nodes and cross referencing our webhooks to see which ones didn’t have data pushed to them.
While this sounds feasible, it seems like the AWS infrastructure handling these nodes doesn’t offer an easy way to do this. The API team is still looking into it and I’ll let you know if they come up with a way to list affected ids. Sadly, it wouldn’t surprise me if that investigation wasn’t complete until late next week and/or the outcome of the investigation might simply be “no we can’t get a list”.
1 Like