Webhooks: Scheduled Downtime on Saturday, January 15 from 8:00-8:30am PST

[Greg_Dardis]

Hi everyone,

We will have scheduled downtime for our webhooks and event streams on Saturday, January 15th. We anticipate that the maintenance will last about 30 minutes, from 8:00 am to 8:30 am Pacific Standard Time (PST).

During that time, webhooks may be briefly delayed, and any events generated during this time may be duplicated and sent to your webhooks/event streams twice. We chose delay/duplication of webhook events because we thought it would be less disruptive than a full downtime during this time period.

Check back here for updates, and feel free to leave any questions or comments directly in this thread.

Greg

[JFrentz]

@Greg_Dardis @Kem_Ozbek @Andrew_Wong
Thanks for all your efforts to upgrade the webhooks.

UPDATE: THIS ISSUE HAS BEEN RESOLVED since the original comment.
See details in post 7d after this one

Original post
Massive delays and missing events still, today.

In 2021, even with all the issues, 94% of all events were received within 1min. Since upgrades in December, this has gotten a lot worse. See attached charts from your client’s perspective.

On top of that, some events are lost forever(never arrive through webhooks). We recently implemented a fallback Quality Assurance process that runs every hour. It fetches all events from token and compares them with events received in webhooks during the same period, we then process missing events. As the webhook events are not reliable.

On the 6 Jan when we started to fetch from token, we got hundreds of events from 3 Jan which hadn’t arrived in 3 days, basically never sent in webhook.

Currently, 3% of all our recorded events are from our hourly token, instead of the stream, this process adds workload to Asana and on our side.

Our business and our Client’s business are dependant on this, but we get extremely little information, and you still don’t use status.asana.com to communicate the issues.

You are forcing us to spend time on solving the problems caused by this instead of continuing to develop our product and organization’s usage of Asana.

If you don’t want this in the public, please reach out to me personally and explain the status, plan, and what we can expect from Asana in the future.

[Phil_Seeman]

Wow, thanks for posting this, @JFrentz - not good news but important to know this.

(FYI Matt Bramlage is no longer with Asana; the other DevRel person to include in general would be @AndrewWong.)

[JFrentz]

Last week I received a nice email from @Kem_Ozbek where he informed me that they don’t see the same numbers as I previously posted 7d ago. He invited me to a chat where we would go through the numbers to work out a solution.

Based on that feedback we started to investigate the issue further and have resolved an issue on our side which was created while implementing the Quality Assurance process using sync token before Christmas.

These are the accurate stats for the past 6h.

So, Asana had its problems during 2021, upgraded 3 times in December, the week after those upgrades they had major issues. We couldn’t risk missing more events so we implemented a fail-safe quality assurance process using Sync tokens. While implementing that we caused our own issue, creating delays and missed events. As we saw Asana continuing with their maintenance and upgrades of Webhooks and event stream, we faulty assumed the issue was still on their side.

Asana is delivering webhooks and events with high quality and performance, and we are now happy with what we have.

Thanks once more @Kem_Ozbek, @AndrewWong, and @Greg_Dardis for your upgrades and for the personal service that you provide when needed.

[Phil_Seeman]

Hey @JFrentz, thanks for your detailed and forthcoming post. This is awesome on many fronts - that you’ve validated that webhooks are working well now, that you shared all of this info with us, and that we have an great team to work with at Asana, spearheaded by the awesome @Kem_Ozbek.