Webhooks: Scheduled Downtime on Saturday, January 15 from 8:00-8:30am PST

Hi everyone,

We will have scheduled downtime for our webhooks and event streams on Saturday, January 15th. We anticipate that the maintenance will last about 30 minutes, from 8:00 am to 8:30 am Pacific Standard Time (PST).

During that time, webhooks may be briefly delayed, and any events generated during this time may be duplicated and sent to your webhooks/event streams twice. We chose delay/duplication of webhook events because we thought it would be less disruptive than a full downtime during this time period.

Check back here for updates, and feel free to leave any questions or comments directly in this thread.

Greg

4 Likes

@Greg_Dardis @Kem_Ozbek @Andrew_Wong
Thanks for all your efforts to upgrade the webhooks.

Massive delays and missing events still, today.

In 2021, even with all the issues, 94% of all events were received within 1min. Since upgrades in December, this has gotten a lot worse. See attached charts from your client’s perspective.

On top of that, some events are lost forever(never arrive through webhooks). We recently implemented a fallback Quality Assurance process that runs every hour. It fetches all events from token and compares them with events received in webhooks during the same period, we then process missing events. As the webhook events are not reliable.

On the 6 Jan when we started to fetch from token, we got hundreds of events from 3 Jan which hadn’t arrived in 3 days, basically never sent in webhook.

Currently, 3% of all our recorded events are from our hourly token, instead of the stream, this process adds workload to Asana and on our side.

Our business and our Client’s business are dependant on this, but we get extremely little information, and you still don’t use status.asana.com to communicate the issues.

Asana - Move Along People Nothing to see here

You are forcing us to spend time on solving the problems caused by this :fire: instead of continuing to develop our product and organization’s usage of Asana.

If you don’t want this in the public, please reach out to me personally and explain the status, plan, and what we can expect from Asana in the future.

:pray:

4 Likes

Wow, thanks for posting this, @JFrentz - not good news but important to know this.

(FYI Matt Bramlage is no longer with Asana; the other DevRel person to include in general would be @AndrewWong.)

1 Like