Hi folks,
My name is Kem and I’m the newest member of the Developer Relations team here at Asana. I’m writing to provide some additional information on the webhooks issue from yesterday. As some of you have noticed, we experienced delays yesterday in sending out webhook notifications. In some cases, these delays were significant. The issue was resolved, and the webhooks started firing on a timely fashion again, by about 8:30 am PDT on Thursday.
First, I want to acknowledge and sincerely apologize for the disruptions incidents like this cause for your apps and users. Rest assured that we take reliability of webhooks very seriously, and working diligently to resolve the underlying issues. Please see this post for more information about what we’re doing in this regard.
Second, I want to make a commitment to providing regular updates as we hit major milestones in this remediation effort. We value your feedback in this process, so please feel free to drop a comment with additional perspective on how these issues have been affecting you.
Finally, I want to address the fact that our status page (status.asana.com) showed no disruptions to service while these delays took place. This was simply because no system experienced an outage: our jobs queue, which works on tasks in the background to keep Asana responsive and is shared by many Asana systems including webhooks, got backed up. So while it was processing jobs, we had delays on some jobs like webhooks. We’ve identified the root cause, and are working with our infrastructure teams to mitigate this going forward. That being said, we recognize that we can do better to inform you about the status of webhooks – down, delayed, or operating as expected – and we’re looking into ways we might be able to do just that on status.asana.com.
Thank you for being a valued part of our developer community. We appreciate your patience as we work to empower your apps with top-notch webhooks performance, and welcome all feedback in the meantime.