Latest on webhook improvements

Hello developers,

I’m here today with another update on webhooks.

If you’ve been following the forum over the past year, you may know that throughout 2021, we saw our webhook system occasionally suffer outages. In most cases these would manifest as delayed webhook notifications, but in some cases they also involved event loss.

In April we laid out a plan for making webhooks more reliable and we’ve been hard at work since then:

  • In December, we hit a major milestone by completing the migration of our event storage to a new database infrastructure, MemoryDB, as we had outlined in a previous post. With this improvement we now consider event loss issues to be resolved.

  • We also made incremental improvements to our event distribution infrastructure that render event delays significantly less likely and more limited in scope if and when they do happen.

  • Finally, we made some process changes to ensure we remain proactive in notifying you about incidents on the forum and through our status page. While we don’t expect to see many of these in 2022, we want to make sure that you first find out about them from us and not your users.

Based on these improvements, we are modifying our standing recommendation for developers to also build a backup system that polls resources directly for changes as a best practice. We now believe that this workaround is needed only if your system requires strong guarantees and cannot tolerate any delayed events, regardless of how rare such incidents might be.

With our improvement plan for webhooks now completed, we’re shifting our development focus in 2022 to other areas of our API infrastructure and developer product. We remain committed to providing you with the most reliable and performant webhooks system possible, and will be looking for opportunities to reduce the likelihood of incidents even further in the future.

As always, please drop a comment below with any questions.

4 Likes

Thanks, @Kem_Ozbek, excellent news.

In addition to the technological changes which we very much appreciate, I also wanted to thank you for your work in making improvements to notifications per your last bullet point!

3 Likes