Hey developers,
Earlier this year, we discussed some upcoming changes to our webhooks system. Today, we’re happy to share with you all an update on our progress, as well as what you can expect in the coming months.
Background
As we mentioned in the original post, one of our goals for redesigning webhooks is to address durability concerns, especially in mitigating catastrophic incidents (e.g., dropping events).
In short, many of our issues stem from our current configuration of Redis as the primary storage for events. As configured, Redis does not provide appropriate durability, which opens the door to potential data loss. Since our current infrastructure complicates deployment of a reconfigured Redis, our needs call for a durable storage replacement solution moving forward.
MemoryDB for event storage
While we’re continuing to stay on track with our goals, we’ve taken on a more expeditious approach to prevent the risk of data loss. Our team has been focused on making impactful repairs and optimizations to our existing infrastructure in an effort to solve durability issues more promptly.
With this approach, rather than migrating Redis to AWS DynamoDB, we’ve moved forward with AWS MemoryDB for Redis. MemoryDB provides a Redis-compatible, durable managed database service for our current architecture. Implementing MemoryDB guarantees that writes are persisted across database node restarts. This allows us to achieve both durability and recoverability guarantees for webhooks.
What to expect
Our team is already in the process of implementing the new storage solution, and we expect the move to be complete in January 2022. As previously mentioned, most changes are under the hood, and won’t require you to change the way your apps are built. Once we achieve durability, we anticipate the risk of catastrophic incidents and data loss should be virtually eliminated.
Again, we appreciate your patience as we work to improve the developer experience. We look forward to sharing more updates with you soon. In the meantime, feel free to share your questions and feedback here with us in this thread!
