Thank you for your patience everyone. Just swinging by again to follow up on my post from last week.
During the initial investigation, our team found that a small percentage of our webhook delivery calls gave sync errors during the incident. This meant that events weren’t delivered properly and were otherwise “missing.” Note that since we had rolled back the deployment that Friday afternoon, it’s possible that you may have seen the missing events in the next sync of each webhook.
Our team is currently having a retrospective on the incident, with the goal of creating action items and process changes to:
- Identify the issue sooner (and rollback any problems faster)
- Prevent issues like this from happening in the first place
Again, thank you everyone for your patience in all this! Please keep sharing your experiences as we work to optimize our webhooks’ durability and performance.