Hi all! Over the past year, we’ve received sporadic reports of developers not being able to list the webhooks in a particular workspace—requests to GET /webhooks were returning empty arrays in certain conditions instead of the list of webhooks, even though getting individual webhooks by ID showed that they did indeed exist. Through all the information we collected through the forum, support tickets, and direct conversations with developers, we were never able to reproduce and trace the source of the bug… until just last week!
The conditions for triggering the bug are:
- You have created a webhook on a resource.
- You lose access to the resource after creating the webhook. Losing access to the resource means that the webhook will eventually delete itself (when it next attempts to deliver data).
- Before the webhook deletes itself, you attempt to fetch a list webhooks that would include the webhook created in step (1).
This issue is exacerbated by the unpredictability of when the webhook will attempt to delete itself, as well as any edge cases that cause that self-deletion to fail.
The result is that our data query system detects that you don’t have access to a subset of the data (the resource of one webhook) and, because of how the query was defined, it returns no webhooks at all. The true source of the bug, however, was on an even lower level than that—our access control rules weren’t properly filtering out webhooks where the user had lost access to the resource, which meant other access controls around the visibility of the resource kicked in and caused the whole response to be blocked.
As of today, we have deployed the fix for this bug and you should no longer experience “missing webhooks” or other unexpected behavior with GET /webhooks. If you still are encountering problems, please let us know as they may indicate yet another bug we need to address.
Special thanks to @David_Haxton for tracking down and fixing the bug!
! Well done 