Ongoing OAuth app degradation when making batch requests

Hi everyone,

We have identified an issue with batch calls made to the API. The issue impacts mutating requests (POST, PUT, DELETE, etc.) made in a batch by OAuth apps, where these requests will fail with a 401 status code. Non-mutating requests (i.e. GET) included in the batch are unaffected. Individually made requests are also not affected. We believe the issue has been causing impact since February 12. We’ve identified the root cause and are currently working on deploying a fix. I will update this thread when the fix has been deployed. You can also monitor status at status.asana.com. We will post an incident report once we’ve completed our investigation.

We apologize for the inconvenience this has caused and appreciate your patience as we work on a fix.

UPDATE: Incident Report

Incident:

On 2024-02-22 we discovered a bug in our API where batched API requests would fail under certain circumstances. This bug was caused by a dependency upgrade (the same as the incident on 2024-02-08). We updated our status page with an incident describing this bug at 01:09 UTC and deployed a fix resolving the issue at 19:32 UTC.

Impact:

The bug impacted all customers accessing the API using batch API calls and an OAuth bearer token. PUT, POST and DELETE requests in the batch would return a 401 HTTP error status code, even with valid bearer tokens. The issue started on 2024-02-12. No data loss occurred during this time; non-batch requests and non-mutating requests included in a batch were not affected. The erroneously failed API requests from this period were not replayed at ingress after the fix was deployed.

Moving forward:

We are making enhancements to our testing framework so we can identify issues with requests like these (mutating batch requests) before deployment to production.

8 Likes

Hi folks,

This issue is now resolved. Your batch API requests should be succeeding as expected. Please feel free to respond here if you’re continuing to experience problems with batch calls.

We’ll follow up with a post-mortem incident report as soon as possible.

2 Likes

What would happen to old batch requests that had been failing. Will the system re-run them and process them as expected. We have been seeing many issues with our webhooks over the past few weeks.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Original post updated with incident report.

1 Like