Remote server timeout when creating webhooks

Hi there,

For a project of ours we’ve integrated the Asana API into a basic Symfony app. This has been working correctly for about a month but now some issues are arising, specifically with creating webhooks. Here are the steps we undertake:

Create webhook

POST /webhooks

resource: "1118598799868276"
target: "https://app.sbsbelgium.sumoapp.be/webhook/1118598799868276"

Confirmation handshake

When logging the request on the given target we see headers that look like these:

Connection:     close
Content-Length: 2
Content-Type:   application/json
Host:           app.sbsbelgium.sumoapp.be
X-Hook-Secret:  5a2ee55dea34b27b627bc97a5ae943a5
X-Php-Ob-Level: 1 

When recreating above request in Postman we see our server returns the following information in about 200ms:

HTTP/1.1 200
status: 200
Date: Mon, 15 Apr 2019 06:28:39 GMT
Server: Apache/2
X-Powered-By: PHP/7.2.14
X-Hook-Secret: 5a2ee55dea34b27b627bc97a5ae943a5
Cache-Control: no-cache, private
Upgrade: h2,h2c
Connection: Upgrade, Keep-Alive
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Length: 24
Keep-Alive: timeout=2, max=100
Content-Type: application/json

Initial request response

However, when receiving the response of the initial request, Asana returns following error:

HTTP/1.1 400

The remote server did not respond within 10000 milliseconds.

Of course I cannot replicate a request of Asana to our webserver but it seems odd that Asana does not receive a response within 10000 milliseconds when I receive one in 200.

Please advise

Welcome @Verkoyen_Tijs!

If you are receiving that error, it means our API is not getting any response from the Post request handshake.

I think the best approach would be to enable debugging on the request received by your Symfony app in order to ensure that it’s not having any issues processing it, and ensure that it completely finishes the process of sending a response.

Let me know if this gives us any new information.

1 Like

@Ross_Grambo I’ve enabled debugging and added some logging to the controller receiving the webhook handshake request, here’s what I found:

# Request from Asana
[2019-04-16 10:54:15] request.INFO: Matched route "app_webhook_receiver". {
    "route":"app_webhook_receiver",
    "route_parameters":{
        "_route":"app_webhook_receiver",
        "_controller":"App\\Controller\\WebhookReceiver::webhookReceiverAction",
        "resource":"1118836544758700"
    },
    "request_uri":"https://app.sbsbelgium.sumoapp.be/webhook/1118836544758700",
    "method":"POST"
} []

# Response from our application
[2019-04-16 10:54:15] app.INFO: Returned 200 OK JSON Response [
    "[object] (
        Symfony\\Component\\HttpFoundation\\JsonResponse: HTTP/1.0 200 OK
        Cache-Control: no-cache, private
        Content-Type: application/json
        Date: Tue, 16 Apr 2019 08:54:15 GMT
        X-Hook-Secret: c8dcc634473c79e39bf0de47c52cb395

        "ok"
    )"
] []

Hmm. Everything looks fine there. On this request, can you confirm that you still got the did not respond error?

I’m trying to re-create the issue on my end but it’s working as expected (Although, I’m using a Node setup instead).

If you are getting that error, I can try and dig into some logs to see if theres an issue on Asana’s side, but I believe thats unlikely since it’s working in other environments. I’m not sure why your app would be treated differently.

Could you also confirm that your logs on the Response are happening after it returns the Response object? Not sure which version of Symfony you’re on, but I found similar issues online caused by using echo rather than returning (or sending) the response.

I can confirm I’m still getting the did not respond error.

I can also confirm the Response does actually get sent, I’m returning the Response as per Symfony’s standards:

return new JsonResponse(
	null,
	Response::HTTP_OK,
	[
		'X-Hook-Secret' => $secret,
	]
);

You can do the actual request using following cURL command:

curl -iX POST https://app.sbsbelgium.sumoapp.be/webhook/1118835437779954 -H 'Content-Type: application/json' -H 'X-Hook-Secret: 5a2ee55dea34b27b627bc97a5ae943a5'

This might help with trying to find what’s wrong.

Hey @Verkoyen_Tijs ,

I do see your app responds correctly to the handshake Post request.

If your app is single threaded, it may be waiting until your Post call to /webhooks has resolved before handling incoming requests. This might be causing the issue, as this call will not be resolved until after the handshake is completed.

Could you use Postman (or Curl) to hit our /webhooks endpoint, and in the payload, have it target your app? Using something like {resource: 1118598799868276, target: https://app.sbsbelgium.sumoapp.be/webhook/1118598799868276} (use whichever resource you’d prefer).
This way, if your Symfony app is a single thread, it will not be blocked, and it will complete the handshake.

I hope this solves it!

I’ve recreated the request and after about 11 seconds it returns the timeout error.

curl -X POST https://app.asana.com/api/1.0/webhooks \
-H 'Authorization: Bearer [MY_BEARER_TOKEN]' \
-d 'resource=1118598799868276' \  
-d 'target=https://app.sbsbelgium.sumoapp.be/webhook/1118598799868276'

I just remembered that a few weeks ago we ran a test that went wrong causing the app to add new tasks and accompanying webhooks every minute. Could it be that we’ve somehow been blacklisted because of this?

I’ll check. Could I get your oauth app id?

Or if you’re using a PAT, could i get your user id and the name of the PAT?

Hi @Ross_Grambo,

Looks like I got absolutely the same issue and it began a few weeks ago: Webhooks cannot be created anymore ("Invalid Request" response)

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

@Verkoyen_Tijs please edit your post if you can and remove that PAT - it should not be made public. See also the private message I sent.

cc: @Marie

1 Like

Thank you @Verkoyen_Tijs,

Could you make another attempt to create a webhook? I have more robust logs when it’s within a few days.

And thank you @Phil_Seeman for being on top of it in case it was a token!

I tried to make a new webhook, still getting the same error. Can you check the logs?

Hey @Verkoyen_Tijs

The logs don’t supply anything helpful besides claiming that your app did not respond. I was able to recreate this locally, and I believe I found the issue.

This is being caused by our node request package not handling http2. Normally the app should do a protocol handshake with your server, decide on which version of http to use, and proceed with that version. It appears that our node package, in combination with your PHP server, does not correctly do this version check, and your server defaults to using http2. The node package does not handle http2, so it throws up.

The long term fix is for us to either use a new package that supports http2 (which requires quite a bit of work and testing because it’s used elsewhere), or for our server to supply a more elegant error when getting a http2 request vs a http1.1.

However, to get you up and running, could you restrict your server to respond with an http1.1 response? (perhaps just for all webhook endpoints).

Let me know if this unblocks you.

Best,
Ross

I’ve altered the code to return http1.1 responses. However, I’m still getting the same error.
Could you check on your end?

Dang. This was the issue for another user, but it appears it didn’t change anything for your issue.

Let’s move over to PM.

Looks like the issue is the same and a cause + workaround can be found here: Webhooks cannot be created anymore ("Invalid Request" response)