503 Error for the past week

English Forum Developers & API
1

I have a custom SSIS Asana ETL package that loads our data from Asana to our local DB for reporting. It has been working well but last week I started getting a 503 error in the middle of the process (at different points each time). I cannot reproduce the error in a simple way since it just fails “eventually”.

Any help would be appreciated!

Here is a bit of my log from the last time it ran:

2019-03-18 14:15:02	Load Asana Data:
    Error: System.Net.WebException: The remote server returned an error: (503) Server Unavailable. 
    at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e) 
    at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers) 
    at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)	\LoadAsanaData\Load Asana Data	Call Asana API [2]
2019-03-18 14:15:02	Load Asana Data:The task, 'Load Asana Data', failed.	\LoadAsanaData\Load Asana Data	None
2019-03-18 14:15:02	Load Asana Data:Finished, 2:15:02 PM, Elapsed time: 00:05:01.641.	\LoadAsanaData\Load Asana Data	None
2019-03-18 14:14:51	Load Asana Data:Information: []	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:51	Load Asana Data:Information: Task: Volunteers/Jobs Review (571300241666142)	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: {"id":"571300241666142","gid":"571300241666142","assignee":null,"assignee_status":"upcoming","completed":false,"completed_at":null,"created_at":"2018-02-24T16:50:16.603Z","custom_fields":null,"due_at":null,"due_on":"2018-04-01","collaborators":[{"id":"233695385019689","gid":"233695385019689","name":"Helen Hutchinson","email":"hhutchinson@partners.org"}],"memberships":[{"section":null}],"tags":[],"modified_at":"2018-02-24T16:50:16.603Z","name":"Volunteers/Jobs Review","notes":"Design Cycle: Volunteer\n\nDescription: Volunteers/Jobs Description Review\n\nValidation Tasks: \n*spot check 3-5 across affiliates if possible, I think most are in NSMC\n*these are site secured, ensure site attached appropriately\n*confirm, job, end date, event, completed, start date converted correctly\n\nDue Date: 4/1/2018","parent":null,"projects":[{"id":"571300032935783","gid":"571300032935783","archived":false,"created_at":null,"due_date":null,"current_status":null,"collaborators":null,"members":null,"modified_at":null,"name":"ARCHIVE COPY CRM TR0 Data Validation 2-24-18","notes":null,"owner":null,"public":false,"team":null,"workspace":null}],"subtasks":[],"workspace":{"id":"8466265247842","gid":"8466265247842","name":"partners.org"},"stories":null}	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 1 Collaborators...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 1 Memberships...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 0 Tags...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading stories for: ARCHIVE COPY CRM TR0 Data Validation 2-24-18 (571300032935783)	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:51	Load Asana Data:Information: []	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:51	Load Asana Data:Information: Task: Volunteers/Check Exception Log Counts (571300241671466)	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: {"id":"571300241671466","gid":"571300241671466","assignee":null,"assignee_status":"upcoming","completed":false,"completed_at":null,"created_at":"2018-02-24T16:50:16.762Z","custom_fields":null,"due_at":null,"due_on":"2018-04-01","collaborators":[{"id":"233695385019689","gid":"233695385019689","name":"Helen Hutchinson","email":"hhutchinson@partners.org"}],"memberships":[{"section":null}],"tags":[],"modified_at":"2018-02-24T16:50:16.762Z","name":"Volunteers/Check Exception Log Counts","notes":"Design Cycle: Volunteer\n\nDescription: Volunteers/Check Exception Log Counts\n\nValidation Tasks: *Review exceptions reports\n*Update cleanup tasks or validation tasks as appropriate\n\nDue Date: 4/1/2018","parent":null,"projects":[{"id":"571300032935783","gid":"571300032935783","archived":false,"created_at":null,"due_date":null,"current_status":null,"collaborators":null,"members":null,"modified_at":null,"name":"ARCHIVE COPY CRM TR0 Data Validation 2-24-18","notes":null,"owner":null,"public":false,"team":null,"workspace":null}],"subtasks":[],"workspace":{"id":"8466265247842","gid":"8466265247842","name":"partners.org"},"stories":null}	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 1 Collaborators...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 1 Memberships...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading 0 Tags...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: Loading stories for: ARCHIVE COPY CRM TR0 Data Validation 2-24-18 (571300032935783)	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:51	Load Asana Data:Information: []	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:51	Load Asana Data:Information: AddTasks(ARCHIVE COPY CRM TR0 Data Validation 2-24-18 (571300032935783)) Done (174)	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:51	Load Asana Data:Information: GETTING TASKS for Data Validation TR1S3 (3/22/19 - ) (569478186800189)	\LoadAsanaData\Load Asana Data	Projects
2019-03-18 14:14:50	Load Asana Data:Information: []	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:50	Load Asana Data:Information: Task: Revenue/Tributes/Tribute donation totals Counts (571300241644400)	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: {"id":"571300241644400","gid":"571300241644400","assignee":null,"assignee_status":"upcoming","completed":false,"completed_at":null,"created_at":"2018-02-24T16:50:15.994Z","custom_fields":null,"due_at":null,"due_on":"2018-04-01","collaborators":[{"id":"233695385019689","gid":"233695385019689","name":"Helen Hutchinson","email":"hhutchinson@partners.org"}],"memberships":[{"section":null}],"tags":[],"modified_at":"2018-02-24T16:50:15.994Z","name":"Revenue/Tributes/Tribute donation totals Counts","notes":"Design Cycle: Revenue\n\nDescription: Revenue/Tributes/Tribute donation totals Counts\n\nValidation Tasks: *check RE counts to CRM counts\n*check RE totals to CRM totals\n\nDue Date: 4/1/2018","parent":null,"projects":[{"id":"571300032935783","gid":"571300032935783","archived":false,"created_at":null,"due_date":null,"current_status":null,"collaborators":null,"members":null,"modified_at":null,"name":"ARCHIVE COPY CRM TR0 Data Validation 2-24-18","notes":null,"owner":null,"public":false,"team":null,"workspace":null}],"subtasks":[],"workspace":{"id":"8466265247842","gid":"8466265247842","name":"partners.org"},"stories":null}	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: Loading 1 Collaborators...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: Loading 1 Memberships...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: Loading 0 Tags...	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: Loading stories for: ARCHIVE COPY CRM TR0 Data Validation 2-24-18 (571300032935783)	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:50	Load Asana Data:Information: []	\LoadAsanaData\Load Asana Data	Stories
2019-03-18 14:14:50	Load Asana Data:Information: Task: Revenue/Check Exception Log Counts (571300241655755)	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: {"id":"571300241655755","gid":"571300241655755","assignee":null,"assignee_status":"upcoming","completed":false,"completed_at":null,"created_at":"2018-02-24T16:50:16.263Z","custom_fields":null,"due_at":null,"due_on":"2018-04-01","collaborators":[{"id":"233695385019689","gid":"233695385019689","name":"Helen Hutchinson","email":"hhutchinson@partners.org"}],"memberships":[{"section":null}],"tags":[],"modified_at":"2018-02-24T16:50:16.263Z","name":"Revenue/Check Exception Log Counts","notes":"Design Cycle: Revenue\n\nDescription: Revenue/Check Exception Log Counts\n\nValidation Tasks: *Review exceptions reports\n*Update cleanup tasks or validation tasks as appropriate\n\nDue Date: 4/1/2018","parent":null,"projects":[{"id":"571300032935783","gid":"571300032935783","archived":false,"created_at":null,"due_date":null,"current_status":null,"collaborators":null,"members":null,"modified_at":null,"name":"ARCHIVE COPY CRM TR0 Data Validation 2-24-18","notes":null,"owner":null,"public":false,"team":null,"workspace":null}],"subtasks":[],"workspace":{"id":"8466265247842","gid":"8466265247842","name":"partners.org"},"stories":null}	\LoadAsanaData\Load Asana Data	Tasks
2019-03-18 14:14:50	Load Asana Data:Information: Loading 1 Collaborators...
2

Hi @Wayne_Pozzar, sorry for all the errors. We are in the process of rolling out some new infrastructure and were gradually passing more and more traffic through it. It wasn’t generating errors last week when we were monitoring it, but began returning 503 Service Unavailable errors over the weekend. There were not enough errors to trip our alarms, and we only noticed the issue this morning when we checked our metrics.

We immediately shut off this new piece of infrastructure, and our logs show that the errors have stopped. We’re still in the process of investigating the root cause, and will monitor the API’s behavior more closely when we start rolling this out again. We’re also reevaluating whether our existing thresholds for alarms should be lower.

1 Like
3

I am also seeing 503s but only for specific users, is this an ongoing issue?

4

One of our users is still hitting 503s

Happy to help debug as much as I can, just let me know!

Thanks,

Jono Kolnik