Hey there! Friendly API engineer here I’m jumping in to hopefully give a high level overview of why API writes are slower than reads.
Between 2008 - 2015, all our code (API, server, web framework) were written in a framework known as Luna. This was a JS framework that used server side simulation to achieve reactivity. This framework made a lot of things that at the time were hard, easier, like reactivity / data loading / etc. But, it was also really complex, and, importantly for this conversation, slow. See this blog post for more details on this framework, why we started moving away from it as a company.
Since then, we have been rewritten our read API routes (
GET requests) in Scala with a new framework (it doesn’t have a snazzy name unfortunately - we just call it the Scala API ). We built this framework from the ground up to be easier to write / understand, and also be faster. I’m having trouble finding the exact numbers, but for most (all?) requests this was a massive perf improvement.
But, we haven’t had the opportunity to rewrite our write API routes yet (
POST, etc. requests). This is due to the tightly coupled nature of the Luna framework: it’s hard to decouple the API code from the writes, and the writes from the business logic to accomplish them. This means these routes have stayed the same / slowed down, since they’ve been stuck in this legacy system.
We’re currently investigating performance here in two lenses:
- what can we do to improve the performance of the write API: as outlined above, if we got rid of Luna and replaced it with something similar to the Scala API, we’d hopefully get better performance. We have some designs here, and are investigating feasibility.
- how can we improve performance generally: we’re currently doing a lot of work on getting better instrumentation, observability (e.g. incorporating Lightstep), and figuring out if there’s ways we can improve performance across the API. We’ve already identified a way to improve API performance after deployments that we’re looking to get out soon, which should improve the worst case performance in our API.
These are unfortunately long term solutions, we don’t have any quick changes we can do to fix this. But know that we are investing effort here, and hopefully that will incrementally improve performance.
In the meantime, I’d definitely recommend doing writes in the Batch API. While this doesn’t eliminate the performance problems, it can help with write performance.