How to speed up a slow app

One of our awesome clients in the trucking industry came to us with a problem: Our app feels too slow.

An app that responds slowly is a common complaint — but the solution isn’t always that clear or simple. It’s likely there are multiple things causing the app to feel slow.

With a month, we'd investigated what was causing the slow app, rolled out changes and launched an update that more than doubled the speed of the driving directions API.

Below, we’ll break down how we were able to make one of many specific performance improvements to this app, having started with no idea where any of the problems were.

Spoiler alert: Computers are faster than ever, but if you’re trying to draw a line on a map using a bunch of circles – you should change that!

Step 1: Find all the slow parts and pick one

As engineers, we know: You can never fix everything. Trying to fix everything often just means you’ll never finish. It’s best to pick something small to work on, especially if it’s something no one has asked you to do (and usually, with performance work, no one asks – because they don’t know what to ask for!).

Our app, like most, has an overwhelming number of different features. Each feature might or might not have its own individual reasons for being slow. We needed some data to help us better define the problem.

Thankfully, this app already had an APM (Application Performance Monitoring) tool enabled. Our APM here was Appsignal, but there are a bunch of others around (we’ve also seen NewRelic, Sentry, Scout, and others). But as common as these tools are, they’re often not looked at enough. So we took a look and found some data that roughly looked like this:

You can see in the above that it’s still not super obvious where we should spend our time. Maybe it’s better to try to make “Find all the widget stores” faster, even though it’s already the fastest feature; it’s also used a lot.

Planning a little ahead was helpful here. We looked for ways to make “Find all the widget stores” faster, but couldn’t find any! Thankfully, we also had some tricks up our sleeves for the other endpoints, and we made the call that the “Search for stores along a route” was the best thing to work on.

Step 2: Make the one thing fast… but wait, how do we do that?

There’s a problem with the idea of making things fast. You can’t. Reaching out across the internet and somehow causing the phone in someone’s hand to be faster is impossible. Instead, you have to find a way to do less.

To do less, we need to learn as much as we can about what it’s already doing. For that, we reach for our APM tools’ tracing capabilities.

A trace might look like this:

See the problem?

In our actual trace for the route search, we didn’t have just 3 repetitions of a moderately expensive query; there were almost 200.

Here’s what it looked like on a map. Each of the below circles was taking up valuable time.

On top of that, circles have another problem: They overlap when there’s several of them in a row — every overlap is a spot we’re searching somewhere we’ve already searched.

That seems like a great way to “do less” – why search a spot twice if we don’t have to?

And on top of that: Circles don’t fit nicely together. Those slivers on the outside leave the possibility of gaps in data (or, in this case, would mean locations get missed in the map results).

Circles are absolutely awful. Let’s change that and see what happens.

Step 3: Now that we know how it works – try something different

After spending time doing even more research, we found out that the big kids have solved this problem already.

Google has lots of maps, and they don’t use circles; they made something called S2. Uber is another big company that uses a lot of maps. They made a competing thing called H3.

Either of these probably could have helped us. But H3 uses hexagons, and hexagons are the bestagons, so we went with that. (Side note: engineers love to argue about which tools are the best, but usually, there’s more than one right answer!).

Using H3, we were able to draw a route like this:

Doesn’t that just look way cleaner?

The real benefit for us here is using H3’s grid as a communication tool between different parts of the app. Instead of querying the database 200 or so times for stores near a point, we can now query the database just once for stores inside any of these hexagons.

If you dig down deep enough, there are some wild mathy things happening here. Someone had to figure out how to calculate the latitude and longitude of a particular face on a truncated icosahedron. But the good news is that it didn’t have to be us! H3 has done all of that already.

Within a month of first looking at the route search, we launched an update to the app – and initial results show it more than doubled the speed of the map search. Success!