How to create powerful APIs with GraphQL and Elixir

The following account was found hidden in the end credits of the new "Mission Impossible 7", detailing Eric Newbury's journey iterating on a GraphQL API for the Mission Impossible super-spy organization "IMF."

As the requirements became more and more complex, Eric found himself a captive of the IMF who monopolize all of his development time. In an attempt to escape, Eric began to develop the [Hex package quarry to dig his way to freedom.

Please consider contributing code and beta testing to the `quarry` package, and Eric may have a chance to escape.

Mission 001: Build a flexible API for "Espionage"

Your mission, should you choose to accept it—build a flexible API for the internal "Espionage" App so that Agents can use applications that efficiently fetch only the data they need and nothing more. As always, if you should fail, your existence will be denied by the IMF and all ties to your consultancy will be severed. This issue will self-destruct in 5 seconds.

My eyes squint as I’m blinded by the sudden white background of the GitHub 404 page. So unnecessary. I must say, this seems like the perfect use case for GraphQL, since it promises to allow clients to fetch exactly what they need in one GraphQL request. But the implementation is another matter. I open the documentation for Absinthe, the Elixir implementation of the GraphQL spec. In Absinthe, the entire schema is defined up front, and every field needs to have a resolver function that describes how to get the value for that field. Fields without an explicit resolver will just use a Map.get(parent, field_name). I install the Absinthe dependencies, configure the router, then start defining my schema file.

# lib/espionage_api/schema.ex
defmodule EspionageApi.Schema do
  use Absinthe.Schema

  query do
    field :missions, list_of(:mission) do
      resolve(&Resolvers.Mission.missions/2)
    end
  end 

  object :mission do
    field :id, :id
    field :title, :string
    field :priority, non_null(:integer)
    field :director, non_null(:director), resolve: &Resolvers.Mission.director/3
    field :agents, list_of(:agent), resolve: &Resolvers.Mission.agents/3
  end

  object :director do
    field :id, :id
    field :name, :string
    field :base, non_null(:base), resolve: &Resolvers.Director.base/3
    field(:missions, list_of(:mission), resolve: &Resolvers.Director.missions/3)
  end

  object :agent do
    field :id, :id
    field :name, :string
    field :base, non_null(:base), resolve: &Resolvers.Agent.base/3
    field :mission, :mission, resolve: &Resolvers.Agent.mission/3
  end

  object :base do
    field :id, :id
    field :name, :string
  end
end

Fields like title on a mission object don’t need a resolver function since it can just use the default Map.get to fetch the title off of the %Mission{} struct, but for nested data structures like agents, I set up a custom resolver function that executes a database query to lookup Agents by mission.id. Full source code is available on github

‍

I start up the app and execute a test query:

{
  missions {
    agents {
    	name
      base { name }
    }
    title
    priority
  }
}

It works, but what's this?! The debug logs for the database are giving some disownment-worthy results ...

[debug] QUERY OK source="missions" db=0.7ms idle=660.8ms
[debug] QUERY OK source="agents" db=0.2ms idle=661.6ms
[debug] QUERY OK source="bases" db=0.2ms idle=662.0ms
[debug] QUERY OK source="bases" db=0.4ms idle=662.4ms
[debug] QUERY OK source="agents" db=0.4ms idle=662.9ms
[debug] QUERY OK source="bases" db=0.1ms idle=663.4ms
[debug] QUERY OK source="bases" db=0.5ms idle=663.5ms

The problem: N+1 queries

Why all these queries? GraphQL promised to be the solution to all the round trips REST APIs make to the database as they fetch all related entities. The clock ticks ominously over my shoulder; the ticks seem to bounce louder and louder across the dusty bare floor and against the rusted metal door of the office they have me working in.

Hmmm, this has the classic signs of an N+1 query. My test app has 2 missions, each with 2 agents assigned, and each of those has their own home base of operations. I can see two database queries for agents, and 4 for bases, corresponding to the total number of individual entities. This won't do. I glance at the door nervously before returning to the comfort of the
Absinthe documentation.

Solution: Dataloader

Aha, this is a common issue, and the solution is now included in Absinthe in the form of a `dataloader` integration.

‍
I briskly convert my resolvers to use dataloader.

defmodule EspionageApi.Schema do
  use Absinthe.Schema
  import Absinthe.Resolution.Helpers, only: [dataloader: 1]

  ...

  object :director do
    field :id, :id
    field :name, :string
    field :base, non_null(:base), resolve: dataloader(Espionage)
    field :missions, list_of(:mission), resolve: dataloader(Espionage)
  end
  
  ...

  @impl true
  def context(ctx) do
    loader = Dataloader.new() |> Dataloader.add_source(Espionage, Espionage.data())
    Map.put(ctx, :loader, loader)
  end

  @impl true
  def plugins do
    [Absinthe.Middleware.Dataloader | Absinthe.Plugin.defaults()]
  end
end

defmodule Espionage do
  def data() do
    Dataloader.Ecto.new(Espionage.Repo)
  end
end

Here I’ve included the dataloader helper from Absinthe and use that instead of a resolver function on all my object’s nested entity fields. At the bottom of the schema file, I add the Dataloader Absinthe plugin using the plugins callback, and I add the loader to the Absinthe context using the callback. I also define the data source of the loader in Espionage as an Ecto source.

I run my query again and check the logs:

SELECT * FROM "missions"
SELECT * FROM "agents" AS a WHERE (a."mission_id" = ANY($1)) ORDER BY a."mission_id"
SELECT * FROM "bases" AS b WHERE (b."id" = ANY($1))

There we go! Now I'm only getting one query per entity type. Dataloader is able to collect all the ids of the agents
and bases that will be needed and fetches them in a single query per entity type.  Looks good, ship it!

GitHub fix/001/dataloader

Before I can celebrate, a dark object smashes through the window landing on the floor at my feet. I dive for cover,
waiting for the explosion ... *Why did I sign this contract?!* Silence. It's a cellphone, with an unread message. I collect myself and open the message.

‍

Mission 002: Filter missions by agent

Your next mission, should you choose to accept it—Our users need a way to view missions, filtered by mission title, and also by user name. If you should fail in your mission ....

... Framed, imprisoned, disowned, I know. I've been hired to modernize their internal systems, but clearly it's
their business practices that needs attention. *Oh well ...*

Based on the requirements, I should be able to make a GraphQL query with a filter like this:

{
  missions(filter: {agent: {name: "Ethan Hunt"}}) {
    title
    agents { name }
  }

This should be easy enough using field args, input objects,
and some additions to my main resolver function.

defmodule EspionageApi.Schema do

  ...

  query do
    field :missions, list_of(:mission) do
      arg(:filter, :mission_filter)
      resolve(&Resolvers.Mission.missions/2)
    end
  end

  ...

  input_object :mission_filter do
    field :title, :string
    field :agents, :agent_filter
  end

  input_object :agent_filter do
    field :name, :string
  end

defmodule EspionageApi.Resolvers.Mission do
  def missions(%{filter: %{agents: agent} = mission}, _) do
    mission_criteria = mission |> Map.delete(:agents) |> Map.to_list()
    {:ok, Espionage.Missions.all_by_agent(mission_criteria, Map.to_list(agent))}
  end

  def missions(%{filter: mission}, _) do
    {:ok, Espionage.Missions.all(Map.to_list(mission))}
  end

  def missions(_, _) do
    {:ok, Espionage.Missions.all()}
  end
end

defmodule Espionage.Missions do
  alias Espionage.Repo
  alias Espionage.Mission

  import Ecto.Query

  def all(criteria \\ []) do
    Repo.all(from(p in Mission, where: ^criteria))
  end

  def all_by_agent(mission_criteria, agent_criteria) do
    query = from m in Mission, join: a in assoc(m, :agents)

    query =
      Enum.reduce(mission_criteria, query, fn {key, value}, q ->
        where(q, [m, _], field(m, ^key) == ^value)
      end)

    query =
      Enum.reduce(agent_criteria, query, fn {key, value}, q ->
        where(q, [_, a], field(a, ^key) == ^value)
      end)

    Repo.all(query)
  end
end

Phew, a lot of boilerplate, but using pattern matching, I'm able to check for those filter arguments and use a modified query to fetch the results.

Running this GraphQL query now only returns one mission since it's the only one Ethan Hunt is assigned to. Ship it!

GitHub PR feature/002/filter-by-mission-agents

A low rumbling rattles the loose pens scattered across my desk, and one of them slips off the edge onto the concrete floor. A hairline crack darts across the concrete under the fallen pen, as the rumbling grows to an ear-shattering roar.

I throw myself against the wall in alarm as the floor opens up in a pool of crumbled rebar and concrete, and a figure clad in a hard hat and grubby coveralls comes clambering out of the cavernous opening left in the floor. Without pause, he tosses a packet on my desk, then fires a grappling hook through skylight of the lofty concrete room and launches through the opening, momentarily blocking out the only source of light, before vanishing.

Mission 003: Filter by more nested entities, and non-equality operations

Your mission, should you choose to accept it--allow filtering of missions by their associated director attributes as well as agent home base attributes and other nested entities as needed. In addition, allow filtering by mission where the priority is greater than or equal to a certain value. Sorting by nested entity fields should also be supported.

‍

This so could have been an email! I glance nervously down the gaping hole in the floor, into the darkness that throbs with silence. And clearly this organization does not follow a "leave no trace" policy.

‍
If requests like this keep coming, my codebase is going to explode with conditionals building queries for every possible nested entity. This is a waste of my life. Why is there no package like `dataloader` to allow me to dynamically join in filter entities?

A GraphQL query meeting the requirements of the mission could look like this:

{
  missions(
    filter: {
    	priority__gte: 10, 
    	agents: {base: {name: "Washington"}}
    },
    sort: [PRIORITY, DIRECTOR__NAME]
  ) {
    priority
    director {
      name
    }
    agents {
      name
      base {
        name
      }
    }
  }
}

‍
...

‍

It's been a month, and I haven't left this room. I've been keeping the IMF distracted from my lack of progress by placing false leads on the dark web about the activities of the "Syndicate," the IMF's greatest opponent.

Meanwhile, I've undertaken the development of packages quarry and an integration for Absinthe absinthe_quarry to escape from this hellhole of boilerplate code and the musty odor of living in a mining operation. It's time to complete this mission.

defmodule EspionageApi.Schema do
  use Absinthe.Schema
  import Absinthe.Resolution.Helpers, only: [dataloader: 1]
  import AbsintheQuarry.Helpers, only: [quarry: 2]
  alias Espionage.{Mission, Repo}

  query do
    field :missions, list_of(:mission) do
      arg :filter, :mission_filter
      resolve quarry(Mission, Repo)
    end
  end
  ...

  input_object :mission_filter do
    field :title, :string
    field :agents, :agent_filter
    field :director, :director_filter
    field :priority__gte, :integer
  end

  input_object :agent_filter do
    field :name, :string
    field :base, :base_filter
  end

  input_object :director_filter do
    field :name, :string
    field :base, :base_filter
  end

  input_object :base_filter do
    field :name, :string
  end

  ...

end

I've added some additional input objects, and more can easily be added as needed. I've also completely deleted the `EspionageApi.Resolvers.Mission` and removed all the boilerplate functions from `Espionage.Missions` modules since all of that is no longer needed. Everything is handled by the quarry helper.

I can also add sort and limit arguments to the field as needed.

  query do
    field :missions, list_of(:mission) do
      arg :filter, :mission_filter
      arg :sort, list_of(:mission_sort)
      arg :limit, :integer
      resolve quarry(Mission, Repo)
    end
  end

  enum :mission_sort do
    value :priority
    value :director__name
  end

It works, I can now run the GraphQL query with dynamic sorting, filtering and limits!

GitHub feature/003/dynamic-filtering

Self-assigned Mission 004: Survive

The IMF has discovered my attempts to distract them. I'm now under constant guard, pushing out features as quickly as I can.

A drop of water splashes dangerously close to my grimy keyboard. I have no more room for delays! The latest request arrived a
few minutes ago, dropped among the thunderous roar of a swarm of attack drones. *Definitely not an environment suited for high productivity.*

The goal is to support GraphQL queries like this where sub selections can have their own filters:

{
  missions {
    title
    agents(filter: {name: "Ethan Hunt"}) {
      name
    }
  }
}

I could use the `quarry` resolver on the nested field instead of `dataloader`, but then I would be back to an N+1 query ... What I really need is a subquery on the main top-level query. Perhaps I can extend my package so `quarry` will introspect
the selected fields and load them as part of the main query. For `belongs_to` relationships, I can just use an inner join,
perhaps sharing the same join as any filter that already requires joining on that relationship. And for `has_many`
I can use a subquery to preserve the functionality of dataloader, but do it all ahead of time embedded with the main query!  Something like this ...

  object :mission do
    field :id, :id
    field :title, :string
    field :description, :string
    field :priority, non_null(:integer)
    field :director, non_null(:director), meta: [quarry: true]

    field :agents, list_of(:agent), meta: [quarry: true] do
      arg :filter, :agent_filter
    end
  end

All instances of `resolve: dataloader(...` are now replaced with a meta tag indicating to a parent that uses quarry
that we want to preload this association. In the case of a `has_many` association like `agents` I can add `quarry` options
like filter, and that will limit that sub-list of items. So the following GraphQL query will show missions, and select
only Ethan Hunt as an agent. Looks like I can also completely remove `dataloader` and its setup from my codebase
since `quarry` provides overlapping functionality.

GitHub feature/004/replace-dataloader

I jump as a new mission packet lands unceremoniously on my keyboard. There's no end in sight. As the lone developer, I have no hope of keeping up with their increasingly specific demands ... if I only had more contributors, maybe they would let me go. No one's looking ... I have no access to the outside world, but maybe ... maybe if I bury my story deep in this filesystem ... someone will find it. Please send help!