Improving dev experience with Overmind

Web apps are born with a healthy amount of naivety.

There aren’t many expectations of what the apps should do so they have the freedom to do … well, anything. This is great for the early stages of development, but as these apps grow and take on more responsibilities a flexible, little app can quickly become hard to manage.

Luckily, there are lots of solutions for these pain points. Instead of putting all the burden on the app, we can turn to well established tools for help. We can move work to the background with tools like Sidekiq or move search responsibilities to search engines like Elasticsearch.

While these tools will lead to happier and healthier apps, they can also make app development harder. Where a developer might’ve only needed to kick off a web server before, now they might need a whole suite of external services running before they can get anything to work.

Luckily there’s already a solution for that too! Foreman was invented for this very problem.

Foreman does a great job of managing all these services for us and Procfiles have become a defacto standard on hosting services like Heroku and AWS. Unfortunately, Foreman doesn’t always work as we’d hope.

Using Foreman

Let’s take a look at Foreman running in an existing app. We’ll use the dev.to repo as an example. At press time, it has a Procfile.dev that can be used for local development (check out the source on GitHub). The Procfile.dev list four processes:

web: bin/rails s -p 3000
webpacker: ./bin/webpack-dev-server
job: bin/rake jobs:work
sidekiq: bundle exec sidekiq

We can start these processes locally using Foreman by running bundle exec foreman start -f Procfile.dev.

› bundle exec foreman start -f Procfile.dev
13:36:40 web.1       | started with pid 5575
13:36:40 webpacker.1 | started with pid 5576
13:36:40 job.1       | started with pid 5577
13:36:40 sidekiq.1   | started with pid 5578

Foreman starts all the processes listed in Procfile.dev so we don’t have to 🎉.

Now that everything’s running, we might want to do some debugging. Since this is a Rails app, we can use Byebug to stop a server request and poke around the running program. We’ll add a byebug statement to the StoriesController index method.

class StoriesController < ApplicationController
  # …

  def index
    byebug # Adding Byebug statement to stop the server request
    return handle_user_or_organization_or_podcast_or_page_index if params[:username]
    return handle_tag_index if params[:tag]

    handle_base_index
  end

  # …
end

Now, let’s see what happens when Foreman hits that byebug statement.

13:57:06 web.1       | Processing by StoriesController#index as */*
13:57:06 web.1       |   Parameters: {"i"=>"i"}
13:57:06 web.1       | Processing by AsyncInfoController#shell_version as JSON
13:57:06 web.1       | GET /async_info/shell_version completed with 200 OK in 130.945ms
13:57:06 web.1       |
13:57:06 web.1       | [2, 11] in /Users/alimi/repos/dev.to/app/controllers/stories_controller.rb
13:57:06 web.1       |     2:   before_action :authenticate_user!, except: %i[index search show]
13:57:06 web.1       |     3:   before_action :set_cache_control_headers, only: %i[index search show]
13:57:06 web.1       |     4:
13:57:06 web.1       |     5:   def index
13:57:06 web.1       |     6:     byebug
13:57:06 web.1       | =>  7:     return handle_user_or_organization_or_podcast_or_page_index if params[:username]
13:57:06 web.1       |     8:     return handle_tag_index if params[:tag]
13:57:06 web.1       |     9:
13:57:06 web.1       |    10:     handle_base_index
13:57:06 web.1       |    11:   end
13:57:06 web.1       | (byebug) Processing by ShellController#bottom as */*
13:57:06 web.1       | GET /shell_bottom completed with 200 OK in 47.766000000000005ms
13:57:06 web.1       | Processing by ShellController#top as */*
13:57:06 web.1       | GET /shell_top completed with 200 OK in 77.289ms

Byebug stopped the program and is waiting for us to do something, but there isn’t any way for us to do anything. Foreman is focused on making sure our processes are running, but it doesn’t give us a way to interact with tools like Byebug. Furthermore, the app is now unresponsive since it’s waiting to move on from that byebug statement.

Foreman does a great job of managing all the services that need to run for any given app, but it doesn’t quite hit the sweet spot for development workflow. This is where Overmind shines.

Replacing Foreman with Overmind

Overmind was created by the  good folks err… evil Martians at Evil Martians. It uses tmux under the covers to manage the different processes but don’t worry—you don’t have to know anything about tmux to use Overmind. However, Windows users are out of luck as Overmind only supports Linux, BSD, and macOS.

Since Overmind also uses Procfiles, it’s a drop-in replacement for Foreman. After following the install instructions, we can use Overmind to start the dev.to processes by running overmind start -f Procfile.dev.

› overmind start -f Procfile.dev
system    | Tmux socket name: overmind-dev-to-xzhZFvpZqnhn0hYuK2o4CH
system    | Tmux session ID: dev-to
system    | Listening at /Users/alimi/repos/dev.to/.overmind.sock
web       | Started with pid 7511...
webpacker | Started with pid 7512...
job       | Started with pid 7513...
sidekiq   | Started with pid 7514...

Overmind’s output looks really similar to Foreman’s, but we can see tmux is the first thing that gets started.

Now let’s see what happens when we re-add that byebug statement to the StoriesController index method.

web       | Processing by StoriesController#index as */*
web       |   Parameters: {"i"=>"i"}
web       | Processing by AsyncInfoController#shell_version as JSON
web       | GET /async_info/shell_version completed with 200 OK in 76.882ms
web       |
web       | [2, 11] in /Users/alimi/repos/dev.to/app/controllers/stories_controller.rb
web       |     2:   before_action :authenticate_user!, except: %i[index search show]
web       |     3:   before_action :set_cache_control_headers, only: %i[index search show]
web       |     4:
web       |     5:   def index
web       |     6:     byebug
web       | =>  7:     return handle_user_or_organization_or_podcast_or_page_index if params[:username]
web       |     8:     return handle_tag_index if params[:tag]
web       |     9:
web       |    10:     handle_base_index
web       |    11:   end
web       | (byebug) Processing by ShellController#bottom as */*
web       | Processing by AsyncInfoController#shell_version as JSON
web       | GET /async_info/shell_version completed with 200 OK in 46.370000000000005ms

Sooo … we’re still stuck 😅. Byebug stopped the program for us, but there’s no way for us to do anything. Or is there?!

Overmind has a connect command that lets us connect to any of the processes it started. The byebug statement is wating for us in the web process, so we can run overmind connect web in a new terminal window.

> overmind connect web

[2, 11] in /Users/alimi/repos/dev.to/app/controllers/stories_controller.rb
    2:   before_action :authenticate_user!, except: %i[index search show]
    3:   before_action :set_cache_control_headers, only: %i[index search show]
    4:
    5:   def index
    6:     byebug
=>  7:     return handle_user_or_organization_or_podcast_or_page_index if params[:username]
    8:     return handle_tag_index if params[:tag]
    9:
   10:     handle_base_index
   11:   end
(byebug)

Once we’re in the web process, we can interact with the Byebug session and poke around to our hearts content.

[2, 11] in /Users/alimi/repos/dev.to/app/controllers/stories_controller.rb
    2:   before_action :authenticate_user!, except: %i[index search show]
    3:   before_action :set_cache_control_headers, only: %i[index search show]
    4:
    5:   def index
    6:     byebug
=>  7:     return handle_user_or_organization_or_podcast_or_page_index if params[:username]
    8:     return handle_tag_index if params[:tag]
    9:
   10:     handle_base_index
   11:   end
(byebug) puts "hello internet!"
hello internet!
nil
(byebug)

Overmind for the win

We’ve seen one way that Overmind gives us a better development experience, but there are a bunch more improvements to explore. Head over to the Evil Martians blog and the GitHub repo to learn more about Overmind. Although we used it in a Rails app here, it isn’t framework/language specific and can work in any setup.

We’ve had a great experience here at Test Double switching clients over to Overmind in the past year and hopefully you enjoy using it too 💚.

H/t to @TheABrad for pointing me to Overmind