Let’s face it: Learning is hard.
When I get a new job, I have to learn all about the new product, how it runs, and what I can do to help contribute. For myself, this is not an issue: I am an experienced software engineer. Unravelling balls of yarn that make complex software stacks function is what I do for a living.
In this post, I’m going to go over some of the ways that software projects handle implicit complexity and how most of the tools that I have used in this space have not really solved any problems. If you aren’t interested in the history lesson, you can skip to the end to read about the specific tools I use to reduce installation steps to cd project-name && devenv up.
The many faces of complexity
What happens when marketing folks need to write a blog post? What about operations? Do you make them learn a new trade just to be able to make a blog post? Not everyone has the time or desire to spend all of their time learning how to run a multi-layer web application on their laptop.
What about when your engineering team is rapidly hiring? How much time should you pay the new developer to get a working stack on their machine? A week? Two days? The answer is very simply “as little as possible.”
In a lot of README.md files, I have seen something along the lines of “please install ruby, nodejs, and hugo.” There is a lot of implicit complexity to that simple statement. Which version do you need to install? There are major revisions that are supported that are largely incompatible with each other. Someone using NodeJS 16 might find a dependency that fails in NodeJS 18. If you aren’t explicit about these kinds of things then you are expecting the target audience to know exactly what you mean and they should know exactly how to find that specific version and install it. This is a pain point.
Most software projects use some or all of these kinds of tools to manage these kinds of details:
- language-specific version tools
- package managers
- Docker and Docker Compose (
podman, too)
With each of these tools, there are a dizzying amount of choices that can be made, and as a developer of software, you have to manage all of those choices for contributors. At this point, I’m going to explain how these tools handle specific parts of the complexity triangles of success. Also with each tool comes another thing that must be managed by each user and provides yet another way that onboarding causes friction.
How to install a programming language
Most popular languages have some sort of version tool.
I know that there are things like rbenv and nvm in this specific example. If you use one of those tools, then you have another dependency on your contributors. This route is pretty popular, and as the saying goes, nobody would be fired for using one of these tools. One downside of these tools is that they usually only work on a single language, so if you used both ruby and nodejs, you would have to use two different tools to get both the language runtimes. There are other tools like asdf, but with Yet Another Tool plus a plugin for each language runtime, your cognitive complexity is still very high.
A lot of applications rely on persistent state, which means some other process on your computer to provide and store information entered by the users. A popular example of this would be PostgreSQL database. The language-specific tools previously mentioned can’t help you install a database.
To approach the combination of requirements, you can use Homebrew if you are using a Mac, but if you are using Linux (or WSL), you can’t expect someone with a perfectly good package manager to install Yet Another Tool to install a database. Things like Homebrew do make it easy to get some version of databases, but they often times don’t have a lot of flexibility in the specific version you need.
The state of bespoke environments
So to quickly recap: language-specific version tools can help you get the versions you need, but they are snowflakes and Yet Another Tool. General package managers can do something similar for non-language runtimes, but with the variety of operating systems used today and the need to have specific versions of tools available, these tools will sometimes have limited effectiveness.
Docker doesn’t solve my problems
At this point, you’re probably screaming Docker! Just use Docker! Before you jump on that bandwagon, you need to know that there are some non-trivial downsides. If your company is over a certain size or revenue amount, the Docker FAQ states that you have to purchase a license to use Docker. Additionally, if you are using a Mac, then you are also running a virtual machine to host the Docker daemon. Docker is inherently a Linux technology, so anywhere you see Docker, it’s running Linux. If you are working on a large project, the time required to keep your local files in sync with the VM can also be non-trivial.
Anecdotal evidence from another Test Double consultant demonstrated a significant overhead when usingdocker+ sync. A small-ish Rails application took ~20 seconds to boot inside ofdocker, but only ~7 seconds to boot in a virtual machine hosted on the same physical host. This example is an almost 3x slowdown, which probably does not scale to larger projects, but you can fully expect there to be a significant penalty for usingdocker+ sync.
A whole new world
Let’s introduce a different ecosystem. I have been learning a lot about Nix recently. With Nix, you can guarantee that each machine uses the exact same binary. In other words, it’s reproducible given a similar architecture and operating system. The package repository is called nixpkgs, which is just a git repository of recipes to build thousands of software utilities. According to Repology, nixpkgs has more packages than any other package repository in existence. This count includes Arch AUR, Debian (and derivatives), FreeBSD Ports, and Fedora.
Nix can let you specify the exact version of the language runtime you want and any other dependent libraries. Even if the specific version isn’t available, nixpkgs contains the steps to build specific version you need (as long as they are similar to another version). There are downsides, though. It’s hard to use for beginners, but in my experience, this isn’t too much of an issue for making developer environments.
I use a tool called devenv which is made by the same people that created cachix (the Nix binary cache). devenv can automate a lot of popular language runtimes (sorry, COBOL is not included) and popular databases out-of-the-box. You can write shell scripts for specific steps, set environment variables, install programs and libraries, and run all necessary processes to have a complete stack on your machine. Their repository has a load of examples that you can use as a starting point for your own application. devenv will help you run processes without the overhead of Docker and without any implicit knowledge about how to install the myriad of tools required to make software function. Just don’t run this in production, containers provide a very real security benefit that you will not get in this default configuration.
There is one more tool that I use to automate local stacks: direnv. I use it to avoid having to explicitly run devenv shell, and instead all I need to do is cd project-name and then I have all of the programs and libraries installed.
I have used these tools to provide a great deal of benefit to people who just need to do their job and have the software get out of their way. Many engineers just don’t care how postgres is running, just as long as the backend server can connect to it and not return an error. I provide the happy, golden path that is well-supported. You can venture into the unknown at your own risk.
What about that devenv.nix file?
One part I am kind of hand-waving away is this devenv.nix file. It’s written in the Nix language and you have to put your specific dependencies into it. It can’t guess what you need, and it doesn’t know what steps you need to run to get your stack up and running. However, this is where Developer Experience can come to the rescue.
With a Developer Experience approach, a single person can write this file that’s then used by every other engineer at an organization. Once written, it usually does not require ongoing maintenance so the value proposition can be quite large for investing this time.
Stack automation is just one of many different facets to the engineering discipline of Developer Experience, and it’s something Test Double has helped clients with to accelerate engineering teams. It’s entirely focused on the workflow of your organization as a whole, finding friction points, and then grinding them down into smooth shapes.
The Happy Path
This is how I go about running a stack on my Mac (even for personal / side projects). For Linux, the steps are largely similar, but you should examine each tool’s installation instructions for more information.
# enable nix flakes
mkdir -p "$HOME/.config/nix"
echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf
# install nix
sh <(curl -L https://nixos.org/nix/install)
# install cachix with flakes
nix profile install nixpkgs#cachix
cachix use devenv
# install devenv
nix profile install --accept-flake-config github:cachix/devenv/latest
# install direnv
nix profile install nixpkgs#direnv
echo 'eval "$(direnv hook zsh)"' >> "$HOME/.zshrc"
# then restart your shellAnd that’s it. Once you have those tools installed and you have your devenv.nix file developed, you can run these steps to get a fully-functional stack:
cd project-name
# only run this the first time you cd into the directory
direnv allow
devenv upThe great part about this is that it works for nearly every kind of project out there. It’ll run a Django application as well as a Ruby on Rails app, or a Java application as easily as a Rust project. In all cases, the steps are same once you have devenv.nix written for your specific needs.
