Overcoming flaky tests in Ruby with environment variables

Lots of apps use environment variables to manage environment-specific config.

For example, an app might use environment variables to set 3rd party API keys. Or an app might use environment variables to toggle features on and off. Since environment variables are so commonly used, we’ll eventually want to test code that depends on these environment variables.

And if we’re not careful, these tests can make our Ruby test suites flaky—tests that should be passing will randomly fail for unexpected reasons.

Testing code that depends on environment variables

For example, imagine we have a StringMultiplier class. New instances are initialized with a string, and there’s a #multiply method that multiplies the string.

class StringMultiplier
  def initialize(string)
    @string = string
  end

  def multiply(multiplier)
    @string * multiplier
  end
end

string_multiplier = StringMultiplier.new("yo")
string_multiplier.multiply(2) # returns "yoyo"

Now we need to change #multiply so it yells when the YELLING environment variable is set.

class StringMultiplier
  def initialize(string)
    @string = string
  end

  def multiply(multiplier)
    multiplied_string = @string * multiplier

    if ENV["YELLING"]
      multiplied_string.upcase
    else
      multiplied_string
    end
  end
end

To test this behavior, we can set the YELLING environment variable in a test. I’ll do this in minitest, but it’ll be mostly the same in RSpec.

require "minitest/autorun"

class StringMultiplierTest < MiniTest::Test
  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end
end

I’ll put that in a file called string_multiplier_test.rb and give it a run.

$ ruby string_multiplier_test.rb
Run options: --seed 29206

# Running:

.

Finished in 0.001057s, 946.0735 runs/s, 946.0735 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

We set the environment variable in the test, the Ruby process sees that it’s set, and the test passes! 🎉

Let’s flesh that test out a little bit and add test coverage for when YELLING is off.

require "minitest/autorun"

class StringMultiplierTest < MiniTest::Test
  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

  describe "when yelling is disabled" do
    it "multiplies the string without yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "yoyoyo", multiplied_string
    end
  end
end

Now, we run the test again and …

$ ruby string_multiplier_test.rb
Run options: --seed 32704

# Running:

.F

Failure:
when yelling is disabled#test_0001_multiplies the string without yelling [string_multiplier_test.rb:36]:
Expected: "yoyoyo"
  Actual: "YOYOYO"


rails test string_multiplier_test.rb:33



Finished in 0.001291s, 1549.1863 runs/s, 1549.1863 assertions/s.
2 runs, 2 assertions, 1 failures, 0 errors, 0 skips

What?! Why is it failing? Let’s try running it again.

$ ruby string_multiplier_test.rb
Run options: --seed 47754

# Running:

..

Finished in 0.001042s, 1919.3858 runs/s, 1919.3858 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

Huh, it’s passing now. Must’ve been some flaky thing we don’t need to worry about. It should pass when we run it again.

$ ruby string_multiplier_test.rb
Run options: --seed 12373

# Running:

.F

Failure:
when yelling is disabled#test_0001_multiplies the string without yelling [string_multiplier_test.rb:36]:
Expected: "yoyoyo"
  Actual: "YOYOYO"


rails test string_multiplier_test.rb:33



Finished in 0.001281s, 1561.2806 runs/s, 1561.2806 assertions/s.
2 runs, 2 assertions, 1 failures, 0 errors, 0 skips

WTF!!! 🤬

Environment variable changes are sticky

What’s going on here? minitest is running the tests in random order.

• In some runs, the test that sets YELLING runs last and all the tests pass. ✅
• In other runs, the test that sets YELLING runs first and the other test fails. 🛑

When the test that sets YELLING runs

  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

it sets YELLING in the Ruby process for as long as that process is running. So when it gets run first, it turns on YELLING for all future tests—whether they want it set or not.

How can we test this code without breaking other tests?

Rewrite the code under test

When our code makes it hard for us to write tests, that’s a sign that there might be something off with our code. In our current example, we’ve tightly coupled our code and our tests to Ruby’s environment variable management. Can we rewrite the code (and our tests) so this isn’t the case?

We can update StringMultiplier#initialize so it optionally accepts yelling as an argument

class StringMultiplier
  def initialize(string, yelling = nil)
    @string = string
    @yelling = yelling
  end
  # ...
end

and change #multiply so it uses that argument instead of the YELLING environment variable.

class StringMultiplier
  # ...
  def multiply(multiplier)
    multiplied_string = @string * multiplier

    if @yelling
      multiplied_string.upcase
    else
      multiplied_string
    end
  end
end

We can still toggle yelling in StringMultiplier#multiply with the environment variable in our application code. But instead of having the environment variable embedded in our class, we can pass it in on initialization.

StringMultiplier.new("yo", ENV["YELLING"])

As a result, we no longer have to use the environment variable in our tests.

  describe "when yelling is enabled" do
    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo", true)
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

And now all our tests will pass happily ever after. 🥰

Be careful when setting environment variables in your tests

When we set environment variables in our tests, they can lead to hard-to-understand test failures. Here we only have two tests, so we don’t have too much to parse through to get to the root of the problem. This quickly changes once our test suite grows to have hundreds or even thousands of tests.

A big integration test that we’re not thinking about could suddenly become flaky because of an environment variable that got set in a little unit test. And when that happens in a parallelized CI environment, good luck trying to find the one test that’s making the others flaky. If this pattern continues, confidence in the test suite will slowly erode over time and soon we’ll be questioning why we even write tests to begin with.

Luckily, with our new knowledge, we can eliminate one source of flaky tests in our Ruby test suites.

We might not always have the chance to rewrite the code under test to be less dependent on Ruby’s environment variable management. When this happens, we can use after or around blocks to reset environment variables after changing them.

before do
  @cached_yelling = ENV["YELLING"]
  ENV["YELLING"] = "1"
end

after do
  ENV["YELLING"] = @cached_yelling
end

There are also gems like Climate Control that’ll reset environment variables for us.

We could also mock out ENV but be careful mocking what you don’t own.