Skip to main content
Test Double company logo
Services
Services Overview
Holistic software investment consulting
Software Delivery
Accelerate quality software development
Product Management
Launch modern product orgs
Legacy Modernization
Renovate legacy software systems
DevOps
Scale infrastructure smoothly
Upgrade Rails
Update Rails versions seamlessly
Technical Recruitment
Build tech & product teams
Technical Assessments
Uncover root causes & improvements
Case Studies
Solutions
Accelerate Quality Software
Software Delivery, DevOps, & Product Delivery
Maximize Software Investments
Product Performance, Product Scaling, & Technical Assessments
Future-Proof Innovative Software
Legacy Modernization, Product Transformation, Upgrade Rails, Technical Recruitment
About
About
What's a test double?
Approach
Meeting you where you are
Founder's Story
The origin of our mission
Culture
Culture & Careers
Double Agents decoded
Great Causes
Great code for great causes
EDI
Equity, diversity & inclusion
Insights
All Insights
Hot takes and tips for all things software
Leadership
Bold opinions and insights for tech leaders
Developer
Essential coding tutorials and tools
Product Manager
Practical advice for real-world challenges
Say Hello
Test Double logo
Menu
Services
BackGrid of dots icon
Services Overview
Holistic software investment consulting
Software Delivery
Accelerate quality software development
Product Management
Launch modern product orgs
Legacy Modernization
Renovate legacy software systems
Cycle icon
DevOps
Scale infrastructure smoothly
Upgrade Rails
Update Rails versions seamlessly
Technical Recruitment
Build tech & product teams
Technical Assessments
Uncover root causes & improvements
Case Studies
Solutions
Solutions
Accelerate Quality Software
Software Delivery, DevOps, & Product Delivery
Maximize Software Investments
Product Performance, Product Scaling, & Technical Assessments
Future-Proof Innovative Software
Legacy Modernization, Product Transformation, Upgrade Rails, Technical Recruitment
About
About
About
What's a test double?
Approach
Meeting you where you are
Founder's Story
The origin of our mission
Culture
Culture
Culture & Careers
Double Agents decoded
Great Causes
Great code for great causes
EDI
Equity, diversity & inclusion
Insights
Insights
All Insights
Hot takes and tips for all things software
Leadership
Bold opinions and insights for tech leaders
Developer
Essential coding tutorials and tools
Product Manager
Practical advice for real-world challenges
Say hello
Developers
Developers
Developers
Testing

Overcoming flaky tests in Ruby with environment variables

Struggling with flaky tests in Ruby? Learn the best practices for using environment variables to stabilize your test suite and boost code reliability.
Ali Ibrahim
|
April 11, 2022
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Lots of apps use environment variables to manage environment-specific config.

For example, an app might use environment variables to set 3rd party API keys. Or an app might use environment variables to toggle features on and off. Since environment variables are so commonly used, we’ll eventually want to test code that depends on these environment variables.

And if we’re not careful, these tests can make our Ruby test suites flaky—tests that should be passing will randomly fail for unexpected reasons.

Testing code that depends on environment variables

For example, imagine we have a StringMultiplier class. New instances are initialized with a string, and there’s a #multiply method that multiplies the string.

class StringMultiplier
  def initialize(string)
    @string = string
  end

  def multiply(multiplier)
    @string * multiplier
  end
end

string_multiplier = StringMultiplier.new("yo")
string_multiplier.multiply(2) # returns "yoyo"

Now we need to change #multiply so it yells when the YELLING environment variable is set.

class StringMultiplier
  def initialize(string)
    @string = string
  end

  def multiply(multiplier)
    multiplied_string = @string * multiplier

    if ENV["YELLING"]
      multiplied_string.upcase
    else
      multiplied_string
    end
  end
end

To test this behavior, we can set the YELLING environment variable in a test. I’ll do this in minitest, but it’ll be mostly the same in RSpec.

require "minitest/autorun"

class StringMultiplierTest < MiniTest::Test
  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end
end

I’ll put that in a file called string_multiplier_test.rb and give it a run.

$ ruby string_multiplier_test.rb
Run options: --seed 29206

# Running:

.

Finished in 0.001057s, 946.0735 runs/s, 946.0735 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

We set the environment variable in the test, the Ruby process sees that it’s set, and the test passes! 🎉

Let’s flesh that test out a little bit and add test coverage for when YELLING is off.

require "minitest/autorun"

class StringMultiplierTest < MiniTest::Test
  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

  describe "when yelling is disabled" do
    it "multiplies the string without yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "yoyoyo", multiplied_string
    end
  end
end

Now, we run the test again and …

$ ruby string_multiplier_test.rb
Run options: --seed 32704

# Running:

.F

Failure:
when yelling is disabled#test_0001_multiplies the string without yelling [string_multiplier_test.rb:36]:
Expected: "yoyoyo"
  Actual: "YOYOYO"


rails test string_multiplier_test.rb:33



Finished in 0.001291s, 1549.1863 runs/s, 1549.1863 assertions/s.
2 runs, 2 assertions, 1 failures, 0 errors, 0 skips

What?! Why is it failing? Let’s try running it again.

$ ruby string_multiplier_test.rb
Run options: --seed 47754

# Running:

..

Finished in 0.001042s, 1919.3858 runs/s, 1919.3858 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

Huh, it’s passing now. Must’ve been some flaky thing we don’t need to worry about. It should pass when we run it again.

$ ruby string_multiplier_test.rb
Run options: --seed 12373

# Running:

.F

Failure:
when yelling is disabled#test_0001_multiplies the string without yelling [string_multiplier_test.rb:36]:
Expected: "yoyoyo"
  Actual: "YOYOYO"


rails test string_multiplier_test.rb:33



Finished in 0.001281s, 1561.2806 runs/s, 1561.2806 assertions/s.
2 runs, 2 assertions, 1 failures, 0 errors, 0 skips

WTF!!! 🤬

Environment variable changes are sticky

What’s going on here? minitest is running the tests in random order.

  • In some runs, the test that sets YELLING runs last and all the tests pass. ✅
  • In other runs, the test that sets YELLING runs first and the other test fails. 🛑

When the test that sets YELLING runs

  describe "when yelling is enabled" do
    before do
      ENV["YELLING"] = "1"
    end

    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo")
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

it sets YELLING in the Ruby process for as long as that process is running. So when it gets run first, it turns on YELLING for all future tests—whether they want it set or not.

To learn more about how Ruby manages environment variables, see Starr Horne’s post on the Honeybadger blog.

How can we test this code without breaking other tests?

Rewrite the code under test

When our code makes it hard for us to write tests, that’s a sign that there might be something off with our code. In our current example, we’ve tightly coupled our code and our tests to Ruby’s environment variable management. Can we rewrite the code (and our tests) so this isn’t the case?

We can update StringMultiplier#initialize so it optionally accepts yelling as an argument

class StringMultiplier
  def initialize(string, yelling = nil)
    @string = string
    @yelling = yelling
  end
  # ...
end

and change #multiply so it uses that argument instead of the YELLING environment variable.

class StringMultiplier
  # ...
  def multiply(multiplier)
    multiplied_string = @string * multiplier

    if @yelling
      multiplied_string.upcase
    else
      multiplied_string
    end
  end
end

We can still toggle yelling in StringMultiplier#multiply with the environment variable in our application code. But instead of having the environment variable embedded in our class, we can pass it in on initialization.

StringMultiplier.new("yo", ENV["YELLING"])

As a result, we no longer have to use the environment variable in our tests.

  describe "when yelling is enabled" do
    it "multiplies the string with yelling" do
      string_multiplier = StringMultiplier.new("yo", true)
      multiplied_string = string_multiplier.multiply(3)
      assert_equal "YOYOYO", multiplied_string
    end
  end

And now all our tests will pass happily ever after. 🥰

Be careful when setting environment variables in your tests

When we set environment variables in our tests, they can lead to hard-to-understand test failures. Here we only have two tests, so we don’t have too much to parse through to get to the root of the problem. This quickly changes once our test suite grows to have hundreds or even thousands of tests.

A big integration test that we’re not thinking about could suddenly become flaky because of an environment variable that got set in a little unit test. And when that happens in a parallelized CI environment, good luck trying to find the one test that’s making the others flaky. If this pattern continues, confidence in the test suite will slowly erode over time and soon we’ll be questioning why we even write tests to begin with.

Luckily, with our new knowledge, we can eliminate one source of flaky tests in our Ruby test suites.

We might not always have the chance to rewrite the code under test to be less dependent on Ruby’s environment variable management. When this happens, we can use after or around blocks to reset environment variables after changing them.

before do
  @cached_yelling = ENV["YELLING"]
  ENV["YELLING"] = "1"
end

after do
  ENV["YELLING"] = @cached_yelling
end

There are also gems like Climate Control that’ll reset environment variables for us.

We could also mock out ENV but be careful mocking what you don’t own.

Related Insights

🔗
How to write clean Ruby RSpec tests
🔗
Why TLDR is the Ruby testing framework you need for faster feedback
🔗
How to stop hating your tests

Explore our insights

See all insights
Developers
Developers
Developers
C# and .NET tools and libraries for the modern developer

C# has a reputation for being used in legacy projects and is not often talked about related to startups or other new business ventures. This article aims to break a few of the myths about .NET and C# and discuss how it has evolved to be a great fit for almost any kind of software.

by
Patrick Coakley
Leadership
Leadership
Leadership
Turning observability into a team strength without a big overhaul

By addressing observability pain points one at a time, we built systems and practices that support rapid troubleshooting and collaboration.

by
Gabriel Côté-Carrier
Developers
Developers
Developers
Why I actually enjoy PR reviews (and you should, too)

PR reviews don't have to be painful. Discover practical, evidence-based approaches that turn code reviews into team-building opportunities while maintaining quality and reducing development friction.

by
Robert Komaromi
Letter art spelling out NEAT

Join the conversation

Technology is a means to an end: answers to very human questions. That’s why we created a community for developers and product managers.

Explore the community
Test Double Executive Leadership Team

Learn about our team

Like what we have to say about building great software and great teams?

Get to know us
Test Double company logo
Improving the way the world builds software.
What we do
Services OverviewSoftware DeliveryProduct ManagementLegacy ModernizationDevOpsUpgrade RailsTechnical RecruitmentTechnical Assessments
Who WE ARE
About UsCulture & CareersGreat CausesEDIOur TeamContact UsNews & AwardsN.E.A.T.
Resources
Case StudiesAll InsightsLeadership InsightsDeveloper InsightsProduct InsightsPairing & Office Hours
NEWSLETTER
Sign up hear about our latest innovations.
Your email has been added!
Oops! Something went wrong while submitting the form.
Standard Ruby badge
614.349.4279hello@testdouble.com
Privacy Policy
© 2020 Test Double. All Rights Reserved.