Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
No one likes working on Rails upgrades, and who can blame them? They can be thankless, time-consuming, and—worst of all—unpredictable. How long will they take? When can we get back to feature work? What’ll break when they’re deployed? It’s tempting to say, “Let’s do it later,” but later is usually never. Instead of living in fear of the big, scary Rails upgrade, your team can set up systems for incremental change that let you:
- continue feature work
- deliver predictable results
- seamlessly jump into the next upgrade (because there’s always a next upgrade)
And I’m gonna show you how.
In the years since I’ve worked on upgrading Rails at GitHub, I’ve refined an approach to Rails upgrades and successfully implemented it at several organizations. In this talk, you’ll learn everything you need to burn down uncertainty and boldly go into your next Rails upgrade.
Links from the talk
📝 From the Test Double blog
- Stop ignoring your Rails (and Ruby) deprecations!
- Get Bootboot to work on Rails 4.x in deployment environments with Docker
- Working strategically through Rails upgrades
🧰 Open source tools
- Bootboot—Bundler plugin for dual booting multiple Rails versions on your app’s main branch
- Strong Parameters (Rails 3.x backport)—an example of backporting new features to older versions of Rails
[00:00] (electronic music)
[00:04] - Hi, everybody.
[00:05] - [Audience] Hi.
[00:07] - I'm Ali, and I'm here today
[00:10] to talk about everyone's favorite subject.
[00:12] Drum roll, please.
[00:14] (Ali imitates drum roll)
[00:19] Rails upgrades.
[00:20] (audience cheers)
[00:23] So there are dozens of us, okay, cool.
[00:30] All right, so I'm surprised.
[00:32] I thought more people were not going to like Rails upgrades.
[00:35] That's fine.
[00:36] But when I look past, through past RailsConfs,
[00:39] I see a lot of talks about Rails upgrades.
[00:41] So there was one in 2017,
[00:44] one in 2018, excuse me, two in 2018,
[00:48] one in 2019.
[00:50] There was one last year, and just
[00:52] yesterday, there was a workshop on upgrading Rails.
[00:55] Great workshop, by the way.
[00:57] And since we all love working on Rails upgrades,
[01:01] why are we spending so much time talking about them?
[01:04] When I think about this question,
[01:05] I think about all the different upgrades I've worked
[01:07] on over the course of my career at Test Double.
[01:10] And one thing stands out.
[01:13] These organizations aren't coming to us
[01:15] to work on these upgrades 'cause they're easy.
[01:18] They're coming to us 'cause they're hard.
[01:21] But what is it about these upgrades that makes them so hard?
[01:25] The first thing that might come to mind
[01:27] are all the technical challenges involved
[01:29] with working on an upgrade.
[01:31] For example, the first thing
[01:33] you might want to do in an upgrade,
[01:34] bundle update rails.
[01:36] Might be a multi-day journey through pain and suffering,
[01:40] trying to get your gem versions to work with Bundler,
[01:43] and you find out, uh, this gem's not maintained anymore,
[01:46] so what do you do about that?
[01:48] Or there's some weird fork and you're just like, really?
[01:50] I just want to run bundle update Rails.
[01:52] A standard command, right?
[01:54] Or sometimes there's changes in the public API.
[01:57] For example,
[01:58] how many people here remember the strong params migration
[02:01] from Rails 3 to Rails 4, right?
[02:03] Pretty big migration.
[02:04] And if you had a big app with a lot of controllers,
[02:06] this is a big change.
[02:07] But no matter what the technical challenges are,
[02:11] all of us here, we can solve these technical problems.
[02:14] And I say that pretty confidently, because if we couldn't,
[02:18] there wouldn't be any upgrades ever.
[02:20] And if there were no upgrades,
[02:22] none of us would be here
[02:24] 'cause who would be building stuff in Rails, right?
[02:28] So maybe we need to reframe this question
[02:31] about why is it so hard for organizations
[02:34] to work on these upgrades.
[02:36] And instead, it's why is it so hard for organizations
[02:39] to invest the time and money to do these upgrades?
[02:43] And to think about that, let's shift gears a little bit
[02:46] and talk about something that organizations do every day,
[02:50] and that's feature work.
[02:51] And let's see how this compares to Rails upgrades.
[02:54] So when your team embarks on building feature X, Y, Z,
[02:58] all the people involved, all the stakeholders,
[03:00] whether it's your product managers, UI/UX, engineering, QA,
[03:05] whoever needs to be involved in making that feature happen,
[03:08] all of y'all have some context,
[03:11] some kind of shared understanding
[03:12] of this is the app that we're working in,
[03:15] or this is our business domain, et cetera, right?
[03:18] All of y'all know something about the app,
[03:20] so you can talk about it.
[03:22] And because you have this shared understanding,
[03:24] it's easier to break up the work.
[03:26] And this might mean identifying the scope,
[03:29] but what's our appetite here?
[03:30] How much of this do we want to do right now?
[03:32] Or maybe it's breaking it up into small discrete steps
[03:35] that you can work on bit by bit by bit,
[03:38] shipping stuff incrementally.
[03:40] And because we can break up the work,
[03:43] it's easier, then, to share progress.
[03:45] You might say this feature,
[03:46] we broke it up into 10 cards and a couple cards in,
[03:49] we kind of know, are we on target for our deadline
[03:53] or do we need to have another conversation and say, hey,
[03:55] like, we found some new information out,
[03:58] we either need to adjust scope or our deadline here.
[04:00] Something like that, right?
[04:01] And all this is easier because everyone involved
[04:04] has some kind of shared understanding,
[04:06] and you can break that work up into smaller pieces.
[04:10] So how does this compare then to Rails upgrades?
[04:13] Well, for one, Rails has a lot of code in it, right?
[04:19] I've gotten deep in the weeds
[04:21] of ActiveRecord a couple times,
[04:23] and I'll say I know maybe 5% of the code in Rails.
[04:27] And if you saw Eileen's keynote yesterday,
[04:29] she said there's no core committer that knows all of Rails.
[04:33] So it's a really big code base
[04:35] that no one really understands, right?
[04:39] And Rails is crucial, crucial infrastructure
[04:42] in our application, because without it,
[04:45] we don't really have an app to talk about, right?
[04:47] So now we're talking about changing something
[04:51] that we don't know really well,
[04:54] and we want to change it, potentially, a lot.
[04:58] Who is going to sign up for this project
[05:01] that has so many unknowns?
[05:03] What could go wrong?
[05:04] How long is this going to take?
[05:05] When will it be done?
[05:08] Nah, no one signing up to work
[05:10] on this risky Rails upgrade project.
[05:13] But what if we could change
[05:15] the way we think about Rails upgrades
[05:17] and think of them more like feature work
[05:20] where we take a big black box of a project
[05:25] and we break it up into smaller pieces
[05:28] that we can work through incrementally,
[05:30] shipping stuff bit by bit
[05:32] and using the information that we learn as we ship stuff
[05:35] to help inform the next step we're going to do.
[05:38] If we could do that,
[05:40] we'd have upgrades that are more predictable,
[05:43] and if they were more predictable,
[05:46] they'd be more repeatable, because spoiler alert,
[05:49] there's going to be another Rails upgrade, right?
[05:52] So how do we get here?
[05:55] How do we get to having Rails upgrades
[05:57] that are more predictable and more repeatable?
[06:02] Well, first things first,
[06:04] we have to figure out what version
[06:05] of Rails are we even upgrading to,
[06:07] which might seem like a pretty obvious question.
[06:10] We just want to get to whatever
[06:11] the latest version of Rails is, right?
[06:14] Well, let's imagine we have an app running
[06:16] on Rails 5.2,
[06:17] and right now, Rails 7 is the latest version.
[06:20] So here's our big black box.
[06:22] We want to upgrade to Rails 7.
[06:25] Is there anything we can do to break this up,
[06:28] break this up into smaller pieces?
[06:30] Well, since we're coming from Rails 5,
[06:32] we can say Rails 7 is made up of two major versions,
[06:35] Rails 6 and Rails 7.
[06:38] And if we want to upgrade to Rails 7,
[06:41] we have to upgrade to Rails 6 no matter what.
[06:43] So we could try to do it in one big giant PR
[06:46] that changes everything,
[06:47] or we could just focus on upgrading to Rails 6.
[06:51] And now, our black box is looking a little smaller.
[06:56] Is there anything else we could do
[06:57] to break this down even further?
[06:59] Well, Rails 6 is made up of two minor versions,
[07:02] Rails 6.0 and Rails 6.1.
[07:05] And just like talking about going from 6 to 7,
[07:06] you have to do 6 no matter what,
[07:08] if you want to go to 6.1,
[07:10] you're going to have to upgrade to 6.0.
[07:11] So instead of trying to do that all at once,
[07:15] let's just upgrade to 6.0.
[07:17] And now, our big black box is looking a lot smaller.
[07:21] So what we're trying to do here is,
[07:24] when we're trying to figure out what version
[07:25] we want to upgrade to,
[07:26] is first target that next minor version.
[07:29] So if you're on Rails 5.0,
[07:31] next minor version is 5.1,
[07:33] that's what you're going to upgrade to.
[07:35] And once you get to 5.1,
[07:37] your next minor version is 5.2,
[07:38] so that's your next upgrade.
[07:40] And once you've exhausted all the minor versions,
[07:43] you're going to upgrade then to the next major version.
[07:46] So if you're now on Rails 5.2, there is no Rails 5.3,
[07:51] you're ready to go on to that next major version,
[07:53] Rails 6.0.
[07:55] And in doing this,
[07:57] we're making our upgrades smaller.
[07:59] Instead of trying to take years and years
[08:00] and years of changes to Rails
[08:01] and apply it to our application all at once,
[08:04] we're just going to focus on a narrow chunk
[08:06] that's right in front of us,
[08:09] and this will help drive down the risks of our upgrade,
[08:11] because again, we're not going to try to make a bunch
[08:13] of changes in one big go.
[08:17] So now we're starting to get a little more clarity, right,
[08:20] of what our target version we want to upgrade to.
[08:24] So now we're ready to run bundle update rails, right?
[08:28] Well, before we get ahead of ourselves,
[08:32] we need to talk about everyone's second favorite subject.
[08:36] We're already talking about Rails upgrades,
[08:37] which everyone loves,
[08:39] but we have to talk about everyone's second favorite subject
[08:42] and that's deprecation warnings.
[08:44] I know, I've got all the hits today, right?
[08:46] Okay, so I'm going to need some audience participation here.
[08:49] Can I see a raise of hands
[08:51] if you've ignored deprecations in your app recently?
[08:55] Yeah, right?
[08:56] Me too.
[08:56] That's what I do every day,
[08:58] ignoring deprecations because they're just warnings.
[09:01] They're not, our app's still working
[09:03] and nothing for us to do about 'em, right?
[09:05] But when you're starting on a Rails upgrade,
[09:07] it's really important that you look at them
[09:08] to see what they're saying.
[09:10] For example, imagine now that we're on an app
[09:13] that's running in Rails 6.0,
[09:14] and we see this deprecation warning
[09:15] flying through our logs.
[09:18] It's saying, accessing hashes from config_for
[09:22] by non-symbol keys will be removed in Rails 6.1.
[09:27] We're on Rails 6.0,
[09:29] so this thing that we're doing,
[09:31] accessing the hashes with non-symbol keys,
[09:33] still works fine, right?
[09:35] But when we go to Rails 6.1, the next minor version,
[09:39] it's not going to work anymore.
[09:41] And the last thing this deprecation warning is telling us
[09:43] is use symbols for access instead.
[09:46] It's not saying upgrade to Rails 6.1
[09:48] and open that giant PR that's changing
[09:50] a ton of things, right?
[09:52] It's saying use symbols for access instead,
[09:54] and this is something you can do in your application today.
[09:58] And we can do that for all
[09:59] the deprecation warnings, right?
[10:00] We don't have to wait for a huge big effort
[10:03] to get our app one step closer
[10:06] to working on that newer version of Rails.
[10:09] And the good news is,
[10:11] we don't have to fix all these deprecations at once.
[10:14] For example, imagine you have a deprecation warning
[10:17] that's popping up across hundreds
[10:18] of files in your application.
[10:19] It's a huge mess.
[10:21] You could try to open it in one giant PR
[10:24] and end up changing a ton of files,
[10:25] but no one on your team is going to want to review them.
[10:28] And eventually, you find somebody who's like, look,
[10:30] don't worry, it's just deprecations,
[10:31] just gimme a check mark, please.
[10:33] So, okay, you get your check mark.
[10:34] But when it's time to hit that merge button
[10:36] and deploy this thing to production,
[10:38] you're going to be pretty nervous
[10:39] because you're changing a lot of stuff in your app.
[10:42] Instead, you could fix that deprecation warning in one file.
[10:48] If you change it in one file,
[10:49] it's going to be easier to talk about,
[10:50] easier to get your team to review it,
[10:52] because okay, it's a smaller change,
[10:53] we can see what's going on here, right?
[10:56] It's going to be easier to test,
[10:57] and it's going to be easier to verify that,
[10:59] yeah, we made this change,
[11:00] and it's working on production the way we expected it to.
[11:03] And with that information, the stuff that you're learning,
[11:06] it's going to be easier then for you to come back
[11:08] and confidently make that change
[11:10] to all the other files in your application.
[11:14] So now you take your deprecation warnings
[11:15] and you divvy them up across the people on your team.
[11:17] Say, Hey, everyone,
[11:18] pick one up and fix it here, fix it there.
[11:21] And you're working really hard, working diligently,
[11:24] getting your app, again, one step closer
[11:26] to running on a new version of Rails.
[11:28] And lo and behold, someone comes behind you
[11:31] and brings back a deprecation that you just fixed.
[11:33] And this is a super, super frustrating experience
[11:36] 'cause you're working so hard,
[11:38] it's like, okay, I'll fix the deprecations.
[11:39] This is like a good thing to do.
[11:41] And why would a developer do this?
[11:42] Why are they not paying attention
[11:44] to all the work that we're doing?
[11:45] But we just said a couple minutes ago,
[11:48] everyone's ignoring the deprecation warnings.
[11:50] So can we really blame a developer
[11:52] for doing something that we all do every day?
[11:56] No.
[11:57] Instead, we can make it easier for them to adopt
[12:00] the patterns that we're going to need to use going forward.
[12:03] So here's a blog post I wrote up about one strategy I like
[12:06] to use when dealing with deprecations,
[12:09] and if you Google,
[12:10] you'll find a bunch of different ways to do it,
[12:12] but they all kind of have the same core idea.
[12:16] Once you've fixed a deprecation,
[12:18] you'll set it up in your app so it errors in dev and test.
[12:21] That way that developer, like me,
[12:24] who's used to writing code a certain way,
[12:26] when they're writing tests
[12:27] or they're trying to boot up the app or whatever,
[12:29] and they write deprecated code,
[12:31] the computer's going to yell at them
[12:32] and say, nope, you can't do this anymore.
[12:34] And because nobody likes the computer yelling at them,
[12:37] eventually, they're going to figure out, okay,
[12:39] I need to adopt this new pattern going forward.
[12:42] And we'll couple that with logging in production
[12:45] because tests aren't perfect,
[12:47] we're never going to catch 100% of everything,
[12:49] so we'll log these on production
[12:51] to give us one last fail-safe
[12:52] to make sure there's no deprecations slipping
[12:55] through the cracks.
[12:58] And in doing this, we're shipping code.
[13:00] We're shipping code bit by bit by bit.
[13:03] Again, that's getting us one step closer
[13:05] to that new version of Rails.
[13:07] Wouldn't it be nice if we could stay in this mode
[13:10] for all of our Rails upgrade?
[13:12] Well, let's see what we can do.
[13:14] So now that you fixed all your deprecation warnings,
[13:17] you're ready to run bundle update rails
[13:19] and you check out a new branch.
[13:22] Nothing new there, right?
[13:23] We always check out a new branch
[13:23] when we do something new in our application,
[13:26] but for as long as this branch is living,
[13:30] you're driving up the risks of your upgrade.
[13:33] So let's imagine you're on your branch down here
[13:35] and you spend a couple days doing the Bundler battle
[13:38] to get the new version of Rails to install,
[13:41] and eventually, you get it to work.
[13:45] And in all that time you keep
[13:46] on working away on this branch,
[13:48] development's been cruising along on main.
[13:51] And not a big deal, it's only been maybe a day or two,
[13:53] so you pull those changes in from main to your branch.
[13:57] And development keeps happening on main,
[13:59] and there was some security patch
[14:01] for another gem that got released in that time,
[14:03] so they went ahead and made that change
[14:04] to the Gemfile, pushed it up.
[14:06] So now, when you come back another couple days later,
[14:09] after you've been trying to fix tests or get the app to boot,
[14:12] you rebase and you run into some conflicts
[14:14] because now the Gemfile's changed in two places,
[14:16] it's like, ugh, this is kind of frustrating,
[14:18] but maybe it's not too bad.
[14:20] You figure it out in an hour or something like that.
[14:23] And then your boss comes through,
[14:24] and says, whoa, whoa, whoa, whoa, whoa, whoa, whoa.
[14:27] We have a really important urgent thing we need
[14:29] to work on right now, so please,
[14:31] get to a stopping point on this upgrade,
[14:33] and we'll get back to it when we have time.
[14:34] You say, okay, I'll push up my changes to my branch,
[14:36] and we'll come back to it, no big deal.
[14:39] And development cruises along on main.
[14:43] Time passes.
[14:44] Could be weeks, months, years.
[14:48] I've seen it.
[14:50] And you come back and now your branch is so far behind main.
[14:54] You try to pull the changes in,
[14:56] but now, the Gemfile's changed again.
[14:57] It's like, ugh, this is frustrating,
[14:59] I have to do this whole Bundler thing again,
[15:01] or those tests that you were working so hard to fix,
[15:03] now they've changed on main,
[15:05] so the changes that you made on your branch
[15:06] don't make sense anymore,
[15:08] and you're scratching your head like, what do I do?
[15:09] How do I get this to work?
[15:10] And in the end, you're like, you know what?
[15:13] I'm just going to start this upgrade over from scratch.
[15:16] And that is a huge risk 'cause what we're saying there is,
[15:20] if at any point we have to stop our upgrade
[15:23] because life changes, things happen,
[15:26] we could lose all of the work
[15:27] that we've done up to that point,
[15:30] and that's a huge bummer,
[15:31] and no one wants to be in that situation.
[15:33] So what can we do to get out of this?
[15:36] Well, we can turn to a strategy called dual booting.
[15:39] And what you'll do with dual booting
[15:41] is you'll set up your application on the main branch
[15:44] so it can run on either the production version of Rails
[15:46] or the target version of Rails you're upgrading to,
[15:49] and you'll have some kind of switch,
[15:50] and by default, that switch will usually be off.
[15:52] And when it's off, your app's running
[15:55] on the production version of Rails.
[15:57] Nothing out of the ordinary there, right?
[15:59] It's just the same thing that's always been happening.
[16:01] But when the switch is turned on,
[16:04] we'll be running the target version of Rails
[16:06] on our app on the main branch.
[16:10] There are a lot of different strategies
[16:12] out there for dual booting.
[16:13] I like using Bootboot.
[16:15] It's a Bundler plugin made by Shopify,
[16:17] and I like using it so much that I'll bend over backwards
[16:21] to get it working on Rails 4.
[16:23] Story for another day.
[16:26] So now, when you go back to your branch
[16:28] and you run bundle update rails,
[16:31] you're not just going to be doing the Bundler dance
[16:33] to get the new version of gem,
[16:34] the new version of Rails installed,
[16:36] you'll also be adding that dual booting functionality,
[16:39] so you can toggle the app back
[16:40] and forth between Rails versions.
[16:42] And as soon as you get those two things working,
[16:45] you can bundle the new version of Rails,
[16:47] and you have dual booting enabled
[16:48] so you can toggle Rails versions.
[16:51] You ship that code.
[16:52] You're not going to wait to say,
[16:53] oh, let me get these tests, I can get these tests to work.
[16:55] Mm, mm.
[16:56] Or, the app's not booting, let me make sure that, no.
[16:59] As soon as you get dual booting working
[17:02] and you can bundle a new version of Rails,
[17:03] you ship that off to production,
[17:05] because from that point forward,
[17:07] any changes that you need to make
[17:09] for your Rails upgrade will be on main,
[17:11] and after today, I never want to hear again
[17:14] somebody maintaining a long-lived branch
[17:16] that they're rebasing every day
[17:17] for months for their upgrade.
[17:19] You don't have to do that anymore.
[17:20] Dual booting is here, please use it.
[17:23] And if you do that, then, you're shipping code.
[17:26] You're shipping code every day to main,
[17:28] and you're driving down the risk of your upgrade.
[17:32] But if you're following along with me,
[17:34] you might be asking yourself some questions
[17:37] of like how is this going to work exactly,
[17:39] because there's got to be new stuff
[17:42] in the new version of Rails
[17:44] that isn't in the old version of Rails,
[17:46] and somehow we're going to deploy that code to main,
[17:48] but how's it going to work without the new version of Rails?
[17:52] I'm just confused saying that.
[17:54] Like, how is this going to work, right?
[17:57] Well, so once you go down this strategy
[17:59] where you have dual booting in place,
[18:01] you're going to see conditionals
[18:02] like this pop up in your application.
[18:03] And here, let's imagine we have an app running on Rails 5.2.
[18:07] We'll say, if our app currently is running on 5.2,
[18:11] here's the code that we want to run on production.
[18:14] Otherwise, here's the code that we'll be using
[18:17] in the future version of Rails.
[18:19] And now, let's just ship the code
[18:21] for both version of Rails to main.
[18:23] And you might see a few dozen
[18:26] of these or whatever, you know.
[18:28] Some of these pop up in your code base, not a big deal.
[18:30] Sometimes there's method signatures,
[18:32] change in Rails versions, that kind of stuff.
[18:35] But if you start to see the same pattern of if-elses
[18:38] repeat over and over and over again,
[18:43] that's going to drive up the risk of your upgrade
[18:46] because that's an indicator that there's some pattern
[18:49] that your team's used to that's not going to work going forward
[18:54] and you're going to work hard adding those if-elses,
[18:57] if-else, if-else, right,
[18:58] every time somebody comes back and writes code
[19:00] that the way they're used to writing it,
[19:03] and you'll be stuck in a game of Whack-a-mole
[19:05] that's going to be really hard to win
[19:07] because all these things are always going to pop back
[19:09] up in your application,
[19:10] and you're never going to know, like,
[19:11] did I catch all these cases?
[19:13] What's failing now, right?
[19:15] Not where we want to be.
[19:17] So what can we do here?
[19:20] Well, we can use a strategy called backporting.
[19:22] And what you'll do with backporting
[19:24] is you'll take whatever that new thing is in Rails,
[19:27] whether it be a new feature or a new way of calling code,
[19:30] whatever it is,
[19:31] and you backport it to the old version of Rails
[19:35] that your app is using
[19:36] so then your team can start writing code using
[19:40] that new feature before we finish our upgrade.
[19:44] And there's an excellent open source example
[19:47] of this with Strong Params.
[19:49] We talked about that briefly at the beginning of this talk.
[19:51] Strong Params was released in Rails 4,
[19:53] and again, it was a big change in the way you have
[19:55] to dealt with your parameters.
[19:58] At the time that Rails released Strong Params in Rails 4,
[20:02] they also released the Strong Parameters gem
[20:05] and this gem targeted Rails 3.
[20:08] So there's your backport.
[20:09] This is Strong Parameters that you could add
[20:12] to your app running on Rails 3.
[20:15] And the way it worked was is,
[20:16] if you have a model in your app, let's say it's a Post,
[20:20] you add the gem to your Gemfile,
[20:23] and then you have this module available to you
[20:26] that you can include to opt in to the new behavior,
[20:29] and you would include it only in the one model
[20:32] and work on migrating that model
[20:34] so it uses params the way we need to use it for Rails 4,
[20:37] the way we need to use 'em going forward.
[20:39] So you work on doing that migration,
[20:41] you test everything out, verify it, ship it to production,
[20:44] and once you ship it to production,
[20:47] no one can use params the old way anymore
[20:50] because this module will error and tell people,
[20:52] nope, you can't do that anymore, right?
[20:55] So as soon as you do that, you do it for the one PR,
[20:57] make sure it's working, everything's good.
[21:00] You then do the same process for the next model
[21:03] and the model after that,
[21:05] shipping small PRs bit by bit,
[21:07] migrating towards how things are going to work going
[21:10] forward in a new version of Rails.
[21:12] And once you've done it for all
[21:14] of your models in your application,
[21:16] you can include it globally
[21:17] for all of the ActiveRecord models in your app.
[21:20] So from that point forward, someone adds a new model,
[21:23] and they have to use params the new way.
[21:26] And again, this is an example of a backport
[21:28] that the Rails core team made for us,
[21:31] and it's really helpful, right?
[21:33] But we can do this ourselves.
[21:36] We don't have to depend on Rails
[21:38] to make all our backports for us.
[21:40] So here's a blog post that I worked on talking
[21:43] about a similar situation I ran into in an upgrade
[21:45] and how we implemented the backports
[21:47] and helped the team adopt a new pattern,
[21:51] because that's really what we're trying
[21:52] to do with these backports.
[21:53] We're trying to coach our teams to adopt a new patterns
[21:57] that we're going to need going forward.
[22:00] And depending on when you bring your backport in,
[22:02] they might be writing code for that new version
[22:05] of Rails for months.
[22:07] So, when you do make that switch on production
[22:09] to running the new version of Rails,
[22:11] they're not going to be surprised that,
[22:12] oh, I can't write code like this anymore, what happened?
[22:14] No, they might have been doing
[22:15] it for a long time already, right?
[22:18] And again, we're shipping code,
[22:20] always shipping code bit by bit by bit,
[22:23] getting us closer to that new version of Rails.
[22:27] So things are going good,
[22:28] we feel pretty confident about our Rails upgrade,
[22:31] but we have one more big black box to talk about,
[22:35] and that's test failures.
[22:37] On all of the upgrades I've worked on,
[22:39] I've seen teams spend the majority
[22:41] of their time fixing failing tests.
[22:44] And what can I say about test suites?
[22:47] They're big, slow, complicated, brittle, hard to understand.
[22:53] I could keep going.
[22:54] (Ali laughing)
[22:56] So what do you do when you're working on a Rails upgrade
[22:59] and you see hundreds or thousands
[23:01] and thousands of test failures
[23:03] in these big complicated test suites?
[23:06] We've been talking today about breaking things
[23:07] up into smaller pieces
[23:09] that we can work through incrementally,
[23:11] so if you're seeing thousands
[23:13] and thousands of test failures,
[23:15] you could break up the work that way and say,
[23:17] everyone, take one of these failures,
[23:19] figure out how to fix it and open your PRs,
[23:22] and we'll ship little tiny PRs bit by bit by bit, right?
[23:26] But personally, I don't want to open thousands
[23:29] and thousands of PRs,
[23:31] or even worse, write thousands and thousands
[23:33] and thousands of Jira tickets, right?
[23:35] No one wants to do that.
[23:36] So there's got to be some middle ground here
[23:38] between having a giant PR that's fixing a bunch
[23:41] of unrelated tests
[23:43] and having little tiny PRs that are narrowly focused
[23:46] where we're doing the same kind of things, potentially,
[23:48] over and over again.
[23:50] What can we do?
[23:52] Well, we could try to group our failures.
[23:55] So let's say we look at our tests,
[23:56] we look at all of our failures,
[23:59] and we notice a bunch of tests failing for,
[24:01] expected 200, got 400.
[24:04] Well, we could pull those off as a group, and say,
[24:07] two people, go look at those tests
[24:09] and see why are they failing?
[24:11] Are they failing for the same reason or not?
[24:13] And after a couple days, they look at it,
[24:14] and they say, oh, it looks like there's a bunch
[24:16] of tests failing for reason A,
[24:18] and then the rest of them are failing for reason B.
[24:21] Well, you can break that down then
[24:22] into two other groups of tests, right?
[24:25] And you can say, person one, go ahead
[24:27] and work on that first group of tests
[24:29] and the other person can then tackle
[24:31] the other group of tests.
[24:33] Or maybe you see a bunch of tests failing
[24:35] for ActiveRecords::StatementInvalid
[24:38] you can't do this anymore.
[24:40] You can pull those apart as a group,
[24:42] and they're probably also failing for the same reason.
[24:45] So give somebody that piece of work to work on,
[24:48] and they can work through a solution.
[24:50] And you'll do this for all the failures in the test
[24:52] where you're trying to identify,
[24:54] trying to identify all these different groups.
[24:57] And you'll start to see they're just finding chunks of work
[25:00] that are just the right size.
[25:03] They're not too big where you have a bunch
[25:06] of unrelated changes going on,
[25:08] and they're not too small where you're just opening
[25:10] the same PR kind of over and over again, right?
[25:13] Kind of a Goldilocks situation.
[25:17] And the grouping you do is going to depend on your application
[25:20] and the failures you're seeing.
[25:22] So for your upgrade,
[25:24] it might make sense to group the failures by file.
[25:27] Totally valid option.
[25:29] Or maybe you run your tests on CI across 40 containers
[25:33] and you have 15 failing.
[25:34] You say, okay, everyone take a container
[25:36] and make that container green.
[25:38] Again, valid option.
[25:39] Depends on your application, the failures you're seeing,
[25:42] and you know, the upgrades you're working on.
[25:45] And what we're trying to do here, again,
[25:46] is find chunks of work that are the right size
[25:49] that keep us in a state where we are building momentum
[25:52] and we're shipping code,
[25:54] always shipping code bit by bit by bit
[25:57] to get us closer to running on a new version of Rails.
[26:02] So now we've fixed all of our deprecations,
[26:04] we've got dual booting working,
[26:06] and we fixed all of our tests.
[26:08] So we're ready to make that switch from production,
[26:10] from the production version of Rails
[26:12] to the new version that we're upgrading to.
[26:16] Are we though?
[26:18] Even after you've done all this hard work,
[26:21] getting your app, again, closer to the new version of Rails,
[26:25] it can be really hard to assess
[26:27] what's the risk left of making this switch?
[26:30] So at this late stage of our upgrade,
[26:33] is there anything we can do to drive
[26:35] down the risks involved in making the switch?
[26:40] Well, we can try small experiments
[26:42] in production environment, excuse me.
[26:44] We can try small experiments in production-like environments
[26:48] to see if the upgrade is behaving the way we expected it.
[26:51] So you can imagine, you have your app running in production,
[26:54] and you decide we're going to make a deployment
[26:56] that has the upgrade that's for internal use only.
[26:59] And we ask all of our employees
[27:00] to please use this upgrade version of the app,
[27:03] and we'll be monitoring our Datadog dashboards
[27:06] or Rollbar errors,
[27:07] whatever observability you have to see,
[27:10] are we seeing new errors?
[27:11] Is the app working the way we expected it to or not?
[27:14] Might call this dog filter, right?
[27:17] Or maybe you want to try something like a canary deploy
[27:20] where you say, we'll ship the upgrade
[27:24] to 5% of our traffic for two minutes.
[27:28] Might have some impact on your users,
[27:30] but it's a really small experiment.
[27:31] We're only going to deploy this out for 5%
[27:33] of our traffic for two minutes.
[27:35] And after that's done, we're going to pull it back.
[27:37] And in those two minutes, again,
[27:38] you have all your dashboards,
[27:40] and you're monitoring everything up to see,
[27:42] are new errors coming through?
[27:43] Is this working the way we expected?
[27:44] All that stuff.
[27:46] And after a couple minutes, you're like, oh yeah,
[27:48] there are a bunch of errors that we did not expect to see.
[27:50] So you go back, fix those errors,
[27:53] and you say, okay, when we come back
[27:54] and do this experiment again,
[27:57] we expect there to be less errors, right?
[27:59] Because if we fix them, the error should go down.
[28:01] So you do that, you see less errors,
[28:04] and you say, okay, let's go ahead
[28:06] and bump this up then to 10% of our traffic for two minutes.
[28:09] And you do the same kind of dance,
[28:11] maybe you find some new errors, go back and fix it,
[28:12] try to experiment again.
[28:14] And after a while, it starts looking really good,
[28:17] so you say, let's bump this up to 10%
[28:18] of our traffic for five minutes,
[28:21] and this time, whoa, wow, it's looking really good.
[28:24] And then finally, you're like, you know,
[28:26] let's bump this up to 40% for 10 minutes,
[28:28] because we're feeling really good about the upgrade.
[28:31] We're not seeing new errors pop through.
[28:33] It seems to be that the app is working really well,
[28:35] and we're starting to feel more confident
[28:38] about this upgrade that we've been working on.
[28:41] And that's really what we've been trying to do,
[28:44] talking through all these different strategies today.
[28:48] We're trying to drive down the risks of our upgrades,
[28:52] gaining new information to help us feel more confident
[28:55] that we're not going to cause an incident
[28:57] by shipping out a Rails upgrade.
[28:59] And if we can do that,
[29:01] we'll all be shipping zero downtime Rails upgrades.
[29:06] All right, so the first upgrade that you do like this,
[29:11] it's going to be the hardest.
[29:13] It doesn't really matter what version
[29:15] of Rails you're on or anything like that.
[29:17] It's going to be the hardest because your team,
[29:19] your organization, probably has never done
[29:21] an upgrade like this before.
[29:23] So you have to figure out, what tooling do we like?
[29:25] How do we want to divvy up the test failures.
[29:27] All that stuff, you have to sort through that, right?
[29:30] And in that process, you're going to be learning a ton.
[29:34] You're going to be learning, oh,
[29:35] we really like using this tool for deprecations,
[29:37] or we were really happy with the way
[29:39] the deploy went out in doing it this way.
[29:41] Whatever it is, you figure that out,
[29:43] document it, reflect on it,
[29:45] and that's going to help you build a playbook, right,
[29:47] a process where this is how we're going to do
[29:50] the upgrade next time.
[29:52] And if you do that,
[29:53] when it's time to do the second upgrade,
[29:55] might be a little bumpy,
[29:56] but it's going to be a little bit easier.
[29:59] And by the time you get to the third upgrade,
[30:02] you're going to be cruising, and like, okay,
[30:03] we've done this before,
[30:04] we know the in and outs of this thing,
[30:06] no problem, we can do it.
[30:09] And maybe one day, you'll get to a place
[30:12] like where GitHub is,
[30:13] where they're upgrading Rails weekly, right?
[30:17] I don't know if anyone wants to live
[30:18] on the cutting edge like that,
[30:19] always running the latest version of Rails on edge, right?
[30:24] But it goes to show,
[30:25] once you refine your process and figure it out,
[30:28] these are the tools that we like,
[30:29] this is the way we want to do it,
[30:30] you can get to this point where it's every week,
[30:32] not a big ceremony to do another Rails upgrade.
[30:36] And as you go through this journey,
[30:38] learning these new things
[30:39] and applying different things to your upgrades,
[30:43] share what you've learned.
[30:44] As I showed you at the beginning of this talk,
[30:46] every year there seems to be another talk
[30:48] about Rails upgrades.
[30:49] So there's always going to be something new to share,
[30:52] so please come back, share what you learn,
[30:55] and you can help everyone else
[30:57] also ship zero downtime Rails upgrades.
[30:59] Or if you don't like public speaking,
[31:01] you can write a blog post.
[31:02] I have a bunch of those too.
[31:05] So that's all I have today.
[31:08] If you have any questions about Rails upgrades
[31:10] or anything, really, feel free to email me,
[31:14] aliwritesback@gmail.com.
[31:16] And I'm also on the Slack for however long
[31:18] that's going to be around, at my full name.
[31:20] And again, I mentioned a few blog posts in this talk.
[31:23] You can find them all on the Test Double blog
[31:25] and other blog posts about upgrades, too,
[31:27] and a bunch of other good stuff.
[31:30] And there's a lot of us here from Test Double.
[31:33] I really enjoyed Daniel's talk yesterday talking
[31:35] about legacy code and being more curious
[31:37] and how can we make it better.
[31:39] I'm looking forward to seeing Landon speak, soon,
[31:43] in a few minutes about machine learning and Ruby.
[31:45] I think that's going to be really cool.
[31:47] And Megan and Justin going to be having
[31:50] a little town hall tomorrow,
[31:51] making standard happen, even better for all of us, right?
[31:55] So that's all I have.
[31:57] Thank you.
[31:58] (audience applauding)