Zero downtime Rails upgrades

[00:00] (electronic music)
[00:04] - Hi, everybody.
[00:05] - [Audience] Hi.
[00:07] - I'm Ali, and I'm here today
[00:10] to talk about everyone's favorite subject.
[00:12] Drum roll, please.
[00:14] (Ali imitates drum roll)
[00:19] Rails upgrades.
[00:20] (audience cheers)
[00:23] So there are dozens of us, okay, cool.
[00:30] All right, so I'm surprised.
[00:32] I thought more people were not going to like Rails upgrades.
[00:35] That's fine.
[00:36] But when I look past, through past RailsConfs,
[00:39] I see a lot of talks about Rails upgrades.
[00:41] So there was one in 2017,
[00:44] one in 2018, excuse me, two in 2018,
[00:48] one in 2019.
[00:50] There was one last year, and just
[00:52] yesterday, there was a workshop on upgrading Rails.
[00:55] Great workshop, by the way.
[00:57] And since we all love working on Rails upgrades,
[01:01] why are we spending so much time talking about them?
[01:04] When I think about this question,
[01:05] I think about all the different upgrades I've worked
[01:07] on over the course of my career at Test Double.
[01:10] And one thing stands out.
[01:13] These organizations aren't coming to us
[01:15] to work on these upgrades 'cause they're easy.
[01:18] They're coming to us 'cause they're hard.
[01:21] But what is it about these upgrades that makes them so hard?
[01:25] The first thing that might come to mind
[01:27] are all the technical challenges involved
[01:29] with working on an upgrade.
[01:31] For example, the first thing
[01:33] you might want to do in an upgrade,
[01:34] bundle update rails.
[01:36] Might be a multi-day journey through pain and suffering,
[01:40] trying to get your gem versions to work with Bundler,
[01:43] and you find out, uh, this gem's not maintained anymore,
[01:46] so what do you do about that?
[01:48] Or there's some weird fork and you're just like, really?
[01:50] I just want to run bundle update Rails.
[01:52] A standard command, right?
[01:54] Or sometimes there's changes in the public API.
[01:57] For example,
[01:58] how many people here remember the strong params migration
[02:01] from Rails 3 to Rails 4, right?
[02:03] Pretty big migration.
[02:04] And if you had a big app with a lot of controllers,
[02:06] this is a big change.
[02:07] But no matter what the technical challenges are,
[02:11] all of us here, we can solve these technical problems.
[02:14] And I say that pretty confidently, because if we couldn't,
[02:18] there wouldn't be any upgrades ever.
[02:20] And if there were no upgrades,
[02:22] none of us would be here
[02:24] 'cause who would be building stuff in Rails, right?
[02:28] So maybe we need to reframe this question
[02:31] about why is it so hard for organizations
[02:34] to work on these upgrades.
[02:36] And instead, it's why is it so hard for organizations
[02:39] to invest the time and money to do these upgrades?
[02:43] And to think about that, let's shift gears a little bit
[02:46] and talk about something that organizations do every day,
[02:50] and that's feature work.
[02:51] And let's see how this compares to Rails upgrades.
[02:54] So when your team embarks on building feature X, Y, Z,
[02:58] all the people involved, all the stakeholders,
[03:00] whether it's your product managers, UI/UX, engineering, QA,
[03:05] whoever needs to be involved in making that feature happen,
[03:08] all of y'all have some context,
[03:11] some kind of shared understanding
[03:12] of this is the app that we're working in,
[03:15] or this is our business domain, et cetera, right?
[03:18] All of y'all know something about the app,
[03:20] so you can talk about it.
[03:22] And because you have this shared understanding,
[03:24] it's easier to break up the work.
[03:26] And this might mean identifying the scope,
[03:29] but what's our appetite here?
[03:30] How much of this do we want to do right now?
[03:32] Or maybe it's breaking it up into small discrete steps
[03:35] that you can work on bit by bit by bit,
[03:38] shipping stuff incrementally.
[03:40] And because we can break up the work,
[03:43] it's easier, then, to share progress.
[03:45] You might say this feature,
[03:46] we broke it up into 10 cards and a couple cards in,
[03:49] we kind of know, are we on target for our deadline
[03:53] or do we need to have another conversation and say, hey,
[03:55] like, we found some new information out,
[03:58] we either need to adjust scope or our deadline here.
[04:00] Something like that, right?
[04:01] And all this is easier because everyone involved
[04:04] has some kind of shared understanding,
[04:06] and you can break that work up into smaller pieces.
[04:10] So how does this compare then to Rails upgrades?
[04:13] Well, for one, Rails has a lot of code in it, right?
[04:19] I've gotten deep in the weeds
[04:21] of ActiveRecord a couple times,
[04:23] and I'll say I know maybe 5% of the code in Rails.
[04:27] And if you saw Eileen's keynote yesterday,
[04:29] she said there's no core committer that knows all of Rails.
[04:33] So it's a really big code base
[04:35] that no one really understands, right?
[04:39] And Rails is crucial, crucial infrastructure
[04:42] in our application, because without it,
[04:45] we don't really have an app to talk about, right?
[04:47] So now we're talking about changing something
[04:51] that we don't know really well,
[04:54] and we want to change it, potentially, a lot.
[04:58] Who is going to sign up for this project
[05:01] that has so many unknowns?
[05:03] What could go wrong?
[05:04] How long is this going to take?
[05:05] When will it be done?
[05:08] Nah, no one signing up to work
[05:10] on this risky Rails upgrade project.
[05:13] But what if we could change
[05:15] the way we think about Rails upgrades
[05:17] and think of them more like feature work
[05:20] where we take a big black box of a project
[05:25] and we break it up into smaller pieces
[05:28] that we can work through incrementally,
[05:30] shipping stuff bit by bit
[05:32] and using the information that we learn as we ship stuff
[05:35] to help inform the next step we're going to do.
[05:38] If we could do that,
[05:40] we'd have upgrades that are more predictable,
[05:43] and if they were more predictable,
[05:46] they'd be more repeatable, because spoiler alert,
[05:49] there's going to be another Rails upgrade, right?
[05:52] So how do we get here?
[05:55] How do we get to having Rails upgrades
[05:57] that are more predictable and more repeatable?
[06:02] Well, first things first,
[06:04] we have to figure out what version
[06:05] of Rails are we even upgrading to,
[06:07] which might seem like a pretty obvious question.
[06:10] We just want to get to whatever
[06:11] the latest version of Rails is, right?
[06:14] Well, let's imagine we have an app running
[06:16] on Rails 5.2,
[06:17] and right now, Rails 7 is the latest version.
[06:20] So here's our big black box.
[06:22] We want to upgrade to Rails 7.
[06:25] Is there anything we can do to break this up,
[06:28] break this up into smaller pieces?
[06:30] Well, since we're coming from Rails 5,
[06:32] we can say Rails 7 is made up of two major versions,
[06:35] Rails 6 and Rails 7.
[06:38] And if we want to upgrade to Rails 7,
[06:41] we have to upgrade to Rails 6 no matter what.
[06:43] So we could try to do it in one big giant PR
[06:46] that changes everything,
[06:47] or we could just focus on upgrading to Rails 6.
[06:51] And now, our black box is looking a little smaller.
[06:56] Is there anything else we could do
[06:57] to break this down even further?
[06:59] Well, Rails 6 is made up of two minor versions,
[07:02] Rails 6.0 and Rails 6.1.
[07:05] And just like talking about going from 6 to 7,
[07:06] you have to do 6 no matter what,
[07:08] if you want to go to 6.1,
[07:10] you're going to have to upgrade to 6.0.
[07:11] So instead of trying to do that all at once,
[07:15] let's just upgrade to 6.0.
[07:17] And now, our big black box is looking a lot smaller.
[07:21] So what we're trying to do here is,
[07:24] when we're trying to figure out what version
[07:25] we want to upgrade to,
[07:26] is first target that next minor version.
[07:29] So if you're on Rails 5.0,
[07:31] next minor version is 5.1,
[07:33] that's what you're going to upgrade to.
[07:35] And once you get to 5.1,
[07:37] your next minor version is 5.2,
[07:38] so that's your next upgrade.
[07:40] And once you've exhausted all the minor versions,
[07:43] you're going to upgrade then to the next major version.
[07:46] So if you're now on Rails 5.2, there is no Rails 5.3,
[07:51] you're ready to go on to that next major version,
[07:53] Rails 6.0.
[07:55] And in doing this,
[07:57] we're making our upgrades smaller.
[07:59] Instead of trying to take years and years
[08:00] and years of changes to Rails
[08:01] and apply it to our application all at once,
[08:04] we're just going to focus on a narrow chunk
[08:06] that's right in front of us,
[08:09] and this will help drive down the risks of our upgrade,
[08:11] because again, we're not going to try to make a bunch
[08:13] of changes in one big go.
[08:17] So now we're starting to get a little more clarity, right,
[08:20] of what our target version we want to upgrade to.
[08:24] So now we're ready to run bundle update rails, right?
[08:28] Well, before we get ahead of ourselves,
[08:32] we need to talk about everyone's second favorite subject.
[08:36] We're already talking about Rails upgrades,
[08:37] which everyone loves,
[08:39] but we have to talk about everyone's second favorite subject
[08:42] and that's deprecation warnings.
[08:44] I know, I've got all the hits today, right?
[08:46] Okay, so I'm going to need some audience participation here.
[08:49] Can I see a raise of hands
[08:51] if you've ignored deprecations in your app recently?
[08:55] Yeah, right?
[08:56] Me too.
[08:56] That's what I do every day,
[08:58] ignoring deprecations because they're just warnings.
[09:01] They're not, our app's still working
[09:03] and nothing for us to do about 'em, right?
[09:05] But when you're starting on a Rails upgrade,
[09:07] it's really important that you look at them
[09:08] to see what they're saying.
[09:10] For example, imagine now that we're on an app
[09:13] that's running in Rails 6.0,
[09:14] and we see this deprecation warning
[09:15] flying through our logs.
[09:18] It's saying, accessing hashes from config_for
[09:22] by non-symbol keys will be removed in Rails 6.1.
[09:27] We're on Rails 6.0,
[09:29] so this thing that we're doing,
[09:31] accessing the hashes with non-symbol keys,
[09:33] still works fine, right?
[09:35] But when we go to Rails 6.1, the next minor version,
[09:39] it's not going to work anymore.
[09:41] And the last thing this deprecation warning is telling us
[09:43] is use symbols for access instead.
[09:46] It's not saying upgrade to Rails 6.1
[09:48] and open that giant PR that's changing
[09:50] a ton of things, right?
[09:52] It's saying use symbols for access instead,
[09:54] and this is something you can do in your application today.
[09:58] And we can do that for all
[09:59] the deprecation warnings, right?
[10:00] We don't have to wait for a huge big effort
[10:03] to get our app one step closer
[10:06] to working on that newer version of Rails.
[10:09] And the good news is,
[10:11] we don't have to fix all these deprecations at once.
[10:14] For example, imagine you have a deprecation warning
[10:17] that's popping up across hundreds
[10:18] of files in your application.
[10:19] It's a huge mess.
[10:21] You could try to open it in one giant PR
[10:24] and end up changing a ton of files,
[10:25] but no one on your team is going to want to review them.
[10:28] And eventually, you find somebody who's like, look,
[10:30] don't worry, it's just deprecations,
[10:31] just gimme a check mark, please.
[10:33] So, okay, you get your check mark.
[10:34] But when it's time to hit that merge button
[10:36] and deploy this thing to production,
[10:38] you're going to be pretty nervous
[10:39] because you're changing a lot of stuff in your app.
[10:42] Instead, you could fix that deprecation warning in one file.
[10:48] If you change it in one file,
[10:49] it's going to be easier to talk about,
[10:50] easier to get your team to review it,
[10:52] because okay, it's a smaller change,
[10:53] we can see what's going on here, right?
[10:56] It's going to be easier to test,
[10:57] and it's going to be easier to verify that,
[10:59] yeah, we made this change,
[11:00] and it's working on production the way we expected it to.
[11:03] And with that information, the stuff that you're learning,
[11:06] it's going to be easier then for you to come back
[11:08] and confidently make that change
[11:10] to all the other files in your application.
[11:14] So now you take your deprecation warnings
[11:15] and you divvy them up across the people on your team.
[11:17] Say, Hey, everyone,
[11:18] pick one up and fix it here, fix it there.
[11:21] And you're working really hard, working diligently,
[11:24] getting your app, again, one step closer
[11:26] to running on a new version of Rails.
[11:28] And lo and behold, someone comes behind you
[11:31] and brings back a deprecation that you just fixed.
[11:33] And this is a super, super frustrating experience
[11:36] 'cause you're working so hard,
[11:38] it's like, okay, I'll fix the deprecations.
[11:39] This is like a good thing to do.
[11:41] And why would a developer do this?
[11:42] Why are they not paying attention
[11:44] to all the work that we're doing?
[11:45] But we just said a couple minutes ago,
[11:48] everyone's ignoring the deprecation warnings.
[11:50] So can we really blame a developer
[11:52] for doing something that we all do every day?
[11:56] No.
[11:57] Instead, we can make it easier for them to adopt
[12:00] the patterns that we're going to need to use going forward.
[12:03] So here's a blog post I wrote up about one strategy I like
[12:06] to use when dealing with deprecations,
[12:09] and if you Google,
[12:10] you'll find a bunch of different ways to do it,
[12:12] but they all kind of have the same core idea.
[12:16] Once you've fixed a deprecation,
[12:18] you'll set it up in your app so it errors in dev and test.
[12:21] That way that developer, like me,
[12:24] who's used to writing code a certain way,
[12:26] when they're writing tests
[12:27] or they're trying to boot up the app or whatever,
[12:29] and they write deprecated code,
[12:31] the computer's going to yell at them
[12:32] and say, nope, you can't do this anymore.
[12:34] And because nobody likes the computer yelling at them,
[12:37] eventually, they're going to figure out, okay,
[12:39] I need to adopt this new pattern going forward.
[12:42] And we'll couple that with logging in production
[12:45] because tests aren't perfect,
[12:47] we're never going to catch 100% of everything,
[12:49] so we'll log these on production
[12:51] to give us one last fail-safe
[12:52] to make sure there's no deprecations slipping
[12:55] through the cracks.
[12:58] And in doing this, we're shipping code.
[13:00] We're shipping code bit by bit by bit.
[13:03] Again, that's getting us one step closer
[13:05] to that new version of Rails.
[13:07] Wouldn't it be nice if we could stay in this mode
[13:10] for all of our Rails upgrade?
[13:12] Well, let's see what we can do.
[13:14] So now that you fixed all your deprecation warnings,
[13:17] you're ready to run bundle update rails
[13:19] and you check out a new branch.
[13:22] Nothing new there, right?
[13:23] We always check out a new branch
[13:23] when we do something new in our application,
[13:26] but for as long as this branch is living,
[13:30] you're driving up the risks of your upgrade.
[13:33] So let's imagine you're on your branch down here
[13:35] and you spend a couple days doing the Bundler battle
[13:38] to get the new version of Rails to install,
[13:41] and eventually, you get it to work.
[13:45] And in all that time you keep
[13:46] on working away on this branch,
[13:48] development's been cruising along on main.
[13:51] And not a big deal, it's only been maybe a day or two,
[13:53] so you pull those changes in from main to your branch.
[13:57] And development keeps happening on main,
[13:59] and there was some security patch
[14:01] for another gem that got released in that time,
[14:03] so they went ahead and made that change
[14:04] to the Gemfile, pushed it up.
[14:06] So now, when you come back another couple days later,
[14:09] after you've been trying to fix tests or get the app to boot,
[14:12] you rebase and you run into some conflicts
[14:14] because now the Gemfile's changed in two places,
[14:16] it's like, ugh, this is kind of frustrating,
[14:18] but maybe it's not too bad.
[14:20] You figure it out in an hour or something like that.
[14:23] And then your boss comes through,
[14:24] and says, whoa, whoa, whoa, whoa, whoa, whoa, whoa.
[14:27] We have a really important urgent thing we need
[14:29] to work on right now, so please,
[14:31] get to a stopping point on this upgrade,
[14:33] and we'll get back to it when we have time.
[14:34] You say, okay, I'll push up my changes to my branch,
[14:36] and we'll come back to it, no big deal.
[14:39] And development cruises along on main.
[14:43] Time passes.
[14:44] Could be weeks, months, years.
[14:48] I've seen it.
[14:50] And you come back and now your branch is so far behind main.
[14:54] You try to pull the changes in,
[14:56] but now, the Gemfile's changed again.
[14:57] It's like, ugh, this is frustrating,
[14:59] I have to do this whole Bundler thing again,
[15:01] or those tests that you were working so hard to fix,
[15:03] now they've changed on main,
[15:05] so the changes that you made on your branch
[15:06] don't make sense anymore,
[15:08] and you're scratching your head like, what do I do?
[15:09] How do I get this to work?
[15:10] And in the end, you're like, you know what?
[15:13] I'm just going to start this upgrade over from scratch.
[15:16] And that is a huge risk 'cause what we're saying there is,
[15:20] if at any point we have to stop our upgrade
[15:23] because life changes, things happen,
[15:26] we could lose all of the work
[15:27] that we've done up to that point,
[15:30] and that's a huge bummer,
[15:31] and no one wants to be in that situation.
[15:33] So what can we do to get out of this?
[15:36] Well, we can turn to a strategy called dual booting.
[15:39] And what you'll do with dual booting
[15:41] is you'll set up your application on the main branch
[15:44] so it can run on either the production version of Rails
[15:46] or the target version of Rails you're upgrading to,
[15:49] and you'll have some kind of switch,
[15:50] and by default, that switch will usually be off.
[15:52] And when it's off, your app's running
[15:55] on the production version of Rails.
[15:57] Nothing out of the ordinary there, right?
[15:59] It's just the same thing that's always been happening.
[16:01] But when the switch is turned on,
[16:04] we'll be running the target version of Rails
[16:06] on our app on the main branch.
[16:10] There are a lot of different strategies
[16:12] out there for dual booting.
[16:13] I like using Bootboot.
[16:15] It's a Bundler plugin made by Shopify,
[16:17] and I like using it so much that I'll bend over backwards
[16:21] to get it working on Rails 4.
[16:23] Story for another day.
[16:26] So now, when you go back to your branch
[16:28] and you run bundle update rails,
[16:31] you're not just going to be doing the Bundler dance
[16:33] to get the new version of gem,
[16:34] the new version of Rails installed,
[16:36] you'll also be adding that dual booting functionality,
[16:39] so you can toggle the app back
[16:40] and forth between Rails versions.
[16:42] And as soon as you get those two things working,
[16:45] you can bundle the new version of Rails,
[16:47] and you have dual booting enabled
[16:48] so you can toggle Rails versions.
[16:51] You ship that code.
[16:52] You're not going to wait to say,
[16:53] oh, let me get these tests, I can get these tests to work.
[16:55] Mm, mm.
[16:56] Or, the app's not booting, let me make sure that, no.
[16:59] As soon as you get dual booting working
[17:02] and you can bundle a new version of Rails,
[17:03] you ship that off to production,
[17:05] because from that point forward,
[17:07] any changes that you need to make
[17:09] for your Rails upgrade will be on main,
[17:11] and after today, I never want to hear again
[17:14] somebody maintaining a long-lived branch
[17:16] that they're rebasing every day
[17:17] for months for their upgrade.
[17:19] You don't have to do that anymore.
[17:20] Dual booting is here, please use it.
[17:23] And if you do that, then, you're shipping code.
[17:26] You're shipping code every day to main,
[17:28] and you're driving down the risk of your upgrade.
[17:32] But if you're following along with me,
[17:34] you might be asking yourself some questions
[17:37] of like how is this going to work exactly,
[17:39] because there's got to be new stuff
[17:42] in the new version of Rails
[17:44] that isn't in the old version of Rails,
[17:46] and somehow we're going to deploy that code to main,
[17:48] but how's it going to work without the new version of Rails?
[17:52] I'm just confused saying that.
[17:54] Like, how is this going to work, right?
[17:57] Well, so once you go down this strategy
[17:59] where you have dual booting in place,
[18:01] you're going to see conditionals
[18:02] like this pop up in your application.
[18:03] And here, let's imagine we have an app running on Rails 5.2.
[18:07] We'll say, if our app currently is running on 5.2,
[18:11] here's the code that we want to run on production.
[18:14] Otherwise, here's the code that we'll be using
[18:17] in the future version of Rails.
[18:19] And now, let's just ship the code
[18:21] for both version of Rails to main.
[18:23] And you might see a few dozen
[18:26] of these or whatever, you know.
[18:28] Some of these pop up in your code base, not a big deal.
[18:30] Sometimes there's method signatures,
[18:32] change in Rails versions, that kind of stuff.
[18:35] But if you start to see the same pattern of if-elses
[18:38] repeat over and over and over again,
[18:43] that's going to drive up the risk of your upgrade
[18:46] because that's an indicator that there's some pattern
[18:49] that your team's used to that's not going to work going forward
[18:54] and you're going to work hard adding those if-elses,
[18:57] if-else, if-else, right,
[18:58] every time somebody comes back and writes code
[19:00] that the way they're used to writing it,
[19:03] and you'll be stuck in a game of Whack-a-mole
[19:05] that's going to be really hard to win
[19:07] because all these things are always going to pop back
[19:09] up in your application,
[19:10] and you're never going to know, like,
[19:11] did I catch all these cases?
[19:13] What's failing now, right?
[19:15] Not where we want to be.
[19:17] So what can we do here?
[19:20] Well, we can use a strategy called backporting.
[19:22] And what you'll do with backporting
[19:24] is you'll take whatever that new thing is in Rails,
[19:27] whether it be a new feature or a new way of calling code,
[19:30] whatever it is,
[19:31] and you backport it to the old version of Rails
[19:35] that your app is using
[19:36] so then your team can start writing code using
[19:40] that new feature before we finish our upgrade.
[19:44] And there's an excellent open source example
[19:47] of this with Strong Params.
[19:49] We talked about that briefly at the beginning of this talk.
[19:51] Strong Params was released in Rails 4,
[19:53] and again, it was a big change in the way you have
[19:55] to dealt with your parameters.
[19:58] At the time that Rails released Strong Params in Rails 4,
[20:02] they also released the Strong Parameters gem
[20:05] and this gem targeted Rails 3.
[20:08] So there's your backport.
[20:09] This is Strong Parameters that you could add
[20:12] to your app running on Rails 3.
[20:15] And the way it worked was is,
[20:16] if you have a model in your app, let's say it's a Post,
[20:20] you add the gem to your Gemfile,
[20:23] and then you have this module available to you
[20:26] that you can include to opt in to the new behavior,
[20:29] and you would include it only in the one model
[20:32] and work on migrating that model
[20:34] so it uses params the way we need to use it for Rails 4,
[20:37] the way we need to use 'em going forward.
[20:39] So you work on doing that migration,
[20:41] you test everything out, verify it, ship it to production,
[20:44] and once you ship it to production,
[20:47] no one can use params the old way anymore
[20:50] because this module will error and tell people,
[20:52] nope, you can't do that anymore, right?
[20:55] So as soon as you do that, you do it for the one PR,
[20:57] make sure it's working, everything's good.
[21:00] You then do the same process for the next model
[21:03] and the model after that,
[21:05] shipping small PRs bit by bit,
[21:07] migrating towards how things are going to work going
[21:10] forward in a new version of Rails.
[21:12] And once you've done it for all
[21:14] of your models in your application,
[21:16] you can include it globally
[21:17] for all of the ActiveRecord models in your app.
[21:20] So from that point forward, someone adds a new model,
[21:23] and they have to use params the new way.
[21:26] And again, this is an example of a backport
[21:28] that the Rails core team made for us,
[21:31] and it's really helpful, right?
[21:33] But we can do this ourselves.
[21:36] We don't have to depend on Rails
[21:38] to make all our backports for us.
[21:40] So here's a blog post that I worked on talking
[21:43] about a similar situation I ran into in an upgrade
[21:45] and how we implemented the backports
[21:47] and helped the team adopt a new pattern,
[21:51] because that's really what we're trying
[21:52] to do with these backports.
[21:53] We're trying to coach our teams to adopt a new patterns
[21:57] that we're going to need going forward.
[22:00] And depending on when you bring your backport in,
[22:02] they might be writing code for that new version
[22:05] of Rails for months.
[22:07] So, when you do make that switch on production
[22:09] to running the new version of Rails,
[22:11] they're not going to be surprised that,
[22:12] oh, I can't write code like this anymore, what happened?
[22:14] No, they might have been doing
[22:15] it for a long time already, right?
[22:18] And again, we're shipping code,
[22:20] always shipping code bit by bit by bit,
[22:23] getting us closer to that new version of Rails.
[22:27] So things are going good,
[22:28] we feel pretty confident about our Rails upgrade,
[22:31] but we have one more big black box to talk about,
[22:35] and that's test failures.
[22:37] On all of the upgrades I've worked on,
[22:39] I've seen teams spend the majority
[22:41] of their time fixing failing tests.
[22:44] And what can I say about test suites?
[22:47] They're big, slow, complicated, brittle, hard to understand.
[22:53] I could keep going.
[22:54] (Ali laughing)
[22:56] So what do you do when you're working on a Rails upgrade
[22:59] and you see hundreds or thousands
[23:01] and thousands of test failures
[23:03] in these big complicated test suites?
[23:06] We've been talking today about breaking things
[23:07] up into smaller pieces
[23:09] that we can work through incrementally,
[23:11] so if you're seeing thousands
[23:13] and thousands of test failures,
[23:15] you could break up the work that way and say,
[23:17] everyone, take one of these failures,
[23:19] figure out how to fix it and open your PRs,
[23:22] and we'll ship little tiny PRs bit by bit by bit, right?
[23:26] But personally, I don't want to open thousands
[23:29] and thousands of PRs,
[23:31] or even worse, write thousands and thousands
[23:33] and thousands of Jira tickets, right?
[23:35] No one wants to do that.
[23:36] So there's got to be some middle ground here
[23:38] between having a giant PR that's fixing a bunch
[23:41] of unrelated tests
[23:43] and having little tiny PRs that are narrowly focused
[23:46] where we're doing the same kind of things, potentially,
[23:48] over and over again.
[23:50] What can we do?
[23:52] Well, we could try to group our failures.
[23:55] So let's say we look at our tests,
[23:56] we look at all of our failures,
[23:59] and we notice a bunch of tests failing for,
[24:01] expected 200, got 400.
[24:04] Well, we could pull those off as a group, and say,
[24:07] two people, go look at those tests
[24:09] and see why are they failing?
[24:11] Are they failing for the same reason or not?
[24:13] And after a couple days, they look at it,
[24:14] and they say, oh, it looks like there's a bunch
[24:16] of tests failing for reason A,
[24:18] and then the rest of them are failing for reason B.
[24:21] Well, you can break that down then
[24:22] into two other groups of tests, right?
[24:25] And you can say, person one, go ahead
[24:27] and work on that first group of tests
[24:29] and the other person can then tackle
[24:31] the other group of tests.
[24:33] Or maybe you see a bunch of tests failing
[24:35] for ActiveRecords::StatementInvalid
[24:38] you can't do this anymore.
[24:40] You can pull those apart as a group,
[24:42] and they're probably also failing for the same reason.
[24:45] So give somebody that piece of work to work on,
[24:48] and they can work through a solution.
[24:50] And you'll do this for all the failures in the test
[24:52] where you're trying to identify,
[24:54] trying to identify all these different groups.
[24:57] And you'll start to see they're just finding chunks of work
[25:00] that are just the right size.
[25:03] They're not too big where you have a bunch
[25:06] of unrelated changes going on,
[25:08] and they're not too small where you're just opening
[25:10] the same PR kind of over and over again, right?
[25:13] Kind of a Goldilocks situation.
[25:17] And the grouping you do is going to depend on your application
[25:20] and the failures you're seeing.
[25:22] So for your upgrade,
[25:24] it might make sense to group the failures by file.
[25:27] Totally valid option.
[25:29] Or maybe you run your tests on CI across 40 containers
[25:33] and you have 15 failing.
[25:34] You say, okay, everyone take a container
[25:36] and make that container green.
[25:38] Again, valid option.
[25:39] Depends on your application, the failures you're seeing,
[25:42] and you know, the upgrades you're working on.
[25:45] And what we're trying to do here, again,
[25:46] is find chunks of work that are the right size
[25:49] that keep us in a state where we are building momentum
[25:52] and we're shipping code,
[25:54] always shipping code bit by bit by bit
[25:57] to get us closer to running on a new version of Rails.
[26:02] So now we've fixed all of our deprecations,
[26:04] we've got dual booting working,
[26:06] and we fixed all of our tests.
[26:08] So we're ready to make that switch from production,
[26:10] from the production version of Rails
[26:12] to the new version that we're upgrading to.
[26:16] Are we though?
[26:18] Even after you've done all this hard work,
[26:21] getting your app, again, closer to the new version of Rails,
[26:25] it can be really hard to assess
[26:27] what's the risk left of making this switch?
[26:30] So at this late stage of our upgrade,
[26:33] is there anything we can do to drive
[26:35] down the risks involved in making the switch?
[26:40] Well, we can try small experiments
[26:42] in production environment, excuse me.
[26:44] We can try small experiments in production-like environments
[26:48] to see if the upgrade is behaving the way we expected it.
[26:51] So you can imagine, you have your app running in production,
[26:54] and you decide we're going to make a deployment
[26:56] that has the upgrade that's for internal use only.
[26:59] And we ask all of our employees
[27:00] to please use this upgrade version of the app,
[27:03] and we'll be monitoring our Datadog dashboards
[27:06] or Rollbar errors,
[27:07] whatever observability you have to see,
[27:10] are we seeing new errors?
[27:11] Is the app working the way we expected it to or not?
[27:14] Might call this dog filter, right?
[27:17] Or maybe you want to try something like a canary deploy
[27:20] where you say, we'll ship the upgrade
[27:24] to 5% of our traffic for two minutes.
[27:28] Might have some impact on your users,
[27:30] but it's a really small experiment.
[27:31] We're only going to deploy this out for 5%
[27:33] of our traffic for two minutes.
[27:35] And after that's done, we're going to pull it back.
[27:37] And in those two minutes, again,
[27:38] you have all your dashboards,
[27:40] and you're monitoring everything up to see,
[27:42] are new errors coming through?
[27:43] Is this working the way we expected?
[27:44] All that stuff.
[27:46] And after a couple minutes, you're like, oh yeah,
[27:48] there are a bunch of errors that we did not expect to see.
[27:50] So you go back, fix those errors,
[27:53] and you say, okay, when we come back
[27:54] and do this experiment again,
[27:57] we expect there to be less errors, right?
[27:59] Because if we fix them, the error should go down.
[28:01] So you do that, you see less errors,
[28:04] and you say, okay, let's go ahead
[28:06] and bump this up then to 10% of our traffic for two minutes.
[28:09] And you do the same kind of dance,
[28:11] maybe you find some new errors, go back and fix it,
[28:12] try to experiment again.
[28:14] And after a while, it starts looking really good,
[28:17] so you say, let's bump this up to 10%
[28:18] of our traffic for five minutes,
[28:21] and this time, whoa, wow, it's looking really good.
[28:24] And then finally, you're like, you know,
[28:26] let's bump this up to 40% for 10 minutes,
[28:28] because we're feeling really good about the upgrade.
[28:31] We're not seeing new errors pop through.
[28:33] It seems to be that the app is working really well,
[28:35] and we're starting to feel more confident
[28:38] about this upgrade that we've been working on.
[28:41] And that's really what we've been trying to do,
[28:44] talking through all these different strategies today.
[28:48] We're trying to drive down the risks of our upgrades,
[28:52] gaining new information to help us feel more confident
[28:55] that we're not going to cause an incident
[28:57] by shipping out a Rails upgrade.
[28:59] And if we can do that,
[29:01] we'll all be shipping zero downtime Rails upgrades.
[29:06] All right, so the first upgrade that you do like this,
[29:11] it's going to be the hardest.
[29:13] It doesn't really matter what version
[29:15] of Rails you're on or anything like that.
[29:17] It's going to be the hardest because your team,
[29:19] your organization, probably has never done
[29:21] an upgrade like this before.
[29:23] So you have to figure out, what tooling do we like?
[29:25] How do we want to divvy up the test failures.
[29:27] All that stuff, you have to sort through that, right?
[29:30] And in that process, you're going to be learning a ton.
[29:34] You're going to be learning, oh,
[29:35] we really like using this tool for deprecations,
[29:37] or we were really happy with the way
[29:39] the deploy went out in doing it this way.
[29:41] Whatever it is, you figure that out,
[29:43] document it, reflect on it,
[29:45] and that's going to help you build a playbook, right,
[29:47] a process where this is how we're going to do
[29:50] the upgrade next time.
[29:52] And if you do that,
[29:53] when it's time to do the second upgrade,
[29:55] might be a little bumpy,
[29:56] but it's going to be a little bit easier.
[29:59] And by the time you get to the third upgrade,
[30:02] you're going to be cruising, and like, okay,
[30:03] we've done this before,
[30:04] we know the in and outs of this thing,
[30:06] no problem, we can do it.
[30:09] And maybe one day, you'll get to a place
[30:12] like where GitHub is,
[30:13] where they're upgrading Rails weekly, right?
[30:17] I don't know if anyone wants to live
[30:18] on the cutting edge like that,
[30:19] always running the latest version of Rails on edge, right?
[30:24] But it goes to show,
[30:25] once you refine your process and figure it out,
[30:28] these are the tools that we like,
[30:29] this is the way we want to do it,
[30:30] you can get to this point where it's every week,
[30:32] not a big ceremony to do another Rails upgrade.
[30:36] And as you go through this journey,
[30:38] learning these new things
[30:39] and applying different things to your upgrades,
[30:43] share what you've learned.
[30:44] As I showed you at the beginning of this talk,
[30:46] every year there seems to be another talk
[30:48] about Rails upgrades.
[30:49] So there's always going to be something new to share,
[30:52] so please come back, share what you learn,
[30:55] and you can help everyone else
[30:57] also ship zero downtime Rails upgrades.
[30:59] Or if you don't like public speaking,
[31:01] you can write a blog post.
[31:02] I have a bunch of those too.
[31:05] So that's all I have today.
[31:08] If you have any questions about Rails upgrades
[31:10] or anything, really, feel free to email me,
[31:14] aliwritesback@gmail.com.
[31:16] And I'm also on the Slack for however long
[31:18] that's going to be around, at my full name.
[31:20] And again, I mentioned a few blog posts in this talk.
[31:23] You can find them all on the Test Double blog
[31:25] and other blog posts about upgrades, too,
[31:27] and a bunch of other good stuff.
[31:30] And there's a lot of us here from Test Double.
[31:33] I really enjoyed Daniel's talk yesterday talking
[31:35] about legacy code and being more curious
[31:37] and how can we make it better.
[31:39] I'm looking forward to seeing Landon speak, soon,
[31:43] in a few minutes about machine learning and Ruby.
[31:45] I think that's going to be really cool.
[31:47] And Megan and Justin going to be having
[31:50] a little town hall tomorrow,
[31:51] making standard happen, even better for all of us, right?
[31:55] So that's all I have.
[31:57] Thank you.
[31:58] (audience applauding)