Angi's services are in high demand, and Handybook is an essential application. Featuring deep integration with third-party services and 40+ microservices, Handybook has a lot of moving parts. Aggressive caching, database sharding, and horizontal scaling are all important factors allowing the application to meet demand.
Hint (UpgradeRails.com's original owner) defined an upgrade manifesto with four focus areas:
Minimize work in progress
Upgrading Rails in a large, very active codebase is a risky business. Falling behind is easy, and if you're not careful, you may find yourself in merge hell. Practically speaking, this means no long-running branches. Instead, all changes must be PR'd to the main branch and be merged swiftly.
Incremental everything
No task is too small to deserve asking the question, "how can we break this up?" We minimize WIP and reduce business risk by taking many small, incremental steps. Put in practice, this means changes for compatibility are made in the smallest possible units, and when deploying changes into production, we perform a canary deployment, starting at 1% traffic and ramping up slowly.
Use data to drive decision making
Sometimes it isn’t possible to determine the correct solution for a given challenge by static analysis, even with excellent git history. In these situations, it's tempting to guess. But don’t do it; it's a trap! Instead, use dynamic analysis (often in production). Gather more information, and make decisions based on fact, not conjecture.
Over-communicate
We'll be making significant sweeping changes (incrementally) to the entire codebase. If something goes wrong and the symptom is outside our purview, we want to know about it. By continuously communicating project status, every member of the organization becomes a secret agent, reporting helpful information, resulting in a smoother upgrade.
This led to a defined upgrade plan:
- Dual Boot: In each stage of the upgrade (Rails 4.1 -> 4.2, 4.2 -> 5.0, and so on), use Shopify's excellent
bootboot gem to dual-boot with two separate sets of dependencies
- Changes for compatibility: obtain green CI results when the suite is run against both sets of dependencies while compatibility changes target to main branch
- Compatibility on all branches: once green CI is achieved, lock it in!
- Smoke test: QA in a production-like environment to ensure the upgraded application is performant
- Canary development: incremental roll out of the upgrade application alongside the current application
- Bake at 100%, strip & repeat: monitor then commit to the upgrade by copying over and cleaning up
