Skip to main content
Test Double company logo
Services
Services Overview
Holistic software investment consulting
Software Delivery
Accelerate quality software development
Product Management
Launch modern product orgs
Legacy Modernization
Renovate legacy software systems
DevOps
Scale infrastructure smoothly
Upgrade Rails
Update Rails versions seamlessly
Technical Recruitment
Build tech & product teams
Technical Assessments
Uncover root causes & improvements
Case Studies
Solutions
Accelerate Quality Software
Software Delivery, DevOps, & Product Delivery
Maximize Software Investments
Product Performance, Product Scaling, & Technical Assessments
Future-Proof Innovative Software
Legacy Modernization, Product Transformation, Upgrade Rails, Technical Recruitment
About
About
What's a test double?
Approach
Meeting you where you are
Founder's Story
The origin of our mission
Culture
Culture & Careers
Double Agents decoded
Great Causes
Great code for great causes
EDI
Equity, diversity & inclusion
Insights
All Insights
Hot takes and tips for all things software
Leadership
Bold opinions and insights for tech leaders
Developer
Essential coding tutorials and tools
Product Manager
Practical advice for real-world challenges
Say Hello
Test Double logo
Menu
Services
BackGrid of dots icon
Services Overview
Holistic software investment consulting
Software Delivery
Accelerate quality software development
Product Management
Launch modern product orgs
Legacy Modernization
Renovate legacy software systems
Cycle icon
DevOps
Scale infrastructure smoothly
Upgrade Rails
Update Rails versions seamlessly
Technical Recruitment
Build tech & product teams
Technical Assessments
Uncover root causes & improvements
Case Studies
Solutions
Solutions
Accelerate Quality Software
Software Delivery, DevOps, & Product Delivery
Maximize Software Investments
Product Performance, Product Scaling, & Technical Assessments
Future-Proof Innovative Software
Legacy Modernization, Product Transformation, Upgrade Rails, Technical Recruitment
About
About
About
What's a test double?
Approach
Meeting you where you are
Founder's Story
The origin of our mission
Culture
Culture
Culture & Careers
Double Agents decoded
Great Causes
Great code for great causes
EDI
Equity, diversity & inclusion
Insights
Insights
All Insights
Hot takes and tips for all things software
Leadership
Bold opinions and insights for tech leaders
Developer
Essential coding tutorials and tools
Product Manager
Practical advice for real-world challenges
Say hello
Developers
Developers
Developers
Software tooling & tips

Code generation tools are easy to start, but hard to master

Software consultant Daniel Huss has experimented with using Copilot. Here's his review of code generation tools as a developer.
Daniel Huss
|
May 22, 2024
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

I’ve been stuck in a constant loop of subscribe-unsubscribe from a code-generating AI service for about a year now.

I’m a software consultant and noticing a pattern in my perpetual on-again, off-again relationship with Copilot that reminds me of how I felt during my undergrad.

Perpetually flipping between excitement and exhaustion, hype and despair, on a roller-coaster that goes something like:

  1. Class registration opens. I look at options and get excited by the possibilities.
  2. Term begins. The first few days are magic. New ideas are plentiful and I learn easily. I feel I could do this forever.
  3. The learning curve climbs. I have to apply knowledge myself in new ways. It’s still fun, but I get pushed to the edge of my understanding.
  4. Deadlines loom, as does my discomfort. Is education about blindly following directions? Or can I follow my heart to explore new things? I question why I do this, but want to see it through.
  5. Panic sets in. I have to triage or drown. I fall back on habits I often hate but that worked before. Frustration turns inward and I blame myself for not being ‘on top of it’. Fun is a distant memory.
  6. Somehow, I finish everything. It feels like nothing new came from this term. I’m tired. I’m getting out, and will never come back.
  7. I rest. My brain decompresses and can actually internalize what I flooded it with. Time washes away the panic. I start to feel thankful for what I learned along the way.
  8. Class registration opens…

My experience as a developer with Copilot

My experience as a software developer with Copilot matches pretty closely. I subscribe to the tool. I dig into a problem, start to write some code, and I’ll be! The autocomplete is so accurate I’m certain it read my mind. Except I didn’t even consciously think that yet, so it can also see the future. I know that’s what I wanted.

I dig deeper. The problem becomes more esoteric, painted with prior art that I need to account for.

Still, the incredibly fancy autocomplete AI tool hits the right thing often, and it’s done well for me so far. It sped my brain up, even if I have to stop and rework a suggestion or two for my context.

The reality of “I need to have something working” looms. My internal clock tends to turn up the pressure earlier than the deadline I’ve agreed to (or estimated). I’m deep in the problem, and have a nice mind-palace built of the system. Indirect cause and effect are clear to me.

The AI, in its rush to help, spits out a subtly incorrect block of code. On its own, incorrect is fine. More than once, however, the tool has spit out a brain worm. A new variable name, or a solution that is just reasonable enough to hijack whatever was going to come out of my subconscious.

I cannot un-think the suggestion, and the thing I wanted disappears. So I follow the AI and inevitably bonk my head on a syntax thing, or a side effect. The energy it takes to resist the magic computer breaks down the mindpalace.

Suddenly I look up, and I’m yak shaving again.

Now I’m upset, so I fall back to habits and old ideas to solve problems. I turn off Copilot and return to slower ways to work that have been effective so far. I wrap up the task and move onto something new. Somewhere a few weeks down the road, I remember I’m paying for a tool I’m not using.

Rest, repeat.

You might say I’m a bit of a legend

My brain is always hooked by the AI candy for the rote things.

Little side project idea? Auto-complete your way through the standard wiring to get at the fun part of the idea. Somewhere between that early joy and regular day-to-day work, I lose my control over the tool and it starts to control me.

I’m too aware of my fallible human nature, so I’m certain a lot of this problem has to do with operator error.

It’s probable I’m the character from an urban legend that came before self-driving cars were real, when cruise control was the hot new ticket in automobiles. The story is about a man who tries out the flashy cruise control feature on his brand new motor-home. He’s cruising on the highway and fires up that cool new tech. Satisfied, he gets up from the drivers seat and heads back to get a cup of coffee. Naturally the vehicle flies off the road, flips, and man sues the manufacturer because “instructions unclear.”

Maybe that’s exactly me with Copilot? (I promise I won’t bring a frivolous lawsuit!)

It’s rare when we get to spend time where a general code-completion tool is inherently an asset; the fundamentals of a framework, the well-trodden paths of data manipulation, clever JavaScript one-liners, or clear database queries. Especially in a language you don’t know. Those situations are the showcase for the value these tools hold.

I worked at a client with a very seasoned engineer who was part of taking over a large Rails monolith who had never written a line of Ruby. Copilot was the perfect ally for him to translate his vast knowledge into Ruby syntax.

For me, it helps shore up my leaky memory that cannot hold bash scripting or regex (even though I’d rather keep those things and forget the lyrics to I Want You by Savage Garden or the theme song to Arthur). I’ve also had success and fun doing an adversarial test-driven flow where I use the AI to write the code to pass the tests I write.

I get better results directing the LLM with a persona (Cranky Staff Engineer), and a method of thinking / responding (in the style of a succinct and firm pull request review of code you don’t like.)

Over time, I am writing better prompts to constrain the dreaming machine to help me be creative in turn, and in context. In those ways, I squeeze a lot more context-driven work out of a few suggestions.

Frankly, very little of what I’ve learned is helpful to you! It’s often not helpful to me, which is why I didn’t write a post called “How To Be A Staff Engineer By Using Copilot.”

This is more art than repeatable experiment, and it goes against a lot of ways I think when I am being a proper Software Consultant.

I’m still very hesitant to pull the tool off the shelf on client work, even on teams who are fully embracing them. I’m a huge sucker for fan of paying for tools really smart people have made that paper over my gaps. (Hi JetBrains, I love you.)

Due respect to the progress the builders of these tools have made. I’m poking fun at Copilot, but it’s grown leaps and bounds. There’s a reason I keep coming back. I just can’t quite figure out what it is, yet.

Every tool in its place

I truly want to find where Copilot can be a core part of my workflow. I’m just struggling to find where that’s productive when me and the tool are out of our depth together. It gets turned off, forgotten for a couple weeks.

Put another way, I have a blast until I get to the part of term where deadlines are looming, and I have to triage back to default habits and tools I know how to work with. That’s when I need my tools that don’t feel like they are actively trying to flip my winnebago on the highway. But I can think of a few ways I’m not using the tool to its greatest effect.

It’s a new term. I’m full of excitement and I know I can do it better this time! That registration button is open. I click it, and turn on the plugin in my IDE…

Are you a developer who’s tried an AI generation tool?

Join the conversation about this post over on our N.E.A.T. community (Not Everything’s About Technology).

Related Insights

🔗
How to tell if AI threatens YOUR job
🔗
Mastering automated refactoring tools: A practical guide
🔗
Forecasting the future: Intro to Machine Learning for weather prediction in Native Ruby

Explore our insights

See all insights
Developers
Developers
Developers
C# and .NET tools and libraries for the modern developer

C# has a reputation for being used in legacy projects and is not often talked about related to startups or other new business ventures. This article aims to break a few of the myths about .NET and C# and discuss how it has evolved to be a great fit for almost any kind of software.

by
Patrick Coakley
Leadership
Leadership
Leadership
Turning observability into a team strength without a big overhaul

By addressing observability pain points one at a time, we built systems and practices that support rapid troubleshooting and collaboration.

by
Gabriel Côté-Carrier
Developers
Developers
Developers
Why I actually enjoy PR reviews (and you should, too)

PR reviews don't have to be painful. Discover practical, evidence-based approaches that turn code reviews into team-building opportunities while maintaining quality and reducing development friction.

by
Robert Komaromi
Letter art spelling out NEAT

Join the conversation

Technology is a means to an end: answers to very human questions. That’s why we created a community for developers and product managers.

Explore the community
Test Double Executive Leadership Team

Learn about our team

Like what we have to say about building great software and great teams?

Get to know us
Test Double company logo
Improving the way the world builds software.
What we do
Services OverviewSoftware DeliveryProduct ManagementLegacy ModernizationDevOpsUpgrade RailsTechnical RecruitmentTechnical Assessments
Who WE ARE
About UsCulture & CareersGreat CausesEDIOur TeamContact UsNews & AwardsN.E.A.T.
Resources
Case StudiesAll InsightsLeadership InsightsDeveloper InsightsProduct InsightsPairing & Office Hours
NEWSLETTER
Sign up hear about our latest innovations.
Your email has been added!
Oops! Something went wrong while submitting the form.
Standard Ruby badge
614.349.4279hello@testdouble.com
Privacy Policy
© 2020 Test Double. All Rights Reserved.