Intro
Documentation is a blessing and a curse.
I work on a client reporting team building a platform for replicating business data from a large Rails monolith with multiple domains into a data warehouse. In the future, the data warehouse will serve as the main source of truth for reporting applications. My team aims to empower other app teams to own their data in these reporting applications. The platform itself largely consists of app teams needing to implement several Ruby abstract classes and interfaces across a few repos to feed the data plumbing infrastructure underneath the hood.
To help other app teams succeed with our platform, I created a setup guide that details all involved repos and how to set them up locally, classes to implement, how to source data changes in the Rails app, and how to backfill and audit data quality in a step-by-step fashion. Once I completed the guide, I marveled and shuddered at what we had created.
There are a lot of steps and bespoke idiosyncrasies for local setup and implementations. Yet, having a visualization of our platform in the form of documentation provided an excellent signal that we can and should strive to simplify our system. If it appears overwhelming to those that built it, imagine how newcomers will feel.
Until we can make future simplification improvements, I posed to myself the problem of how we could ease onboarding for app teams at that moment. Attempting to transfer a mountain of context and knowledge via a lower fidelity medium is challenging (I couldn't figure out Matrix-style knowledge downloads for this unfortunately). Placing myself in another developer's shoes, I would feel like I'm navigating a ship in a sea of information enveloped by a thick fog.
However, what if I had a captain companion who knew this sea well and could traverse it carefully to avoid a shipwreck?
Claude Skills
At that time, my experience with AI-assisted work was limited. I used Copilot autocomplete in vim and had tried out Cursor with an older Claude model for some personal projects. I had recently heard of Claude skills, which had just been released maybe a couple weeks prior. I knew where the trajectory software development was headed with AI and realized I needed to level up my knowledge on AI tooling.
Therefore, knowing the challenge of helping teams onboard and desiring to increase my AI experience, I presented to my team the idea of creating a skill, well-versed with our data ingest pipeline, that could help a developer set up data ingest for their team's domain.
I wanted to ensure that a developer using the skill could still digest what the skill was helping them build and that the skill didn't just dump tons of lines of code on them, so I landed on a wizard-like skill that would take them step-by-step through the process with the setup guide as a reference.
I used Claude to help me write the skill. I went through multiple loops of manually test running the skill, noting issues along the way, and discussing the issues with Claude to incorporate changes into the skill.
For the rest of this post, I will share tips on what not to do, what worked well, what didn't work well, and a sneak preview of a future blog post on a new skill that will succeed this one. Whether you're a developer, an engineering manager, or even a product manager, I encourage you to read this through the lens of your work to see how you might enhance your own documentation for others through the use of Claude skills.
Tip #1: Context Matters
As I mentioned, I was a bit behind on LLMs despite using ChatGPT personally and using Cursor while coding. I had largely ignored words like "tokens" and "context windows". I quickly discovered why those words are important.
Initially I told the skill to just read in an entire document and run through it like a wizard. The document is over 1000 lines of Markdown and over 55,000 characters. Claude Code gave me the lovely warning about large token usage immediately. Of course in a trial session with the skill, Claude had to compact pretty quickly as it loaded the entire document and traversed the codebase to understand existing data ingest implementation examples.
Without a clear direction of how to be a helpful wizard with the setup guide, the skill was prone to overcomplicating and hallucinating, exacerbated by compaction. What I discovered is that managing context both in token usage and focused prompting for Claude was critical to the skill's success.
Takeaways:
- Claude's reading a whole document can eat up tons of tokens
- Claude will not magically absorb your mental context through the document
Tip #2: Provide Direction
After realizing my error from the previous section, I opted to try a chunked, more directed approach with the skill. I updated the skill to work in steps with sections similar to the document without duplicating document information. The skill instructed Claude to read in sections of the document by headings and to only focus on aspects of that section that pertained to Claude's role in data ingest setup.
Giving Claude a direction of how to actually use the document as a setup wizard allowed me to provide more of my mental context of what's important for data ingest, allowing Claude to stay focused and reduce token usage. Granted, reading in the document in chunks alleviated the upfront token cost, but over time it eventually accumulated lots of tokens because we just spread the document context over time within the session. The next tip will cover that.
Takeaways:
- Let Claude read in portions of the document instead of it all
- Provide direction and context on what's important to Claude with each section read
Tip #3: Persist Context
At the end of the day, I still needed the skill to understand the whole document in order to accomplish its goal. I reflected on what helps software development teams succeed. We break up work into smaller tasks and deliver incremental value. Trying to build or review (pull requests) everything at once strains our own ability to hold in all the context. We also make a lot of design and tradeoff decisions while we implement features (which we also document, right?).
So, the skill should embody the same principles of how a software development team works. I brainstormed with Claude and we developed the idea of a progress file that the skill would maintain as it helped a developer set up data ingestion for their domain. The progress file is a Markdown file that follows a template format and serves these functions:
- Checklist of all data ingestion steps to track progress of an implementation
- Captures notes on the app team's domain data
- List of domain data fields to ingest
- Capture design decisions made by the developer while using the skill about e.g. caveats related to ingesting their domain data
I updated the skill so when it starts, it asks the developer what database table or domain they want to ingest and then searches for an existing progress file. This allows developers to break up the work across multiple Claude sessions so they can commit smaller PRs and avoid needing to compact conversations from using up context windows. The notes in the progress file also provide a new Claude session important context about design decisions from previous sessions.
Takeaway:
- Persist important context in a separate file to break up work and allow picking back up in new sessions
Tip #4: Bridge the Gap
Note: this tip is more specific to my use case, but I believe there is a high level aspect worth exploring for any adaptation of a documentation wizard-like skill.
The previous tip hinted at this, but in order to be successful, the skill needs to know what data to actually ingest from a domain. The setup guide provides some guidance to developers on how to source their data for their data ingest pipeline. This process typically involves identifying which ActiveRecord models and attributes to use and how to publish changes to that data, usually through ActiveRecord callbacks.
Trying to digest the document on top of thinking through what data is valuable to ingest can be overwhelming, so when the skill asks which table or domain the developer wants to ingest for the first time, it will explore their domain within the Rails application to discover models, model attributes, and any existing callbacks or other events they can use for publishing data. The skill helps ease the tension for the developer in how to reason about their domain and what they can surface to the data warehouse. After researching, it presents its recommendations on the models, attributes, and events, and confirms with the developer what they actually want to ingest. This reduces the initial mental investment needed by the developer because now the skill can speak in their domain language to them and they can begin setting up data ingestion quicker.
Takeaway:
- Find opportunities to let the skill connect documentation to the human and their context, domain, etc., in order to ramp them up
Tip #5: Be Specific
Preface: this tip is nuanced based on my experience. I still recommend the following approach, but I would also recommend exploring Claude hooks, which I have not yet had the opportunity to do with this skill.
I've learned that writing skills requires a balance between brevity and specificity. Through several refinements, I pared down the number of words in the skill to help reduce token usage. However, I discovered from multiple manual evaluations of the skill that it could be inconsistent with ActiveRecord column discovery, naming conventions, and how to run automated tests.
This inconsistency in how it approached problems could lead to more token usage anyway as it encountered issues and tried different approaches, or I had to correct it. I realized I couldn't expect perfect determinism, but I wanted to rein in some aspects of the skill to be more consistent.
Before the step-by-step wizard portion of the skill, I added a section of conventions to help it know how to structure directories and name entities. For certain steps in the process, I updated it with specific instructions on how to do a task or to run a custom Ruby script. I also peppered the skill with lots of **IMPORTANT** or **CRITICAL** in front of prompts, which felt like a smell.
Overall, consistency and token usage improved considerably across multiple manual evaluations thanks to those changes. However, I still ran into the occasional issue where the skill ignored my **IMPORTANT** notes. When everything is **IMPORTANT**, nothing is **IMPORTANT**.
What I've learned from this issue is that brevity and specificity are still necessary, but so is responsibility management. I think the skill has too many responsibilities and the amount of prompting with so many constraints can overwhelm it and still lead to hallucinations and inconsistency. I'll expand on responsibility management next with how I'm incorporating this feedback into a newer skill.
Takeaways:
- Be specific about conventions and patterns
- Be specific about commands to run for consistency
- Create custom scripts the skill must use for true determinism
- Explore hooks as an even better approach for consistent command usage
Tip #6: Break Up Responsibilities
After many trial runs with this skill, I've taken my learnings and begun work on a new data ingest pipeline setup skill, focused on higher quality output of code and design decisions. Although my previous tips helped a ton with the skill, the realization of the skill having too many responsibilities prompted me to dive deeper into multi-agent workflows.
Claude subagents have existed for a while now as a means of parallelizing work and reducing the token usage in the main Claude session because the subagents have their own context windows to deal with the details of a task, and then they can provide just a summary of their work to the main session to avoid bloating its context.
Claude also supports agent teams, which allow a Claude session (team lead) to coordinate multiple other Claude sessions (not subagents) as independent teammates that can also communicate with one another. The typical use case is to spin up a team of a few general agents to take work off a task list similar to a backlog on a Scrum board.
However, nothing prevents you from creating specialized agents in an agent team that have specific roles and responsibilities.
By designating certain agents to take on specific tasks in the data ingest pipeline setup, I've been able to spread out responsibilities, enabling the agents to perform better at their one task through narrow scopes instead of shouldering multiple tasks like the previous one-session skill. Even the team lead has one responsibility as a coordinator and integrator, spinning and tearing down multiple teams depending on the steps in the setup.
For my use case, an agent team has been a boon, but for your use case, subagents might be just fine. You can still define agent files used by the subagents to give them specific tasks.
One advantage of the agent team is the cross communication that has let me create QA type roles that can review the output of other agents against acceptance criteria for the ingestion implementation
On the flip side, though, agent teams eat up a lot more Claude usage because each agent is a full-fledged Claude session. If your company has a nice enterprise setup, then usage may not be as big of a concern, but still be mindful of it.
Regardless of how you slice it up, spreading out the responsibilities can help reduce drift, confusion, and hallucinations when using a setup wizard skill like mine.
In my next post, I will expand further on what I've built with agent teams in the new skill.
Takeaways:
- Narrow scope of responsibilities by using subagents or agent teams to give specific roles to agents
- Let the main skill act as a coordinator to keep it focused and out of the implementation weeds
Conclusion
I've provided a sample of my learnings here, but just as context window matters for Claude, context for your specific use case matters too. What has worked for me may not work for you and your experience will likely differ.
I hope this sparks an interest in folks writing tons of documentation with how you might leverage AI to make your documentation more accessible to audiences as a companion. It reminds me of progressive web apps (PWAs) in that the documentation can still stand on its own, but having an AI partner can enhance the experience of applying the documentation similar to PWAs' using newer web APIs or technologies to enhance their UX. Returning to the metaphor at the beginning, your documentation is the map, and the skill is the seasoned captain that knows how to plot a course to the destination.
In general, what stands out to me beyond this specific kind of skill is that context management is critical. There are many tools at our disposal such as brief prompting, specific prompting, hooks, custom scripts, MCP server tools, persisting context and plans, subagents, and agent teams. I find that the more I embrace these tools, especially with agent teams, the more the work these agents do resembles how software engineers and human teams work.
We do best when we ship small, incremental changes that are narrowly scoped.
