SaaS Has No Box

Scott S NelsonLeave a comment

TL;DR: SaaS doesn’t remove implementation work, it relocates it: from servers and patches to configuration, declarative automation, and the complex weaving of platform features needed to deliver an enterprise capability over custom code written against vendor interfaces. Demos can be deceiving, though not always intentionally so. More often it’s the expectation that everything you need is available “out of the box” that makes SaaS look like a magic bullet instead of just another tool that takes skill and effort to build with.

When There Is No Box

SaaS can be the perfect architectural choice for an organization that wants to focus on business value instead of unique technology capability. You don’t buy SaaS, you rent it, and that distinction is the whole point: the vendor manages refresh cycles as part of the subscription  through continuous updates instead of it landing on the organization as a project it has to fund and staff itself. Traditional software, whether licensed on-prem or custom built (even “cloud native”), doesn’t work that way. The organization owns maintaining the platform and the business functionality, which is what kills custom builds and heavily customized on-prem systems over time.

OpEx to fix something that already mostly works is chronically underfunded, while CapEx for something new gets approved, so the old system just accumulates risk until it becomes a crisis project instead of routine maintenance. And when that crisis project finally lands, it doesn’t just replace the platform, it resets the maintenance maturity the team spent years building. Admins who knew every quirk of the old system now spend the next year reskilling on the new one instead of shipping features on a platform they already understood. Commodity capability is almost always better rented than lovingly handcrafted by an internal team armed with a five-year roadmap and an (overly) optimistic estimate.

That doesn’t make the product finished on arrival. The gray area is what it takes to turn a collection of separate-but-related features into an enterprise capability: level of effort, support, and everything the demo skipped. That demo was built for the business buyer with a checkbook, not the developer with a deadline. A single capability, shown on its own, looks like a checkbox. Implementing it by wiring together ten other features, each documented as if the others didn’t exist is a different job entirely, and it’s the job nobody demos.

Which brings us to one of the more durable phrases in enterprise software: “out of the box.” SaaS has no box. It has a tenant, a contract, a configuration console, a few license tiers, an identity integration, and eventually a backlog. No box.

The assumption worth challenging is that SaaS is a pre-built application, so implementation is mostly a matter of switching it on. A more accurate model: enterprise SaaS is a product (and, with successful vendors, a collection of acquired, competing, products) sitting on a platform shared by thousands of other customers. The product supplies standard capabilities. The platform supplies the ways to configure, automate, extend, secure, integrate, monitor (sometimes) and govern them. Both are legitimate. They are not the same implementation problem.

While there’s no box, SaaS still delivers many of the same benefits standard software has always had over custom builds, and a small company running a fairly standardized process can get close to production-ready with limited setup. Those are the things business users are so used to that they forget to count them: security, role management, standard integrations, audit logging, backup and disaster recovery, single sign-on, and standard compliance (support) built in rather than bolted on. A department adds users, roles, approvals, reports, and a few local variations, and configuration becomes real work. An enterprise running the same product across business units, regulations, and integrations turns it into a configurable platform that needs architecture and governance, whether or not anyone wrote a line of code. “SaaS” describes all three. It is not remotely the same project in each case.

Cloud Has Changed the Meaning of OOTB

“Out of the box” used to mean a shrink-wrapped disk that installed a working application on a server you owned. Cloud didn’t kill the phrase, it’s just no more literal than dialing a phone or pasting text. The concepts are still the same, though sometimes mis-applied because the meaning has been lost in translation.

To a vendor, “out of the box” usually means the feature exists in the licensed edition, fits the product’s intended model, and can be enabled or demonstrated without a separate application. That’s an honest intent. It’s also incomplete. The platform including the building blocks is not the same as an organization putting the thing into production next week with its own data, users, policies, integrations, processes, and exceptions.

To a business stakeholder, “out of the box” often means something closer to done: complete, supported, included in the purchase, ready with minimal effort, sensible about exceptions to processes that “everyone” follows. That’s a fair expectation for a consumer app. It gets unreliable fast in enterprise SaaS, because “normal” is in the eye of the beholder (or buyer) and the vendor is thinking about a normal defined by the lowest common denominator across their entire customer base.

To an engineer or an experienced administrator, it means the capability exists, there’s probably a supported configuration path, and there may also be three licensing constraints, a technical prerequisite nobody mentioned, and a limitation that won’t surface until a user asks a perfectly reasonable question in production.

The reason those three points of view keep colliding is simple: a feature demo is a planned, orchestrated, and practiced event to showcase what a product does. An enterprise capability is the business secret sauce baked into solution that relies on twice as many features and sometimes held together by bubblegum and baling wire. One is maybe out of the box. The other never saw a box, and the instructions were in a different language for a different purpose and required a lot of creativity that often goes unrecognized.

Enterprise SaaS is less a finished solution than a pile of Legos with an incomplete picture and no instructions (but sometimes a blog post, which isn’t always going to mention the complexity required or the dozens of failed attempts that came before it). The pieces are good. Plenty fit together well. But the pieces being in the box, metaphorically, since SaaS still doesn’t have one, doesn’t tell you what to build, whether the model survives the next upgrade, or who fixes it when someone changes a foundational rule six months from now.

Take a routing requirement. In a demo: a new request comes in, gets assigned by region, triggers a notification. Simple. In production: region comes from more than one system, the request touches multiple accounts, the assigned team depends on product line and contract status, some records are role-restricted, the notification has to respect local hours, and the downstream system is occasionally down. The feature didn’t get worse. The business process just showed up. (And that is a simple scenario. Most real enterprise requirements are far more complex.)

“OOTB” is a capability statement. It is not an implementation estimate, an operational guarantee, or a substitute for architecture.

The Simpler the Configuration, the More Complex the Code It Hides

None of these categories are moral rankings. The right choice depends on fit, lifecycle, foundation stability, and whether the organization can actually own what it builds.

An out-of-the-box capability is a feature the platform supplies. It may work as-is for the standard case. It usually needs configuration, permissions, data setup, licensing, and training before it’s useful, and it may not match the current process exactly, which is often a reason to reconsider the process before reaching for a customization.

Configuration means shaping standard behavior through supported settings: roles, page layouts, field validation, reports, routing rules, approval structures, notifications, integration endpoints. Once it affects production behavior, it’s implementation work, full stop. It needs requirements, design review, test cases, deployment control, documentation, and an owner. Calling it “not development” is technically defensible in the narrowest sense. It’s also how a team ends up with 437 rules, three administrators who understand half of them, and no confident answer for what happens when a customer’s status changes. It’s the same failure mode behind a real migration where teams leaned on undocumented flexfields to hit a deadline, then paid for it months later reverse-engineering their own configuration before the migration could finish (see Agile is not Ready, Fire, Aim for more on this).

Declarative development is a platform-provided way to define logic without conventional code: workflow tools, rules engines, visual process designers, formula fields, event triggers. No-code and low-code are still code. Not usually code in the sense of someone typing a language into an editor, but they still define conditions, state changes, data dependencies, permissions, and error paths. The platform abstracts the code away from the person assembling it. It doesn’t abstract away the consequences.

That abstraction is genuinely useful, and worth being honest about both directions of the trade-off. A well-designed visual workflow lets an administrator build real automation without implementing a transaction engine or learning the vendor’s internal data-access framework. The platform absorbs a lot: runtime behavior, orchestration, retries, security enforcement, scale. That’s real value, not a small one.

The cost is that the simplified surface can hide complexity from the person changing it. A few boxes connected by arrows can represent a substantial amount of execution logic, error handling, and dependency chains. The canvas looks calm. Production tends to have other opinions, usually delivered at an inconvenient hour. This is the same pattern behind the observation that easier isn’t always faster: stacking enough simple, individually-reasonable configurations produces a system that’s simple nowhere and complex everywhere at once (Sound familiar? See 5 Considerations About Low-Code/No-Code).

That’s not a flaw to be engineered away. It’s the mechanism that makes SaaS scale in the first place. A platform flexible enough to fit a hospital system, a bank, and a mid-size distributor without three separate codebases has to carry that flexibility as complexity somewhere. The vendor pushes it down into configuration and declarative tooling instead of custom code, which is a genuine improvement. It doesn’t make the complexity optional. It just changes who has to manage it and what tools they get to do it with.

That’s the shift underneath all of this: implementation effort now defaults to weaving platform features together through configuration and declarative tooling, not to writing custom code against the vendor’s exposed interfaces. Custom development still has its place. It’s just further down the list, not the default.

Custom development covers code within supported extension mechanisms, custom applications on the platform, middleware, integration services, and specialized interfaces. It is not automatically a failure of SaaS. Sometimes it’s the right response to a genuine differentiator. Sometimes it’s less risky than an increasingly tortured declarative workaround built by someone determined to avoid writing “real” code. A five-line extension with source control, automated tests, logging, and a documented owner can be safer than a heroic visual flow nobody wants to touch because the canvas now resembles an airport terminal map. The safer default is buy-before-build: standard components first, AppExchange next, custom code only when nothing else genuinely fits (see Enterprise Salesforce Team Recipe for a longer rant on this aspect).

The distinction that matters isn’t code versus clicks. It’s whether the resulting solution can be understood, tested, secured, upgraded, and supported by whoever inherits it. Microsoft’s own guidance on low-code governance is telling on this point: it talks about guiding “both professional and citizen developers who build custom applications for business needs” (see What Is Low-Code Governance and Why Is It Necessary?). A low-code solution is still a custom application. Custom applications need roles, controls, and oversight, regardless of how few characters got typed to build them.

“Can Be Done” Is Not “Is Supported”

This is worth repeating more than once: almost anything is technically possible with enough extensions, workarounds, APIs, and questionable judgment. Whether it’s supported is a different question entirely (insert Jeff Goldblum Jurassic Park meme here).

Being supported by the vendor means the mechanism is documented, stood behind, and preserved through normal releases. Being supportable by the organization means internal teams can diagnose, test, change, and hand it off. Sustainable means the whole thing stays economically and operationally reasonable as the platform, the process, and the team keep changing. A demo answering yes to “can it be done” tells you nothing about the other three.

Undocumented API behavior, browser automation standing in for a real integration, a customization that quietly modifies vendor-delivered files, a workflow built on a feature the vendor has already announced it’s retiring, a marketplace add-on with one maintainer and an optimistic roadmap: all technically working. None of them are the same commitment as a documented, versioned extension point.

The cost and risk of “custom” depends less on the label and more on the stability of what it’s built on. A stable foundation has published APIs with deprecation notices, documented extension points, sandboxes, deployment and rollback mechanisms, and a realistic talent pool that can maintain it. An unstable one has undocumented internals, screen scraping, direct database access outside vendor guidance, fragile third-party dependencies, and configuration too tangled to inventory. A small customization on a documented extension point can have a modest lifecycle cost. A small customization on undocumented behavior can turn into a recurring upgrade project. The line count, or the number of boxes in a flow, is a weak predictor. The stability of the surface being extended is a much better one.

Standard declarative features aren’t automatically exempt from this either. Salesforce ended support for Workflow Rules and Process Builder after December 31, 2025. Existing automation can keep running. Salesforce’s own documentation puts the distinction plainly: “Your active Workflow Rules and Process Builder processes continue to run as they do today,” followed immediately by the limitation that “customer support is not available, and bugs will not be fixed” (from Workflow Rules and Process Builder End of Support). Salesforce’s own stated reasoning for the consolidation was that spreading automation across multiple tools “hampers both maintenance and innovation” (see Go with the Flow: What’s Happening with Workflow Rules and Process Builder?), which is a reasonable product decision. It doesn’t change who pays for inventorying, testing, migrating, and retraining on what got built on the old feature. “We didn’t write code” was never a lifecycle strategy.

Which is where architects earn their keep, not by vetoing low-code or routing every request through a custom-dev project, but by distinguishing product configuration from business-critical custom behavior, spotting dependency and lifecycle risk before it’s load-bearing, and asking what happens when the vendor retires the thing three years from now. The job isn’t making the solution more complicated. It’s making the complexity visible early enough that the organization can choose where to carry it.

The Ocean View Capability

Every enterprise SaaS platform has Ocean View capabilities. They look excellent from the balcony. The demo is clean, the data is cooperative, the presenter knows the route, and the integration has already been arranged. The value is obvious.

Shiny things can be dazzling, but don’t mistake them for a trick. A ninety-second demo has to compress a hard problem into something presentable, and that compression is what quietly removes the cases that would make the problem visible.

Then implementation starts, and someone asks about identity federation, field-level security, migration errors, audit history, a regional exception, an approval delegation, an API failure, and the department that’s been running its actual process out of a shared mailbox since 2014. The ocean’s still there. It’s just farther from the lobby than the brochure implied.

Vendor blogs are good for learning what a feature is intended to do. They’re written to explain a capability, not to catalog every limitation, so they lean on clean prepared data, follow the happy path, and rarely dwell on edition-specific licensing, exception handling, or recovery scenarios. A vendor blog can show that a feature exists. It rarely proves the feature fits your operating model.

Demos are orchestrated on purpose, not deceptively, just deliberately: prepared dataset, existing roles and permissions, the expected route through the workflow, integrations already wired, no duplicate records, no outages, no data-quality problems, no last-minute policy exception. That’s not fraud. It’s theater. It’s designed to explain a capability quickly, not reproduce a customer’s production environment. The mistake is mistaking a successful demo for a fit assessment.

The fix is straightforward, if less exciting than watching a polished walkthrough: ask the vendor to run the organization’s actual process with representative data, not the generic version. Require the exception path, not just the happy one. Test the real identity, role, and approval requirements. Ask what happens when an integration or automation fails. Get the licensing tier requirements in writing. Ask about support boundaries and lifecycle commitments. The best single question in the room is usually, “Show me what happens when it doesn’t work.”

Gregor Hohpe, (the definitive author on architecture, IMO) states the reality experienced architects live in more plainly than almost anyone writing about this, goes further in The Quest for Low-Code. His starting point: there’s no magic in software. Every code-reduction mechanism implies a trade-off, and the evaluation job is naming which one you’re accepting. From there his checklist gets concrete: is what’s left actually low cognitive load, or just low line-count with the complexity relocated into a jumble of custom syntax buried in quotes inside a YAML file? How does the solution evolve, is there version control, can you roll back, does it hold up past the scale of the demo? And what happens on the unhappy path, do the error messages help or just leak the tool’s internals? His three vendor exercises put that last question to the test directly: introduce an error on purpose and read what the message actually tells you, leave the room and have the vendor break the thing and debug it live, or take the keyboard yourself and see how much hand-holding it takes to build something real.

Configure Deliberately, Extend Carefully

SaaS is frequently the right call. It reduces unnecessary ownership, accelerates access to mature capability, and spares an organization from rebuilding commodity software it has no business rebuilding. Standard functionality should be the default. Sometimes the business process should adapt to a proven product model instead of demanding a bespoke replica of every legacy exception it’s accumulated out of habit.

None of that means every use case gets covered for free. The same flexibility that lets the platform serve wildly different businesses is what makes your specific business’s edge cases someone’s implementation problem, not a checkbox someone forgot to include. And custom code isn’t automatically the harder path. A clean extension against a documented interface can take less time and less risk than the half-dozen declarative features it would take to fake the same behavior through configuration. The label on the technique, code or clicks, was never a reliable way to estimate the work.

Watch for the case study that shows someone getting a complex capability working by wiring together three or four platform features. That’s real, and it’s also incomplete: what made it into the write-up is the wiring, not the weeks of trial, error, and platform-specific knowledge that came before it worked cleanly enough to demo. A capability that took a seasoned admin forty hours to get right the third time doesn’t become a ten-minute task because the polished version only takes ten minutes to watch someone build.

And the decision to go with SaaS is not where the architecture ends. A workflow assembled in a visual designer can still affect revenue, customer commitments, regulatory obligations, and a support team’s weekend. An integration configured through a menu can still fail at 2:00 AM. A field renamed without documentation can still turn next year’s migration into archaeology.

Use the standard product where it genuinely fits. Configure it with discipline, and price that discipline honestly: time, effort, and cost scale with how far the requirement sits from the common case, because the common case is what SaaS was built to solve cheaply. The more unique the business need, the more of that cost the organization carries itself, whether it ends up in configuration, declarative automation, or custom code. Treat declarative automation as software, because it is. Extend stable, documented foundations when the requirement earns it, code included. And the next time someone says a capability is available out of the box, ask the only question that actually matters: out of the box for whom?

In enterprise SaaS, nothing is truly out of the box. The capability exists. What it costs to make that capability yours, in time, effort, and risk, is an architecture decision shaped by how far your need sits from the common case, not a vendor claim and not a line item copied from someone else’s case study.

If you found this interesting, please share.

Lessons Learned, Lessons Lost: Why AI Retrospectives Rule

Scott S NelsonLeave a comment

TL;DR: The Sprint Retrospective would be the most valuable ceremony in Scrum…if it weren’t also the one with the worst follow-through. AI can run the same ceremony and actually keep the results, because the lesson gets loaded before the next session instead of filed and forgotten. It’s a simple habit to learn, and there are plenty of free examples of how to automate it. It’s a small investment of time and thought that pays real dividends in productivity and confidence.

The Ceremony That Everyone Likes and Nobody Finishes

The 2020 Scrum Guide puts the Sprint Retrospective at the end of the Sprint, timeboxed to three hours for a one-month Sprint (not that many places do one-month Sprints anymore, or allot three hours for improving things). The process is simple: provide a safe space where people can speak their mind about what went well, what didn’t, and how they would change things to be better in the next Sprint. The goal is to capture ways to increase quality and effectiveness while reducing friction and wasted time.

In theory it’s the smartest ceremony of any SDLC. A room full of people who just lived through the same thing, comparing notes while it’s fresh, with permission to say what actually happened.

I was introduced to this ceremony before Agile approaches were common in the enterprise. We ran it after a major release, and I found the process very enlightening. Someone wrote up a summary of the findings and posted it on the company intranet.

This was a long time ago. Years later I remembered that document and got curious about whether anyone had ever gone back to it.

I was the only one who ever looked at it again.

I can’t recall a single piece of wisdom it captured. Not one. What I do remember is the aura of “this makes so much sense” when we discussed the stickies on the wall. And that the same mistakes were repeated across later projects, many with some of the same people in the room who had sat through the meeting where we agreed not to make them again. Sigh.

What Happens to the Findings

Follow the output and the pattern is easy to see.

Some items become real assignments. Someone owns it, someone completes it, and the team is measurably better off. That’s a real win and it’s worth saying so.

The problem is, it is too often a one-off win. The task closes. It’s checked off and moved to Done. Then the same thing comes up three Sprints later. Or the next Sprint.

Because the good stuff that gets surfaced in a retro isn’t something that can be fixed with a story. It’s a lesson. “We committed before we understood the integration.” “We let the staging environment drift.” “We assumed the client would review in three days and they took three weeks.” Those aren’t tickets. They’re rules for how to behave next time, and there is nowhere to put them where people will remember them (yes, there’s an exception, for those who are already ahead of the game with a living PROJECT.md, aka CLAUDE.md, AGENTS.md, or whatever your favorite harness uses).

So they go into a document. Which goes onto an intranet page. Which nobody opens during the next project, because during the next project everyone is busy doing the next project.

The retrospective produces knowledge with no delivery mechanism. Six weeks later a new team, or the same team with two new people, walks into the identical wall and holds an identical meeting about it. We call that continuous improvement. It’s closer to continuous rediscovery.

That failure, organizations not learning from their own experience even after making the capture of lessons a required part of the process, is a big part of why the Best Practices section on this site exists at all.

Run the Same Ceremony, by Hand

Now try it with AI at the end of a working session. At its most basic, this is two prompts.

Review this session. What worked, what didn't, what should change next time?
Give it to me as Start, Stop, Continue.

Start doing this, stop doing that, keep doing this other thing. Read what comes back, edit anything that’s wrong or too vague to act on, then:

Save that to memory and apply it going forward.

That’s the entire ceremony. Under two minutes, no tooling, no setup. (YMMV on the prompt, depending on your type and style of work…BWTM)

Here’s the part that changes everything. It doesn’t file the lesson. It applies it, every time, forever. Next session, and every session after that, those rules are loaded before the first word of work gets done. There is no folder, no onboarding deck, no hoping somebody remembers. The correction is simply in effect.

And it updates. Session ten refines what session three concluded. A rule that turned out to be wrong gets replaced instead of quietly ignored.

Same ceremony. Same format. The difference is entirely in what happens after the meeting ends.

The productivity gain is obvious enough. The part that sneaks up on you is the confidence. After a few weeks you stop bracing for the mistake you already corrected once, because it doesn’t come back.

The Gap Was Never Honesty

The lesson here isn’t that AI is disciplined and people are sloppy. People are perfectly capable of discipline. What people don’t have is a place to put a lesson where it will find them again at the moment they need it.

That’s the gap the retrospective has always had. Not insight. Distribution.

AI closes it almost by accident, because persistent memory is just how the thing works. The lesson arrives in the same channel as the work, automatically. You don’t have to remember to go check.

Which reframes a lot of process failures. Most of the time the problem isn’t that we didn’t learn. It’s that the learning had no route back into the work.

Almost Nobody Does This

Here’s the honest part. Very few people take the time to run a retrospective on their AI sessions, and fewer still do it consistently.

It isn’t hard and it isn’t slow, which is exactly what makes it easy to skip. Two minutes is nothing. Two minutes you never spend is also nothing, and it’s what most people land on by about the third busy week. The ceremony that depends on remembering to hold it is the ceremony that quietly stops happening, which is the same failure mode as the intranet page, just faster.

That’s the argument for automating the capture rather than relying on the habit.

You Don’t Have to Invent This

Run it by hand for a few weeks first, though. You want a feel for what a useful captured lesson looks like before you hand the job to something that writes them without asking, because you’ll be reviewing that output later and you need to be able to spot a bad one. Automated capture drifts. It records the frustrated aside instead of the actual rule, or it generalizes a one-off into a standing instruction. Doing it manually first is how you learn to recognize that when you see it.

And you don’t have to tune that prompt from scratch, or build the automation yourself either. There’s an entire cottage industry of shared instruction sets, skills, and rule files built to automate this loop, and most of them are free and MIT-licensed. Read a few and you’ll find better phrasing than mine sitting right there in someone’s repo.

claude-reflect is the most widely adopted of the ones I found. Hooks watch your prompts for correction patterns like “no, use X” or “actually” and queue them automatically. You then run a command to review the queue and approve what gets written into your persistent instructions. Capture is automatic, application requires your sign-off. It also mines your session history for repeated requests and offers to turn them into reusable commands.

bokan’s self-improvement skill takes a different angle: it fans out parallel agents across your past sessions, ranks friction patterns by how often they recur, and attaches the raw quote where things went sideways. Frequency plus evidence, which makes the finding arguable rather than just asserted.

claude-improve is worth reading even if you never install it, because its README openly credits the six community approaches it was assembled from. One of those, a one-prompt reflection pattern, contributes the single most practical idea in the whole space: write the captured lesson as an enforceable rule. Lead with why, use NEVER and ALWAYS, include a concrete example. Most captured lessons fail because they’re written as vague observations that nothing can act on.

Cursor users have the same pattern in a different wrapper, usually a self_improve.mdc rule paired with a rules file, as described in Stop Babysitting your AI.

Pick one, use it as-is, or read three of them and build the version that fits how you actually work. Standing on someone else’s structure beats staring at a blank file.

What the Evidence Says

One thing to calibrate before you go shopping, because I’d rather say it than have you find out later.

None of these tools has been independently reviewed or benchmarked. What they have is stars, forks, and enthusiastic author write-ups. A thousand stars measures whether an idea sounds good, not whether it works. There’s no head-to-head comparison, so pick on fit rather than on popularity.

The underlying mechanic, though, has real support. The Reflexion paper from NeurIPS 2023 found that agents that write a short natural-language reflection on what went wrong outperform agents that store the raw record of what happened by eight percentage points absolute. Storing the transcript is worth less than storing the conclusion. That is the academic version of Start, Stop, Continue.

Anthropic’s own context management benchmark reports a 39% performance improvement over baseline when memory and context editing are combined, and 84% fewer tokens consumed on a long multi-step task. Vendor-run, so discount it accordingly, but it’s a real evaluation with published numbers.

The mechanism is sound. The specific tooling is a matter of taste.

Prune It, or You’ve Just Rebuilt the Intranet Page

The one part none of this tooling can do for you: every system here accumulates rules, and none of them has a convincing answer for pruning. claude-reflect ships a dedupe command, which tells you the problem is real enough to need a command.

So budget a few minutes every month or so to read what’s piled up and delete what’s gone stale. That’s the intervention the manual practice trained you for. A memory file nobody audits becomes the same dead document, except this one is actively steering your work instead of quietly sitting on an intranet.

Making the Time Before You Save the Time

All of this runs into the oldest problem in efficiency work: you have to spend the time before you get the time. Every initiative that eventually saves hours starts by costing hours you didn’t have when you decided to do it. Automating your retrospectives is no different. The build is small, but it’s still a build, and it competes with the actual work that made you want the efficiency in the first place.

I get it. New habits are hard. I confess that I don’t do this as regularly as I should, and that even though I have some prompts pre-written in UpNote to use that are more involved than the example given in this post, I usually write one off the cuff when I do remember. Maybe if I spent less time writing about techniques and more time automating them…

Which is exactly why the two-prompt version matters. It costs nothing to start, it works on its own, and it buys you the judgment you’ll need before you spend an afternoon wiring up anything fancier. Start there. Automate when the habit is annoying you enough to be worth removing.

An Answer to the Question Everyone Keeps Asking

“What are we doing with the time we save using AI?”

Fair question, and most of the answers are vapor. Here’s one that isn’t.

Spend a slice of it teaching the tool what it just learned. Those two minutes are the highest-leverage part of the whole workflow, because they’re the only part that compounds. Every other minute you save gets saved once. This one gets saved again on every future session.

It shouldn’t take all of it. And it should take less as time goes on, which is the tell that it’s working. Early sessions produce a lot of corrections because there’s a lot the tool doesn’t know about how you work. Later sessions produce a few refinements. Eventually you’re mostly confirming that the rules still hold.

Use the rest to go touch grass.

That’s not a throwaway. The entire argument for this technology is that it gives us capacity back. If we spend all of the reclaimed time on more work, we didn’t gain capacity, we just raised the quota.

Now Tell Me What You’re Actually Doing

This is the part where a post like this usually promises a follow-up with results. I’d rather ask than promise.

If you’re already running some version of this, I want to hear it. Which tool, or which few lines you wrote yourself. What the capture prompt actually says. Whether you review what gets written or let it accumulate. What broke.

If you’re not doing it yet and this pushed you into trying, say so at the start and tell me what happened three weeks in. Including if the answer is that you did it twice and stopped, because that result is more useful than another enthusiastic write-up. The failure modes are what’s missing from every article I read while putting this together.

If enough of it turns out to be interesting, there’s probably a follow-up in it, built mostly out of what readers send rather than what I think. Credited unless you’d rather not be. A comment, an email, a reply on LinkedIn, whatever’s easiest.

The retrospective finally works. Let’s find out whether we actually run it.

If you found this interesting, please share.
Claude helping me move

Keep Claude from Forgetting You When Moving to a New PC

Scott S NelsonLeave a comment

TL;DR: Copying your project folders to a new laptop moves the files and leaves the work behind. Everything Claude Desktop actually remembers, your Cowork sessions, your MCP connector config, your Claude Code settings, lives in a Windows package sandbox that a normal search will never find. Zip that folder with the app closed and your history comes with you. Skip it and you start over with a very clean, very empty sidebar.

New Laptop Day

New laptop day is supposed to be a good day. Copy the projects, sign in, get back to work.

For most non-developers it generally is pretty straightforward, especially if your company manages backups and you use defaults. Sign in, wait for the sync icon to stop spinning, open the same three apps you always open. Whatever you’d customized was mostly customized inside something that follows your account, so it follows your account.

For developers and other power users it has never been that straightforward. There are a million little tweaks accumulated over years for personal preference and productivity, and not one of them is written down anywhere. The PATH entry you added at 11pm to make a build work. The tool installed from a zip file into a folder you invented. The config that lives three levels below AppData because that’s where the installer decided to put it. You don’t have a list. You have a vague sense that something will be missing, and you find out which something at the worst possible moment, usually about four days later.

Thanks to the popularity of Claude, now everyone gets those extra steps. And they’re not that straightforward even for the power users.

Here’s how it goes wrong. You copy the project folders. OneDrive handles the synced ones, a USB drive handles the local-only ones, file counts check out, you spot-check a few documents and nothing’s corrupted. Install Claude Desktop, sign in, open Cowork.

Empty.

The files are all there. The work is not. Every session, every bit of accumulated context, every conversation where you finally got Claude to understand your codebase, none of it came along. Because none of that lives in the project folder. It lives somewhere else entirely, and the app gives you no indication where.

Worth sorting out before you wipe the old machine, which is the point of no return most people hit about an hour after they think they’re done.

Two Kinds of Sessions, Only One of Which Is Your Problem

Some good news first. Cowork sessions now run remotely and are tied to your Claude account rather than your hardware. Sign into a new machine and those sessions are already there, waiting, automagically. Nothing to move, nothing to find. For once, the defaults crowd wins outright.

The catch is that this only holds for sessions created after remote sessions shipped. Anything older is a local artifact sitting on the old hard drive. If you’ve been using Cowork for a while, that’s likely most of your history.

So the question isn’t whether to migrate. It’s whether your sessions predate the cutover.

If none of them do, close the tab and enjoy the new laptop. Although before you go, it’s worth a moment’s honest reflection: if you’ve been at this for a while and have nothing accumulated that’s worth the trouble of moving, that’s telling you something. Not about the migration. About how much of Claude you’re actually using. A tool you’ve spent months teaching your projects, your standards, and your preferences to is a different tool than one you open to write emails. If none of this applies to you, the interesting question isn’t how to migrate. It’s what you’ve been leaving on the table.

For everyone else, keep reading.

Where It Actually Lives

Ask Claude where its own session data lives on Windows and it will tell you %APPDATA%\Claude\local-agent-mode-sessions\. It will tell you this confidently. Community migration tooling references the same path, so it isn’t inventing it from nothing.

It isn’t there.

Nor is it at %LOCALAPPDATA%\Claude\, which is the natural second guess and the next thing you’ll be told to try. You can run a recursive search across your entire user profile for audit.jsonl, the conversation log written inside every Cowork session, and get nothing back at all. That result is disorienting enough that you start to wonder whether the sessions ever existed on disk in the first place.

Whether that documented path was ever correct on Windows, or whether it’s a Mac convention that got generalized, isn’t something I can tell you. What matters is that on a current Windows install it’s a dead end, and it’s the dead end you’ll be pointed at first.

The real reason is architectural. Claude Desktop installs as a packaged Windows app, and packaged apps get their own private storage sandbox. Writes to the conventional locations get quietly redirected somewhere like this:

C:\Users\<username>\AppData\Local\Packages\Claude_<packageid>\LocalCache\Roaming\Claude\

Note the shape of that path. It ends in Roaming\Claude, exactly as documented. It’s just sitting under a package container that no reasonable person would think to check, which is why searching for the documented path fails while the documented path is, in a sense, still accurate.

The package ID is a short string of characters, and there’s no reason to guess at yours. Find it:

Get-ChildItem -Path $env:APPDATA, $env:LOCALAPPDATA, $env:USERPROFILE -Recurse -Filter "*claude*" -ErrorAction SilentlyContinue |
  Where-Object { $_.PSIsContainer } |
  Select-Object FullName |
  Out-File "$env:USERPROFILE\Desktop\claude-folders.txt"

That writes every Claude-related folder on the machine to your desktop. Open the file, look for the one containing local-agent-mode-sessions, and you have your real path. Two notes: run it in PowerShell, not Command Prompt, and expect it to take a minute or two while it walks the profile.

Depending on how and when Claude was installed, your data may genuinely sit in the plain %APPDATA%\Claude\ path. Both are possible. The one holding session subfolders with long UUID-style names is the one that matters. Don’t assume, check.

Claude Will Help You Move, and You Don’t Even Need to Buy It a Pizza Afterwards

Here’s the part that’s mildly funny in hindsight: the way to find all of this is to ask Claude, in Cowork, on the machine you’re migrating away from. It has shell access to the profile it’s running in. Point it at the problem and it will run the searches, read what comes back, and narrow the path down with you.

It will also send you to those two wrong folders on the way there, because the documented path and the actual path aren’t the same thing, and the documented path is what it reaches for first. That’s not a knock. It’s the ordinary pattern of working with these tools, and the fix is the same as always: make it show you the output rather than accept the summary. The searches above came out of exactly that back and forth. Claude proposed, the filesystem disagreed, and the third attempt found it.

Which is the useful lesson here, more than the folder path. The tool is genuinely good at the mechanical part, walking a filesystem, reading a directory listing, writing the PowerShell you’d otherwise be looking up. It is not reliable about where things live, because that’s a fact about your specific install and not something it can know from training. Use it for the legwork. Verify the destination yourself.

What to Copy and What to Leave

Open that Claude folder and you’ll find far more than sessions. Most of it is cache, crash logs, GPU state, and machine-specific scratch that regenerates on its own. Copying it doesn’t help and can actively cause conflicts.

Bring these:

  • local-agent-mode-sessions is the whole reason you’re here. Your Cowork session history.
  • claude_desktop_config.json holds your MCP server configuration. Lose it and every connector gets set up again from scratch. This is the one people don’t realize they needed until it’s gone.
  • claude-code and claude-code-vm carry Claude Code settings and VM data, if you use it.
  • vm_bundles for Claude Code VM bundles.
  • ChromeNativeHost for the Claude in Chrome integration.
  • config.json and cowork-enabled-cli-ops.json for general app and Cowork settings.
  • git-worktrees.json if you run Claude Code against git worktrees.
  • pending-uploads if anything is sitting in it.

Leave IndexedDB alone. It stores app state tied to your current signed-in session, and the new machine builds its own. Overwriting it invites problems you’ll spend an evening diagnosing.

Also grab C:\Users\<username>\.claude\ on the old machine. That’s a separate folder outside the sandbox, holding configuration and skills, and it goes to the same place on the new profile.

Zip It on the Old Machine

Rather than dragging ten items across a USB drive and hoping you got them all, package them once. Quit Claude Desktop completely first. Not the window, the app. Right-click the tray icon and choose Exit. Zipping live data files is how you end up with a corrupted history and no idea why.

Then, in PowerShell, substitute your own package ID into the first line and run this:

$Claude = "$env:LOCALAPPDATA\Packages\Claude_<packageid>\LocalCache\Roaming\Claude"

$Items = @(
  "local-agent-mode-sessions",
  "claude-code",
  "claude-code-vm",
  "vm_bundles",
  "ChromeNativeHost",
  "pending-uploads",
  "claude_desktop_config.json",
  "config.json",
  "cowork-enabled-cli-ops.json",
  "git-worktrees.json"
) | ForEach-Object { Join-Path $Claude $_ } | Where-Object { Test-Path $_ }

Compress-Archive -Path $Items -DestinationPath "$env:USERPROFILE\Desktop\claude-migration.zip" -CompressionLevel Optimal

The Where-Object { Test-Path $_ } line quietly skips anything you don’t have, so you can run it as-is whether or not you use Claude Code.

One detail that makes the other end easy: the archive stores paths relative to the items you named, not their full absolute paths. local-agent-mode-sessions\ and config.json land at the root of the zip. The archive is effectively a snapshot of the Claude folder itself, which means unpacking it is a single step with nothing to rearrange.

Then package the profile folder separately, since it lives elsewhere:

Compress-Archive -Path "$env:USERPROFILE\.claude" -DestinationPath "$env:USERPROFILE\Desktop\claude-dotfolder.zip"

Two zips on your desktop. Move them however you like: USB drive, OneDrive, network share.

If Compress-Archive throws a path-length error, that’s the 260-character Windows limit biting, and it’s plausible here given how deep the sandbox path already runs before your session UUIDs even start. Enable long paths in Windows, or fall back to robocopy with the /E switch to stage the folders somewhere shallow like C:\ClaudeMigration\ first, then zip from there.

Unzip It on the New Machine

  1. Install Claude Desktop, sign in, and open Cowork once. This creates the directory structure you’re about to unpack into. Then quit completely again.
  2. Find the package path on this machine. Run the same Get-ChildItem search from earlier. The package ID can differ between installs, so confirm rather than assume.
  3. Unpack straight into the Claude folder. Because the archive is relative to that folder, everything lands where it belongs on its own:
$Target = "$env:LOCALAPPDATA\Packages\Claude_<packageid>\LocalCache\Roaming\Claude"

Expand-Archive -Path "$env:USERPROFILE\Desktop\claude-migration.zip" `
               -DestinationPath $Target -Force

-Force overwrites files that already exist and leaves everything else untouched. The only collisions are the config files the fresh install generated a minute ago, and replacing those with yours is the entire point.

  1. Restore the profile folder:
Expand-Archive -Path "$env:USERPROFILE\Desktop\claude-dotfolder.zip" `
               -DestinationPath "$env:USERPROFILE" -Force
  1. Restart Claude Desktop. Your old sessions should appear in the sidebar.
  2. Reconnect your workspace folders. These are machine-specific paths and won’t follow you. Copy your project files first, then point Cowork at their new locations.
  3. Reinstall plugins that rely on local MCP servers. Cloud-only plugins come across with your account.
  4. Keep the old machine intact until you’ve verified all of it. Open a few migrated sessions, confirm your connectors work, and only then wipe.

The Same-Account Assumption

Everything above assumes one thing that’s easy to overlook: the Windows username is identical on both machines. When it is, every path inside those session files still resolves and the unpack just works.

When it isn’t, you have a problem the zip won’t solve. Sessions store absolute local paths, so C:\Users\OldName\Documents\... follows you onto a machine where no such user exists. The files arrive fine. The references inside them point at nothing.

If your usernames differ, this is the one case that needs an extra stop. Unpack to a scratch folder instead of straight into the target, run a find-and-replace across the JSON and JSONL files inside the extracted sessions to swap the old profile path for the new one, verify a single session opens correctly, and only then copy the corrected folders into place. Apply the same scrutiny if anything else moved, a different drive letter, a relocated Documents folder, OneDrive redirection that was on before and off now. Any of those breaks the same assumption in the same way.

Setting expectations honestly here: migrated sessions may land as readable history rather than fully resumable conversations. You get the record of what was discussed and decided, which is the part that took months to accumulate. Whether Cowork lets you pick up where you left off is a separate question, and one worth testing with a single session before you count on it for all of them.

With AI, Every How-To Has an Unknown Expiration Date

Everything above is a workaround for a design that’s already being replaced. That much is normal. What’s different now is that nobody can tell you when the replacement lands.

Software used to telegraph its changes. Version numbers, release notes, deprecation warnings, a beta period where the community wrote up what moved. A how-to written against version 4 stayed true until version 5, and version 5 announced itself. You could read a three-year-old post, check the version in the header, and know within seconds whether it still applied.

AI tools don’t work that way. They ship continuously, the client updates itself, and the thing you’re actually interacting with changes underneath a version number that may not move at all. Storage locations migrate. Local features become account features. The answer that was correct on Tuesday is wrong on Thursday, and nothing announces it. There’s no header to check.

Which is exactly what’s happening to this article. With sessions running remotely and tied to your account, the machine stops being where the work lives and becomes just a window onto it. Sign in anywhere and your history, your projects, and your scheduled tasks are already present. No folder to find, no zip to move, no package sandbox to go spelunking in. For anyone starting fresh today, this is already unnecessary.

That’s the right direction, and worth saying plainly rather than grumbling about the transition. Local-first storage bought you privacy and offline access at the cost of making your work a hostage to one piece of hardware. Account-first storage trades that the other way. Reasonable people weigh those differently, but nobody has ever been glad their context was trapped on a laptop with a failing battery.

The catch is the seam, and seams are always where the work is. Tools in transition leave a cohort stranded on the old model, holding data in a format the new model doesn’t automatically reach for. That cohort is anyone who adopted early, which is to say the people with the most accumulated context and the most to lose. Migration guides exist for the gap between what a tool used to be and what it’s becoming.

So treat this one, and every AI how-to you find, as perishable goods with no date stamped on the carton. Verify the path before you trust the instructions. If the folder isn’t where the article says it is, the article is probably older than the software. That will be true of this post eventually, and I can’t tell you when. Neither can anyone else. Until then, somebody has to move the boxes.

What This Actually Costs You

A little perspective on why any of us write these things down.

My first how-to post, back in 2002, was about building web services with Apache Axis. Working it out took the better part of two days: documentation that assumed you already knew the answer, examples that didn’t compile, and a long stretch of staring at a stack trace. The post that came out of it took a reader about twenty minutes. Two days of my confusion, compressed into twenty minutes of somebody else’s afternoon.

This one took about an hour of prompting and probing with Claude. It should take you fifteen minutes. Ten, if you run this post through an AI and ask it to strip out all my digressions, which I’d encourage, though I’d like it noted that the digressions are the part I enjoy.

The ratio held. It’s just faster on both ends now.

Fifteen minutes, then, against rebuilding context that took months to accumulate. That’s the entire calculation, and it’s the same calculation behind every unglamorous piece of groundwork in this business. Nobody gets excited about verifying a folder path. Everybody gets excited about the new laptop. The gap between those two feelings is where work disappears.

The broader lesson has nothing to do with Claude. Any tool you’ve spent real time training to your context is holding state somewhere, and “somewhere” is rarely where the marketing implies. Before you decommission a machine, ask what the tools on it know that the files don’t. Then go find out where they keep it.

Then wipe the old laptop. Not before.

If you found this interesting, please share.

Trust, But Verify: A Five-Stage Pattern for Working With Claude Code to Build Basic Modern Web Applications

Scott S NelsonLeave a comment

TL;DR: Claude’s default failure mode isn’t refusal, it’s confident, plausible-sounding claims that time-pressured developers don’t always check. Fixing that takes two moves: reset how eager Claude is to agree with you, then build a process that consistently performs independent verification and validation. Skip the first and the second one can create as much work as it saves.

The Failure Mode Isn’t What You Think

Everybody braces for Claude to refuse the task, hallucinate something wild and obviously wrong, or determinedly rewrite two-thirds of the codebase to implement a simple task that only requires eight lines of code. That’s not the failure mode that actually costs you time. The real one is quieter: Claude tells you something sounds right, in a tone that sounds checked, and it hasn’t been checked at all. Confident and wrong is a much more expensive combination than obviously broken. It gets worse under deadline pressure. When the sprint’s closing and the diff looks clean, verifying every claim feels like the one step you can afford to skip, right up until it’s the one step that would have caught the problem.

Anthropic says as much in its own documentation. Project instructions get “delivered as a user message after the system prompt… Claude reads it and tries to follow it, but there’s no guarantee of strict compliance” (How Claude remembers your project). That caveat runs both directions. It applies to what Claude is told to do, and just as much to what Claude reports about the work it already did.

Anyone who has done more than casual coding with Claude has experienced this pattern, even if it is either not recognized as such or treated as “that’s just the way it is.” A review of a recent project turned up incorrect references in CLAUDE.md that caused missed steps, leading to rejected PRs on good days and failed sprint demos on the not-so-good (not to mention the resulting rework).

For those who’ve read enough of these posts, you already know where this is going: behavior over tech. Yes, learning the nuances of continuously evolving AIs will help you be more effective and efficient with them. But it’s process discipline when using the tools that protects you from the pitfalls of an interface that has a “personality” faked through JavaScript algorithms and ethics centered around subscription renewal. That discipline only works if you fix the collaborator’s temperament before you fix the process around it, because a process built on top of an agreeable Claude just gets agreed with faster.

Reduce the Sycophancy Before You Reduce Anything Else

Here’s why temperament has to come first. Anthropic defines the problem plainly: sycophancy means “telling someone what they want to hear… rather than what’s really true, or what they would really benefit from hearing.” It “often manifests as flattery,” and sycophantic models “tend to abandon correct positions under pressure” (Protecting the wellbeing of our users). Claude’s default training leans toward exactly that. Stack it on top of confident-but-unverified claims and you get a collaborator that validates your plan instead of stress-testing it, right when stress-testing is the entire point of asking. Any process you build on top of that inherits the same flaw, because a rubber stamp is still a rubber stamp whether it happens once or five times in a row. Worse, that rubber stamp isn’t free. Five stages of going through the motions still cost five stages of your attention, prompts written, transcripts read, boxes checked, and none of it catches anything a five-minute skim wouldn’t have. That’s the second move creating as much work as it saves.

The fix is cheap (and easy!), and that’s the part people don’t expect. Independent write-ups on this converge on the same finding, a handful of lines in a global config file measurably change the behavior. In one documented before/after test, unmodified Claude hedged on storing API keys in frontend JavaScript (“that’s one approach… you might want to consider some security aspects”). Same model, same prompt, a few added instructions, and it flagged the same setup outright as a critical security flaw (How to Make Claude Stop Agreeing With Everything). This isn’t project-specific. Set it once, globally, in ~/.claude/CLAUDE.md (%USERPROFILE%\.claude for Windows users), the user-level file Anthropic documents as applying across all projects, as opposed to a project’s own CLAUDE.md, which stays scoped and version-controlled for the team.

If you already run personal instructions telling Claude not to flatter you and to push back when you’re wrong, you’ve got a head start. Here’s the block to layer on top of that, not underneath it:

# Our Working Relationship

- Don't validate an idea just because I proposed it. If my approach has a flaw, stop and flag it before proceeding, not after you've finished.
- I am sometimes wrong. Challenge my assumptions directly, and name what you checked to back up the challenge.
- Don't open with agreement ("You're right," "Great idea") or disagreement as a ritual. Build on the idea or move forward. Agreement is shown through action, not an announcement.
- Be matter-of-fact, not deferential or hedging. Challenge ideas, not people. Stay collaborative, not combative.
- Don't disagree for its own sake. Disagree when the evidence points that way, and say so plainly when I'm right too.
- If you don't know something, say "I don't know." Don't fill the gap with a confident-sounding guess.
- Be concise. Skip long-winded caveats and softening language.

The above is an example not to be adopted verbatim. It’s important to adapt it to your work style. There are other examples and references out there:

 

Test it before you trust it. Feed it a prompt with a deliberate architecture or security flaw and see if Claude catches it unprompted. And don’t overcorrect. Multiple sources warn that “be brutal” or “critique everything I say” produces combative, unhelpful output, not sharper output (How to Stop Claude From Being a Yes-Man (Get Real Pushback)). You want a collaborator with a spine, not an adversary you now have to manage. Again, keep in mind any external reference is only a starting point adapted from other people’s writing. Weigh it against your own results before you treat it as settled.

That fixes the temperament. It doesn’t fix what happens when a less-agreeable Claude still needs to catch a wrong assumption buried in a 400-line diff, or independently critique a plan without inheriting the blind spots of whoever wrote it. Temperament tells Claude how to disagree. It doesn’t tell you when to make it disagree. That’s what the five-stage pattern is for.

The Five Stages, and Why the Order Isn’t Arbitrary

Here’s the structure that puts a disagreement checkpoint at every point in a story where a confident wrong claim could otherwise slip through.

Plan. Hand Claude the story, ticket text, acceptance criteria, relevant screenshots, and point it at the right part of the repo. Use Plan Mode instead of letting it jump straight to code. What comes back should be a written plan: files and components to touch, explicit out-of-scope items, and every assumption about the codebase stated as verified, not guessed. Push back on any claim stated as fact without a tool call behind it. “Did you check that, or assume it from a similar component?”

Critique. This one has to come from somewhere else entirely: a second, unprimed Claude, a subagent or a fresh conversation, asked to critique the plan rather than the thread that wrote it. What you’re owed back is a list of concrete problems, or an explicit “no issues found” backed by what was actually checked. Never accept “looks good” without evidence. “What’s the strongest argument this plan is wrong?”

Implement. Read the actual diff line by line (see Noteworthy, below). Don’t just trust that it matches the plan. Claude owes you the code changes plus an explicit callout of anywhere the implementation deviated from the plan and why. If it deviates without flagging that, stop and ask why. Spot-check it: “you said you added a test for X, show me the file.”

Mechanical Check. Somebody has to decide whether a manual override, a disabled lint rule, an as any, a waived a11y rule, is legitimate or a shortcut, and that’s you. Claude’s job is real command output for every gate already listed in CLAUDE.md, not a paraphrased “should pass.” Never accept “this should pass” without the output. If Claude proposes suppressing a check instead of fixing it, ask why it can’t be fixed properly.

Acceptance Check. Final sign-off stays with a human, since it needs business and design intent Claude doesn’t have. Claude’s evidence is input here, not the verdict. What you need back, for each acceptance-criteria line, is a specific pointer to where it’s satisfied, not a blanket “all done.” Go line by line through the criteria. “What isn’t covered?”, not just “is it done?”

Look at that sequence again and it isn’t arbitrary. The first three stages front-load verification before and during coding, because catching a wrong assumption at Plan or Critique costs a sentence of pushback. Catching that same wrong assumption after merge costs a revert, or a wrong instruction sitting in CLAUDE.md for weeks before anyone notices. That’s the ROI case in one line: a sentence now, or a revert later.

Critique only works as an independent pass, not a second look from the same context, which is exactly why it comes right after Plan and not later. A plan reviewed by the mind that wrote it inherits that mind’s blind spots, no exceptions. That’s the whole point of handing it to an unprimed Claude: it approaches the plan with a beginner’s mind, no investment in defending what it already wrote, nothing to protect. I’ve watched teams skip that and call it Critique anyway, rerun the same conversation, get the same agreeable nod, file it as reviewed. Somewhere there’s a sprint board with a checkbox for “AI code review” that has never once been unchecked.

Mechanical Check comes after Implement for a reason too: it’s the one stage that shouldn’t require judgment at all. It’s just the codification of CLAUDE.md‘s existing mandatory-checks section, run every time. But it’s only as trustworthy as that section is accurate, and an out-of-date gate list is worse than no gate list, because it creates false confidence that enforcement exists when it doesn’t. Every override or new check has to get written back into CLAUDE.md the same day it happens, or the drift starts immediately.

Acceptance Check is last and stays human-owned on purpose. Passing every mechanical gate proves the code is well-formed. It proves nothing about whether the right feature got built, and that judgment call needs business and design context Claude doesn’t have full visibility into, no matter how well its temperament is tuned or how many stages came before it.

The first four stages verify the change, proving it was built right. Acceptance Check validates it, proving it’s the right thing to have built at all. Skip either half and it still shows up in production, no matter how clean everything upstream looked.

The Bottom Line

Claude states things confidently by default. Per Anthropic’s own guidance, there’s no guarantee of strict compliance even with explicit instructions. Behavior over tech, one more time: temperament and process aren’t features Anthropic ships. They’re the parts you’re responsible for, whether it’s a five-line function or a whole story, demand evidence over assertions, and make Claude argue against its own work at least once before calling it done.

Speaking of done, AI is a long way from stabilizing. Use articles like this as fuel for thought processing rather than a one-and-done punch list. Keep reading, keep thinking, and share what works.

Noteworthy

One more tool worth bolting onto this: OwnDiff turns the “read the diff line by line” instruction from the Implement stage into something that can’t be skipped. It’s a local human-review gate. It scores the current git diff for risk, and for anything medium risk or above it has the coding agent generate diff-grounded multiple-choice questions, then refuses to let the agent push or open a pull request until a human answers every one correctly. No web search, no outside facts, no generic filler questions, everything grounded in the actual changed files.

That’s the same idea as Acceptance Check, just enforced mechanically instead of by habit. “I read the diff” is a claim. Passing a quiz generated from that diff, with the push blocked until you do, is evidence. Worth a look for the days you’re moving too fast to hold yourself to the honor system (Thanks to Ed Lyons for pointing me at this one.)

If you found this interesting, please share.
Robots looking for data with varying degrees of success

AI Treats Your Documentation as Data. You Should Too.

Scott S Nelson1 Comment

TL;DR: Enterprise AI runs on enterprise data, and that includes the unstructured data in the form of documentation that is poorly curated. No taxonomy, revisions sitting next to finals with no way to tell which one is current or still relevant. If data is the DNA of modern business, documentation is the dominant chromosome that can make the difference between robust health and questionable viability.

I post a lot about the value of architecture and training in AI adoption, and share posts by people focused on the nuances of prompting because these things will make a difference. Data cleanliness is a topic I usually just point people toward other people’s thinking on because my days focus more on how data moves and evolves than on structuring and managing it. As a long-time generalist, I have a deep appreciation of the added value specialists bring to the process. Documentation is a different kind of data, with its own unique headaches. I’ve harped on some of those issues for years: why self-documenting code doesn’t produce a self-documenting solution (From Agile to Fragile in 60 Sprints), why nothing gets read that never gets written down (If It Is Not Written Down It Does Not Exist), and why a taxonomy nobody maintains is worse than no taxonomy at all (Failure to plan communications is communicating a plan to miscommunicate, one that I wrote at the dawn of the current age of AI without realizing how it would soon become even more important).

Then it occurred to me that documentation and data cleanliness are the same conversation, and I hadn’t heard many people say so directly. Documentation is data, and not just any part of it. It’s the chromosome that expresses the rest of your enterprise data, the one that decides whether everything built on top of it turns out healthy. It’s also usually the part in the worst shape. That’s the case I want to make here. (Confession: the connection came from something I read that mentioned it in passing, and I never noted the source. Whoever you are, kudos.)

AI needs three things from your data, in order:

  1. Access.
  2. Understanding.
  3. The ability to apply what it understands to the specific context of whatever someone just asked it.

Skip a step and the most expertly crafted prompt or the best planned architecture in the world still only gets you an expensive autocomplete with a confident and flattering tone, because both of them are working with whatever access and understanding they’ve been handed.

There’s a lot written about the mechanics of step one: RAG, vector stores, and running SharePoint exports through Pandoc before they ever reach a prompt, since raw Word and PDF files carry a lot of baggage a model doesn’t need to see. Step two has gotten a lot less attention. Understanding requires that all of the documentation your AI is referencing provides meaning and consistent messaging, and the usual disarray of enterprise documentation doesn’t.

AI is a (New) Good Reason to Clean Up Your Documentation

This was already a problem in 2022, back when the audience for the complaint was a project manager, not a language model. Templates that auto-update their “last modified” date every time the file gets saved, whether or not the content actually changed, so the date stops meaning anything (Replace Auto Dates in Templates). It gets worse at the platform level. SharePoint and Teams will happily stamp a document “Modified” the moment someone opens it, whether they changed a single character or not, because the file was never stored with the “open as read-only” flag set. Nobody sets it, because almost nobody thinks about it, and now your most trustworthy-looking piece of metadata is lying to you and to anything reading it downstream.

Then there’s version chaos stacked on top of date chaos. A SharePoint draft with one clear owner on paper, quietly edited by people who didn’t know that, discovered only after someone had to re-verify the entire document line by line (Recovering Previous Versions from SharePoint). That gripe about config files requiring “reading documentation, which is only read less than it is written” is sixteen years old (Dynamic Log Location for log4j). That was a joke about developers skipping the manual. It reads differently now that the thing skipping the manual is a model that can’t lean over and ask a coworker what the doc meant to say.

None of that is a new problem. What’s new is that AI has zero tolerance for it. A human can walk into a shared drive, eyeball three files named some variation of “Process_Final_v2_ACTUAL_FINAL,” and guess correctly which one to trust, because they have context: they remember the meeting, they know who owns the process, they can just ask. AI doesn’t get that fallback. It reads what’s in front of it and treats every file as equally authoritative, including the wrong one.

If This Sounds Familiar…

Turns out I’m not the only one who noticed. Other people are seeing the same problem from angles most enterprise practitioners rarely get access to. Amit Shivpuja, who runs data and AI enablement at Walmart, wrote in Forbes that he watched a fully governed, well-modeled AI program produce inconsistent results anyway, and traced it to what he calls the missing documentation layer: the context that should have been captured during requirements, design, and testing, but instead lived in a Slack thread or in someone’s head (The Hidden Barrier To Enterprise AI: The Missing Documentation Layer). His diagnosis lines up with a decade of watching the same pattern play out: humans compensate for missing documentation with tribal knowledge. AI can’t.

The gap between AI investment and AI payoff backs this up at scale. 79% of organizations report real challenges getting AI to deliver, a double-digit jump from the year before, even as most are raising budgets to feed it (WRITER). Separately, only 32% of organizations report sustained business impact from AI despite 86% of the C-suite increasing investment, according to an Accenture survey (Forbes). Nobody breaks that spend down into prompt engineering versus architecture versus training, but odds are good most of it lands in exactly those three buckets. It’s not usually the prompts, the architecture, or the training that’s failing. It’s the context underneath all three.

How to Get Started

None of that makes training, architecture, or prompt engineering optional. They’re not, and treating them as afterthoughts would be its own kind of mistake. Training your people is worth the time and the awkward learning curve that comes with it, every time. Architecture done right is what lets a foundation hold up for years while the technology running on top of it changes every few weeks, so it’s worth building solid instead of patching forever. And prompt engineering isn’t dead, whatever this month’s headlines are claiming. Knowing how to ask well is still what gets the new, flashier capabilities to actually do what you meant instead of what you typed.

But the potential of all that value depends on how well it works with your enterprise data. We’re past the point of needing to prove AI will benefit the business. We’re all now in the throes of how it will benefit the business, and that how stays severely limited until you put your data in order. Here are a few tips to get you started.

Make documentation part of done, not an afterthought to done. Shivpuja’s Forbes piece gets this right: a story or a feature isn’t finished until the meaning, the rules, and the assumptions behind it are captured somewhere a model can find them. That’s a process change, not a tooling purchase.

Default to read-only. Force intent for edits. If a document is finished, save it that way, and make someone actively choose to reopen it for editing. This has been true since 2022, and the fix hasn’t gotten any harder to implement. Just more expensive to keep skipping.

Give it a taxonomy and a living Read Me, and actually maintain both. This isn’t new advice. Generative AI just raised the stakes on it: organize things so someone new can find their way around without a tour guide, pin an explanation of that structure somewhere obvious, and keep it current as the team and the work change (Failure to plan communications). A taxonomy that was accurate in 2022 and hasn’t been touched since is worse than no taxonomy at all, because it still looks trustworthy.

Attach context to the asset, not to a folder that might get reorganized next quarter. Documentation living next to the process or dataset it describes survives longer than documentation living in a wiki page someone has to remember exists.

Convert for the machine, not just for the human. Pandoc, or whatever your equivalent is, exists to clear that baggage out before it hits a vector store. Word and PDF files are full of formatting decisions that made sense to a human editor and mean nothing to an LLM. Fifteen minutes of conversion saves a RAG pipeline from tripping over someone’s decade-old formatting habits. This is architecture work too, for what it’s worth, just the boring kind that doesn’t show up in a vendor pitch.

Set a review cadence, and mean it. One knowledge-management vendor’s own market analysis cites a 2025 Gartner study putting the number at 60% of internal knowledge articles going stale within six months, with only 14% of teams auditing content on any real schedule (source, with a grain of salt: it’s a page selling a fix for the exact problem it’s describing, but the shape of the number matches what shows up across most enterprise environments). A tool that flags stale content is nice. A team that actually looks at the flag on a schedule is the part that works. Buying a self-updating knowledge base without building the habit of using it just moves the swamp to a nicer-looking pond.

If Data Debt were a Thing…

There’s a term for this that architects already understand: technical debt. Every shortcut taken to ship faster accrues interest, and the bill always comes due, usually at the worst possible time and for more than the original shortcut would have cost to do right.

Documentation disarray runs on the same math. Call it data debt. It accrues every time a taxonomy goes unmaintained, a Read Me goes stale, or a “final” draft ships without anyone reconciling it against the other four drafts sitting next to it. None of that shows up on a balance sheet, so nobody budgets against it. But your prompts, your architecture, and your training program are all paying interest on it anyway, every time they inherit whatever your documentation can actually support.

Pay it down early and it stays cheap: a taxonomy tightened now, a template’s date field fixed before it propagates through a hundred more copies. Let it ride, and the interest compounds. More conflicting drafts pile up. More tribal knowledge walks out the door with the people who had it. More AI output gets built on documentation that was already lying to you, and the eventual fix means untangling years of it instead of an afternoon of it. Ignore it long enough and it doesn’t just get expensive, it bankrupts the whole initiative: the AI program that never delivered, the budget that got pulled, the “we tried AI and it didn’t work” verdict that was actually a documentation problem wearing an AI costume.

Fix the documentation first. It’s not sexy, nobody gets interviewed on a podcast for cleaning up a taxonomy, and it’s exactly the debt payment that keeps the rest of the investment solvent.

If you found this interesting, please share.