I was recently working on a Spring Boot codebase I know well. I asked my favorite coding agent to make what should have been a simple change. The code compiled. At first glance, everything looked fine. But as I read through it more carefully, I found a few problems. It pulled in dependencies it didn't need. It used a mutable type where an immutable one belonged. And the test it wrote technically passed, but didn't actually test the thing I asked for.
I fixed it, moved on, and thought the usual thought: "these tools still have a long way to go."
Later that same day, I was working on my first iOS app. I've been building it on the side to learn, and I lean on the assistant for almost everything. I asked it to help me with a feature, and the code it gave me looked amazing. Clean. Idiomatic. I dropped it in, it worked, and I closed the tab feeling like I'd just been handed a gift.
Here's the thing that's been bothering me ever since. I have no idea if that code was actually any good. I don't know the best practices in the iOS world. I don't know what a senior Swift developer would flag on sight. I just know it ran. And that's a very different thing from knowing it's right.
Two codebases, same assistant, completely opposite reactions from me. And the reaction I should trust less is the one that felt better.
A name for the feeling
There's a term for this, and it comes from journalism, not software. It's called Gell-Mann amnesia, coined by the writer Michael Crichton. The idea goes like this. You open the newspaper and read an article about a subject you know well. You notice the reporter got half of it wrong. Names misspelled, causation backwards, nuance gone. You roll your eyes.
Then you turn the page and read the next article, about something you don't know much about, and you just believe it. You forget that the same paper, written by the same kind of reporters, just botched the thing you're an expert in. You give the next article a trust it hasn't earned.
Swap "reporter" for "AI agent" and you have exactly what happens in our editors every day.
Why this matters
You might think of this as a harmless byproduct of using new tools. It's not. When you can't tell good code from bad in a given ecosystem, you ship things you don't understand. You build on patterns that'll break. You learn the wrong lessons from a tool you trusted too much. And the longer it goes on, the harder it is to untangle.
When AI seems like magic in a language or framework you don't know, what you're really seeing is the limit of your own ability to critique it. Your coding agent of choice is backed by an LLM, and LLMs have real limitations. The most glaring one is that they're trained on a snapshot of data with a cutoff date. That means your agent might be hallucinating APIs that don't exist. It might be using a pattern the community abandoned years ago. It might be writing code that works on your local machine today but collapses under any real load. You don't know, because you don't know what good looks like in that world yet.
And here's the uncomfortable part. Even in the code you know well, your confidence is bounded by your own blind spots. I caught three problems in that Spring Boot change. I have no idea if there were more. The difference isn't that experienced developers catch everything. It's that they know where to look.
The inverse is also true, and it's actually good news. The reason you catch mistakes in your own codebase is that you've built up the judgment to see them. That same judgment is what lets experienced developers work around the LLM's limitations in the first place. They know to give it the right context through MCP servers, skills, documentation, and examples from the actual codebase. They know when to trust it and when to verify. That judgment is the whole game. It's what separates developers who get real leverage from AI tools from developers who just generate more code they'll have to debug later.
What to do about it
A few things I've been trying, and recommending to people I work with.
Trust what you know, verify what you don't. When you're using AI in a language, framework, or domain you don't know well, assume the output is wrong until you've verified it. Run it. Read the docs. Ask it to explain its own choices and then sanity-check the explanation. The confidence you feel is not evidence of correctness. It's evidence of your own blind spot.
Use AI to learn, not just to ship. If you're learning a new language, don't just accept the code. Ask why. Ask what the alternatives are. Ask what a senior developer in that ecosystem would flag. You're using the tool to build the fundamentals that will eventually let you critique it, which is the whole point.
Master the fundamentals of your craft. The critical eye you bring to your own codebase is the single most valuable thing you have right now. Naming, refactoring, reading code, understanding systems. These are the skills that turn AI output into shipped software instead of technical debt. They matter more now, not less.
The future of this job isn't about who can type the fastest prompt. It's about who can tell when the answer is wrong. That's a skill you build one fundamental at a time.