The Autopilot Trap: Why Your AI Dependence Is Quietly Eroding Your Judgment
A 2023 BCG study found consultants using AI for tasks beyond AI's capability performed worse than those without AI—their judgment had atrophied.
Researchers draw a parallel between aviation's automation complacency problem and knowledge workers' growing reliance on generative AI. A 2023 BCG study revealed that while AI boosted performance on tasks within its capabilities, it degraded performance on tasks beyond them—because workers had lost the ability to recognize errors. The author argues for "hand-flying policies": scheduled periods of unaided work to preserve critical thinking skills that matter most when AI fails.
In 1997, aviation researchers Raja Parasuraman and Victor Riley published a paper that named a pattern the industry had been living with for years: automation complacency. Pilots who let the autopilot handle everything gradually lost the ability to notice when the system was failing—and the ability to recover manually when it did. The more reliable the automation, the deeper the skill decay. They didn't solve this by removing automation. They solved it by mandating deliberate, uncomfortable periods of flying without it. The FAA now requires pilots to log regular hand-flying hours—not because manual flying produces better flights, but because it preserves the diagnostic and corrective skills that matter when automation hits its limits. Knowledge workers are walking into the same trap with generative AI. The 2023 BCG field experiment by Fabrizio Dell'Acqua and colleagues found that consultants using GPT-4 for tasks inside the AI's capability frontier performed better, but those who used it for tasks outside that frontier performed worse than the control group. They had offloaded judgment to a tool that wasn't up to the job, and their own skill had atrophied enough that they couldn't recognize the errors. Cognitive psychologists call a related phenomenon the generation effect: information you produce yourself sticks far better than information you merely read. When you skip the first draft, you skip the encoding. The aviation fix translates directly. You need a personal hand-flying policy: scheduled blocks where you produce without AI assistance, not because the output will be better, but because the struggle itself keeps your diagnostic faculties calibrated. Pick a recurring task—a strategy memo, a code review, a research synthesis. On Monday morning, block 45 minutes. Close the chat window. Write your own first draft from scratch. Notice where you hesitate, where you reach for a phrase and can't find it, where you're unsure of a fact. Those friction points are exactly where your skill is decaying. Fly manually, not to replace the autopilot, but to stay ready for the moment it can't fly for you.