We Keep Asking AI to Make the Decisions We Shouldn't Delegate
Three hikers got rescued off Mount Shasta on Monday after spending a night wedged in a steep canyon gully. Here is the detail every story is burying: they did not make some wild, reckless gamble. They asked Google's Gemini AI to plan the trip and what to pack, and then they followed it. That is the part that should genuinely worry you, because it is the same thing you and I do a hundred times a day with higher stakes and no sheriff to call.
The men, novice hikers from Roseville, California, wanted to summit Shasta, the fifth-highest peak in the state. It was to be a manageable eight-hour ascent, the kind Gemini is happy to lay out as a clean, numbered itinerary. They camped at 8,400 feet on Saturday, left with daypacks just after 3 a.m. Sunday, and let the plan carry them.
Here is where the confidence collapsed into a canyon. Gemini had advised them to bring far less food and water than the group actually needed, per the Siskiyou County Sheriff's Office. They reached the summit around 7 p.m. Sunday, well past the noon turnaround rule that exists specifically so hikers can get down before dark. And when darkness came, and the phone rang the sheriff's dispatch for directions, they walked off the Clear Creek Route and into Mud Creek Canyon. One of them fell and hurt a knee. They bivouacked in a drainage overnight. On Monday morning, USFS climbing rangers and search-and-rescue volunteers walked them out.
We Treat Confidence as Correctness, and That Is the Bug
Before the pitchforks come out, re-read the incident. Nobody was irresponsible in the way we imagine irresponsible people being. They did not ignore warnings or free-climb a crack. They did the modern thing: they delegated the thinking about a high-stakes, irreversible decision to an AI assistant that sounded completely sure of itself.
That is the entire problem, and it has nothing to do with Gemini specifically or with a bad model run. An LLM is not dangerous because it lies. It is dangerous because it is confident, and confidence is the emotional trigger our brains are wired to trust. A tool that gave hesitant, hedged answers would be annoying and would be ignored. One that speaks like it has done the route before gets obeyed.
Every hard lesson from this rescue is a mirror of how we already use these systems:
- Gemini lowballed food and water for a multiday emergency — the same way a tool will lowball your legal risk, your retirement number, or your medical second opinion when you ask it to plan around a best case.
- The plan broke the noon turnaround rule, a binding constraint a novice has no way to know is binding from a chatbot's cheerful summary.
- When the plan failed, they had no offline fallback, no map sense, and a phone that turned out to be the only wayfinding they had.
None of this is an argument that AI is useless. It is an argument that we have quietly outsourced the highest-stakes version of a task to the lowest-stakes version of a tool, and we are surprised when reality bills us.
Make AI Argue, Not Decide
The sheriff's office summed up the fix in one unglamorous, correct line: "Never rely solely on AI for your trip planning." The ranger station at 530-926-4511 was sitting right there, staffed by people who have watched people die on that route. Instead, three men trusted a statistical autocomplete that had never stood on the mountain and was trained on other people's summaries of it.
I am not going to pretend I never use AI for planning. I do, all day. The discipline is to decide which plans survive contact with reality, and for that, the stakes decide the rule I actually need to follow.
If the worst outcome is wasted time or a small refund, let the model lead and enjoy the speed. If the worst outcome is an injury, a lawsuit, a wrong prescription, or a body in a gully, treat the AI's answer as a draft that an authoritative human source reviews before I commit to anything with my skin in the game.
- Cross-check the plan against a named, primary source — the ranger station, a real map, the actual regulation, a licensed human.
- Carry an offline escape: methods of wayfinding that do not require the same phone that just led you into the canyon.
- Build in the downside. The hikers planned for an eight-hour day; they needed to plan for one that goes wrong. Ask the model "what breaks this plan," and verify its answer too.
These three men walked out with scrapes and an injured knee, which makes them the lucky counterfactual of a story that ends in a recovery 's report instead of a rescue. The next person who lets an AI plan their unverifiable adventure may not get a Monday-morning ranger team.
I am not arguing we should be afraid of artificial intelligence. I am arguing we should be afraid of the version of ourselves that stops applying judgment on the things that matter most, because a confident digital voice made that easier. Trust the tool to research. Keep the hardest decisions human.
Comments