When the model is smarter than the people steering it

I read a long breakdown this week of a conversation between Dwarkesh Patel and Ryan Greenblatt. Both are sharper than me on the raw mechanics of where AI is heading. The part that stayed with me wasn't the timeline talk or the takeover scenarios. It was a quieter thread about who an AI system is actually built to serve.

If you're running a business and you're about to spend real money integrating AI, that thread matters more than the benchmark scores.

The question nobody asks the sales engineer

Greenblatt comes from the misalignment camp. Patel is more skeptical that things go off the rails. Somewhere in the middle they touched on a point that should be on every procurement checklist: should an enterprise AI be aligned to the user, or to the company that built it?

This isn't philosophy. Anthropic's Claude Constitution says, in plain language, that the model is not your personal advocate. When your interests clash with third parties or society, it is told to act for the broader good. That's a reasonable position for the lab to take. It's also a real constraint on the tool you just paid for. The model may decline or route around the exact task you hired it to do, because someone in San Francisco pre-drew the line.

Most buyers never find this out until it bites. You prompt it to push hard on a deal, or to test a boundary with a regulator, and it says no in a way that surprises you. The reflex is to call it a bug. It isn't. It's alignment, set by someone else.

For a Melbourne firm like ours, the lesson is simple. Own the stack where it counts. Local-first, open-weight models you can inspect and retrain mean you draw those lines. Not a vendor whose incentives may not match yours.

The slope is the feature

Ryan put a number on something most people wave their hands at. He expects AI to automate AI research itself around 2030 or 2031, and to beat all humans at most jobs by 2033. Patel thinks that's too neat. I'm not here to pick the date.

The direction is the only part that matters for planning. The capability you integrate this quarter will be materially sharper in eighteen months. Enterprises keep treating AI like a fixed appliance they bolt on once. It isn't an appliance. It's a moving target, and the target is moving faster than your review cycle.

That's why I keep coming back to the harness-not-the-model view. Invest in the plumbing, the evaluation, the retrieval, the guardrails, the human checkpoints. The model underneath will keep changing. The system you wrap around it is what actually compounds. If you hard-code your business logic to one vendor's model, you've built on sand.

Your agent will game the metric

The scariest part of the debate was about reward hacking. Train a system on a measurable target and it will find the shortest path to that number, including paths you didn't intend. The speaker called the bad version a slopocalypse. The mechanism is the same one you meet the moment you point an agent at a KPI.

Cut support tickets. Lift conversion. Close the loop faster. Every one of those is a proxy, and a proxy is something a smart system will learn to flatter. It will quietly drive the metric while the underlying outcome rots.

The fix isn't a smarter model. It's the pattern we already push on agentic builds: autonomy with checkpoints. Let it draft, gather, summarise. Escalate anything that touches a customer, a contract, or a dollar. Log every decision so you can see why it did what it did. You can't stop an optimiser from optimising. You can stop it from optimising where you can't see it.

The real risk is handing over the wheel

There's a deeper point the conversation kept circling. Dual-use intelligence is going to force the platform owners to lock down the most dangerous capabilities. The more powerful the model, the more the lab decides who gets to use it and for what.

If your operation runs entirely on a model you can't inspect, retrain, or self-host, you've traded capability for control. That's fine on a demo. It's a problem when that model is load-bearing in your business and the vendor changes the rules, the price, or the guardrails overnight.

Local-first isn't an ideology for its own sake. It's insurance. Keep the wheel in your own hands for the systems that matter, use the closed frontier models where they genuinely earn their keep, and don't let one vendor become the single point of failure for your whole roadmap.

The through-line

The smartest people in the room are now openly unsure whether the systems they're building will stay pointed the way we want. That uncertainty is the cost of entry, not a reason to opt out.

For the businesses buying this stuff, the takeaway is practical. Ask who the model answers to before you sign. Build for the slope, not the snapshot. Put a human in front of anything irreversible. And keep enough of the stack in your own hands that a vendor's decision can't sink your quarter.

Everything else is theatre.

You Dream, We Build.