Pope: Don’t Build a Monster You Can’t Manage
The Pope’s AI warning for CEOs: move fast, but don’t scale systems nobody owns or trusts.
Pope Leo XIV has written a substantial papal letter on AI. It deserves attention. Your next meeting starts in seven minutes. Here is the boardroom version.
AI is not just software, it is power
It decides what people see, what gets approved, who gets hired, who gets served, who gets watched, who gets replaced, and who gets ignored.
Pope Leo XIV’s core warning is crisp:
“technology is never neutral.”
That should make every CEO uncomfortable.
Because most companies are not building AI around values.
They are building it around speed, cost, scale, and margin.
Those are fine goals. But they are not enough.
If AI is trained to cut cost, it may cut corners.
If it is trained to maximise engagement, it may feed addiction.
If it is trained to optimise labour, it may treat people as waste.
If it is trained to personalise everything, it may quietly manipulate everyone.
That is the Pope’s point.
Not “AI bad.”
More like: AI makes your hidden operating system visible.
If your culture is extractive, AI can extract faster.
If your culture is sloppy, AI can scale the slop.
If your culture is paranoid, AI can become surveillance.
If your culture is humane, AI can help.
The warning is simple: do not build Babel.
Babel is the biblical story of people building a tower to prove their power. Big, clever, impressive, and doomed.
It lost the plot because the tower became the mission: power, pride, and control replaced purpose.
That is the risk with bad AI strategy.
The dashboards improve. The system gets faster. The automation spreads. But no one can explain who owns the outcome, where the judgment sits, or why customers and workers should trust it.
That is not transformation.
That is Babel in business dress.
CEOs love scale. Fair enough.
But AI does not just help you scale.
It helps you scale what you already are.
The Pope is asking: scale of what?
Scale help, or scale harm?
Scale judgment, or scale noise?
Scale trust, or scale control?
Scale human ability, or scale human replacement?
That is the whole game.
Here is my take of the papal letter.
AI needs owners
Not committees. Owners.
If an AI system denies a claim, rejects a candidate, prices a customer, flags a worker, or gives medical advice, someone senior must own the outcome.
“The model did it” is not a defence.
It is an admission you lost control.
I made the same point in Who Is Accountable When AI Decides?: AI can recommend, rank, draft, and detect. But it cannot be accountable. That remains a human job.
People need an appeal path
If your AI touches a person’s money, job, health, safety, status, or rights, they need a way to challenge it.
Fast. In plain English. With a human who can fix it.
No appeal path means no trust.
And no trust means the business model eventually breaks.
That is why Trust Is The New AI Battleground. As AI output becomes cheap and everywhere, trust becomes scarce.
Scarce things become valuable.
Do not automate your conscience
Some decisions are too human to hand off.
Hiring. Firing. Policing. Care. Credit. Education. War.
The more serious the consequence, the more human judgment must stay in the loop.
Not as theatre. As authority.
This is the difference between delegation and abdication. In AI Moves Work, Not Magic, I argued that AI does not remove the need for management. It moves the work. It changes where judgment is needed.
Bad leaders use AI to avoid decisions.
Good leaders use AI to make better ones.
Your workers are watching
If AI is just a headcount weapon, expect fear, resistance, leaks, and brand damage.
If AI removes dull work, grows capability, and shares the upside, people may come with you.
“Reskilling” is not a slide. It is a deal.
Who gets trained?
Who gets moved?
Who gets protected?
Who shares in the gains?
Who is told early enough to plan?
That is the actual work.
The better example is not “replace people with bots.” It is redesigning work around new capability. That is why CBA Redesigns Work for AI matters.
The dumb version of AI strategy is cost-out.
The serious version is capability-up.
Truth matters again
AI makes cheap content infinite.
That means trust becomes a moat.
Do not pump out machine-made noise and call it marketing.
Do not fake intimacy.
Do not hide synthetic work.
Do not make customers wonder whether anyone real is home.
Every company will be tempted to use AI to say more, publish more, target more, personalise more, and automate more touchpoints.
Most of it will be junk.
Some of it will be harmful.
A little of it will be useful.
The CEO job is to know the difference.
Institutions still matter
The Pope is not really worried that machines will become human.
He is worried that humans will become machine-like.
Measured. Ranked. Predicted. Optimised. Managed by systems nobody can explain.
That is the deeper warning.
AI can simulate output. It can simulate tone. It can simulate care. It can simulate authority.
But it cannot create trust by itself.
Trust comes from institutions that keep promises, own errors, tell the truth, and protect people when things go wrong.
That is why Synthetic Scale Fails Without Real Institutions is the business version of the same argument.
Scale without trust is fragility
Scale without accountability is risk.
Scale without humanity is just a taller tower.
The Pope says the task is to put “the human person at the center of our choices.”
That is the CEO takeaway.
Use AI.
Move fast.
Compete.
But do not confuse capability with wisdom.
Do not ship systems nobody owns.
Do not hide behind the model.
Do not reduce people to behavioural data.
Do not turn workers into cleanup crews for bad automation.
Do not scale what you would be ashamed to explain.
The Pope’s message is not anti-tech.
It is anti-stupidity.
Build AI that helps people do more, know more, make better calls, and live with more dignity.
Anything else is just Babel with a better user interface.