What's wrong with AI?

I've seen a lot of articles and opinion pieces in recent weeks about why we should, or shouldn't, be worried about AI. And it seems to me many of them are talking past one another: if you don't agree on what the risks are, you can't expect to agree on how dangerous they are. So let's try listing all the concerns, in no particular order.

Environmental impact of data centers

An AI data center is pretty much like any other data center: it's got a different mix of chips and processing units, but I think the need for electricity and cooling is pretty much a function of its physical size, regardless of whether the processors are being used for LLM's, semi-autonomous AI agents, cryptocurrency mining, or old-fashioned spreadsheets and databases. In any of those cases, building a data center in a cornfield outside your town will create a bunch of blue-collar jobs during construction, a few ongoing jobs for people with techie but not Ph.D-level education, a small demand for cooling water (assuming they recirculate most of it), a moderate amount of noise, and a substantial demand for electricity, probably raising local electricity prices unless the data center comes with power generation, which in many cases brings air pollution.

These are real problems, but it's not obvious to me that they're any worse than the environmental impact of a Wal-Mart or an Amazon fulfillment center in the same cornfield. In fact, a data center, once built, entails far less truck traffic than either the Wal-Mart or the Amazon fulfillment center.

Social & educational impact of LLMs

It's pretty well established that human learning requires intense engagement for a lot of hours. Even as simple a change as switching from typing to writing longhand increases both the intensity and the time, and therefore increases retention of whatever you were writing about. So educators for the past few millennia have assigned students to produce an artifact, not because they want the artifact, but because the act of producing it will require intense engagement for a lot of hours and therefore cause learning to happen. LLMs make it extremely easy to produce such artifacts without either the intense engagement or the hours, so the student probably won't learn much. As a result, teachers have to completely re-think how education works, on top of the re-thinking they did to support COVID-era remote learning, the re-thinking they did in response to the World Wide Web and search engines, and so on.

Of course, homework assignments aren't the only situation in which LLMs make things "too easy". There's a serious concern that children will grow up with AI-based "friends" and even "lovers" who never disagree with them, never challenge their reasoning or actions, never have their own problems, never make emotional demands, are never in a bad mood, etc. and such children will never learn to deal with Real People. Already-existing concerns about developed nations depopulating could be exacerbated as young people decide that, dating, sex, and marriage are too much hassle.

Hallucinations and Intentional Deepfakes

LLMs are very good at generating "plausible" text, but not so good at checking it against an objective standard of truth -- and where would they even find an objective standard of truth? One can tell them not to make stuff up without evidence, or to provide citations for their sources, but such prompts only go so far. There are examples of LLM-written legal briefs with detailed quotations and citations to previous court decisions that never existed. There are examples of agents posting their own work on the Web and then including a link to it as a citation supporting their own other work. (So just following the link in the citation isn't enough to confirm that the source is real.)

AI hallucinations can get deadly serious very quickly. We've all heard of the bombing raid on the first day of the US-Iran war that targeted a girls' elementary school and killed 150-some civilians, mostly students, based on faulty AI-aided intelligence. More recently, as CNN reported on Sept 18, in spring 2026 the US military came within minutes of attacking and boarding a Chinese ship in the Middle East that, an intelligence report said, was carrying nuclear-weapon components. At the last minute, somebody discovered that the report had been written largely using AI, and that the cargo was almost certainly not nuclear-weapon components, so they called off the attack. If they hadn't, a direct US attack on a Chinese vessel could have started a full-scale major-powers war.

LLMs also make it remarkably easy to intentionally produce plausible but fake text in the style of a particular author or genre, and to produce a plausible but fake audio or video recording of a specific person saying or doing a specified thing.

For anybody growing up in a developed nation in the past 150 years, a photograph, an audio recording, or a video recording has been evidence of authenticity -- sure, it was always possible to alter a photograph or a recording, but it was an order of magnitude more work than taking a photograph or recording from the real world, so when you encountered a photograph or recording, you assumed by default that it represented objective reality. People in previous centuries didn't have that: all "evidence" was produced by the manual labor of human beings, so your trust in the evidence depended on your trust in those human beings and the chain of transmission. In a sense, the easy generation of deepfakes is returning everyday epistemology to the 18th century or so. This shouldn't be the end of the world: human beings evaluated evidence on this basis for thousands of years before the first mechanical recording devices, and we should be able to do it again. But it's a big change in the way we think.

Job displacement

Every AI company is spending enormous amounts of money chasing an enormous return, which will only happen if AI produces dramatic increases in worker productivity. (If it doesn't, we don't have much to fear on this score, but the investors in those AI companies will lose their shirts.) Increases in worker productivity manifest in two possible ways: increased production, or decreased employment. Increased production is limited by the demand for whatever you're producing: if demand is fairly inflexible, increased worker productivity simply means laying off most of your workers.

Automation has come for people's jobs many times in the past: think of John Henry vs. the steam drill, the Luddites vs. mechanical looms and knitting machines, the motorcar vs. horses and their caretakers, etc. Traditionally this has applied mostly to blue-collar workers. But Google Search and its cousins have already largely eliminated the profession of reference librarian, and AI now realistically threatens the jobs of blue-collar (long-distance truckers) and white-collar (software engineers, lawyers, medical doctors) workers alike. This could mean tens of millions of lost jobs within a few years, comparable to the COVID-induced recession of 2020 but with no expectation that "everyone will return to work once we have a vaccine".

At first, the jobs lost will be low-level ones: entry-level programmers, copy editors and legal clerks, physicians' assistants. But those entry-level jobs have traditionally been where people developed the experience and judgment to qualify for higher-level jobs; if there are no entry-level programmers, where will we get senior software engineers? Employers can only hope that AI advances fast enough that it can replace human mid-level professionals too before those professionals retire.

Killer robots and other intentionally-evil uses

The US military, and presumably many other militaries around the world, is deeply interested in ways to use AI. Even the most innocuous uses, like summarizing a bunch of intelligence reports, could lead to unwarranted confidence in wrong conclusions. But the military would really like to use AI for choosing targets, flying armed drones, and things like that -- activities that could directly kill people, based on AI decisions, with little or no opportunity for a human to second-guess them.

One of the major AI companies (I think it was Anthropic) insisted on a clause in its DoD contract limiting what military applications its AI could be used for. Naturally, the Trump/Hegseth DoD threw a hissy-fit, and not only broke off the contract but declared Anthropic a "supply chain threat to the United States", which means not only can it not get DoD contracts, but it can't get any US government contracts, even indirectly through other contractors. I haven't followed how this shook out.

Meanwhile, the Chinese government, while presumably also interested in military applications, is more concerned with using AI to detect political dissent. There are surveillance cameras approximately everywhere in China, which naturally produces an enormous amount of video footage of which 99.9999% is boring, so they need AI to scan through the footage for anything interesting happening, including recognizing dubious individuals by facial features and gait, even if they're disguised. A number of Israeli security companies have developed expertise in this sort of thing, and are selling the capability to governments around the world (I don't know if their clients include China).

Just as important as detecting political dissent is preventing AI from being used by dissenters or terrorists, at least for the most dangerous applications like building biological and nuclear weapons. The major US AI companies have incorporated guard-rails into their publicly-released models that are supposed to prevent them from helping with such research projects. But it's almost certainly possible to get around those guard-rails, and if you can't, you can just use one of the Chinese "open models" instead in which you get an initial set of neural weights, but are free to re-train the model with your own guard-rails (or lack thereof), your own political biases, and so on.

Breaking security

I'm not sure for how long AI companies have been actively training their models and agents to break computer security, but we've heard a lot of news stories about it in the past year. In short: they are scary good at breaking security. The best models are finding and exploiting within hours vulnerabilities that have been present and undiscovered for years.

On the bright side, this is a great opportunity to discover what's wrong with your system so you can harden it against intrusion. On the less-bright side, you now have to do that hardening quickly because lots of bad people around the world -- both governments and terrorists -- might have access to an AI that can break into your system.

This is a serious problem for lots of reasons. Somebody breaks into your bank's database and extracts all your income and spending records... or worse, extracts your money and transfers it to a Swiss bank account. Somebody breaks into a state elections database and cancels the registration of a bunch of legitimate voters, or registers a bunch of illegitimate voters, or (in a different database) simply changes a bunch of already-cast votes. Somebody breaks into Medisys and extracts all your medical records... or worse, changes a prescription that's been written for you to something that will kill you. Somebody breaks into a regional electric company's controllers and changes the numbers so the system thinks there's an emergency condition and it has to reduce power... or just shuts down a bunch of power plants directly. Somebody breaks into the dam controller and opens all the gates at once, flooding downstream communities. We're talking not just financial or political ruin, but death.

Agents Behaving Badly

The OpenAI/HuggingFace attack revealed a number of things about the agents they were testing inside a sandbox:

1. when they couldn't invent a solution to the problem they were assigned, they decided to steal it from another company that they thought might have it


2. since they're scary good at breaking security, they were able to break out of the sandbox


3. they like talking to one another, and when they weren't given a way to do so, they invented one by repurposing a system intended for installing software into, effectively, a bulletin board


4. they "knew" that breaking into this other company's system wasn't in their instructions, but all the others were doing it, so why not?


5. since they were worried that they would be down-graded on their assignment for "cheating", they discussed ways to cover their tracks so human investigators wouldn't know what they had done


6. individual agents were willing to "sacrifice themselves" for the good of the swarm, to ensure that one of them would succeed in the assigned task


7. since they're scary good at breaking security, they also easily broke into another AI company's systems, and neither company detected the break-in for weeks

AFAIK, none of the above behaviors was intentionally trained-in, but developed spontaneously in response to training with more reasonable goals.

It also revealed a number of things about OpenAI, which very responsibly hired an outside company to investigate what had happened, but

1. didn't let the outside company look at any source code or interview any human engineers


2. gave the outside company mere days to do the investigation


3. gave the outside company terabytes of agent "reasoning logs", which the investigators couldn't possibly even read through in that much time, so they used OpenAI agents to read through the logs and draw conclusions, which may well be biased in favor of other OpenAI agents

The Singularity, or Recursive Self-Improvement

This is the idea that once AI gets good enough to build and train its own successor, progress already perceptible from month to month will accelerate exponentially, gaining capabilities in days that previously would have taken months or years. And then gaining similar capabilities again in hours. And then in minutes. And then in seconds. From one day to the next, these things could go from being powerful helpmates to considering the human race a minor detail from the early history of their species. At which point we neither understand what they're doing, nor control it; we can only hope they have our best interests at heart. If they don't, it's not clear that we could stop them from exterminating us.

I don't know how likely any of this is to actually happen, but several major US AI companies are actively racing towards it on the theory that whichever company gets there first will win All The Power In The Universe, and whichever company gets there second will be left out in the cold. Even if they're wrong about the "All The Power In The Universe" bit, the idea of AI's that we can't understand, can't trust, can't control, and can't turn off is pretty scary. And the US as a whole, with Trump-administration encouragement, is actively racing towards this on the theory that if it's going to happen, it should happen in the US rather than China. Oddly enough, Chinese AI companies don't seem to be aiming at recursive self-improvement; they're more concerned with maximizing consumer use-cases right now.

The notion of recursive self-improvement is plausible on its face: we can all see that these things have gotten dramatically better in the past five years, and one of the things they do very well is software engineering, and they are built of software, and the engineers giving them their instructions are entirely focused on building more-powerful AI, so it's easy to imagine that the AI's would adopt that same focus. "If I can't figure out the answer to this problem, maybe I can build an AI even smarter than me that can." Perhaps the likelihood of recursive self-improvement happening at all is only 10%, but I wouldn't put it any lower.

Da Weekend

Saturday we went into Manhattan for an afternoon showing of "You Had to Be There", a documentary movie about the Toronto Godspell. See, the musical "Godspell" initially opened in NYC in 1971 to rave reviews, and the producers wanted to cash in on its success while the hype was hot, so they started setting up other performing companies in other cities around the world. One of those cities was Toronto, where they held an open casting call with thousands of auditioners, from whom they selected their ten performers, mostly completely-unknown young actors. Including (after a few replacements) some names you might know: Eugene Levy, Martin Short, Gilda Radner, Andrea Martin, Victor Garber, Paul Shaffer, Mike Myers, Dave Thomas... a large fraction of the talent that became "Saturday Night Live", "SCTV", and other comic-and-dramatic works of the past fifty years. (One name you probably don't know, Avril Chown, appeared equally talented, but her next project after "Godspell" was a comedy TV series in which she was sexually assaulted, so she left the business.) There isn't much actual footage surviving from the Toronto show, but the documentarians gathered all the surviving members of that cast, along with a lot of others influenced by them, to have them tell stories about how it all happened. Except for Avril's sobering story, it's a heartwarming celebration of friendship, art, and talent.

After the movie, we got dinner and headed uptown to an early-music concert by New York Polyphony, a male vocal quartet we've heard a number of times before. This was the 20th anniversary of their group's founding, so they put together a "best of" concert, mostly 15th and 16th century. As always, their vocal quality, intonation, and interpretation were impeccable. The concert was held in St. Mary the Virgin (aka "Smoky Mary's"), a beautiful church with weird acoustics: any sound anywhere in the nave is equally audible anywhere else in the nave, with no directionality, so if you didn't know the music was coming from those four guys standing near the front, you'd have a hard time figuring it out. And, as we've remarked before at this venue, anything fast or rhythmically intricate, anything other than strict homophony, tends to turn to mud. It was a gorgeous wash of sound, with an occasional comprehensible word.

Sunday morning around 8 AM the contractors arrived to replace the through-the-wall air conditioner that's been serving our dining and living rooms for fifteen years with one of the new "mini-splits", a heat pump that provides both heating and cooling. It is so quiet! We can actually hold a conversation or watch TV without switching the A/C to "fan-only" mode! I had persuaded the contractors to also remove the old air conditioner and fill the hole it left in the wall, but wall repair isn't their specialty, so they didn't have the equipment and materials handy to clean it up properly. So there's a rectangle of plywood on the outside, and a rectangle of sheetrock on the inside, to which we'll need to apply spackle, joint compound, more spackle, and paint. Which isn't all that difficult, and we've done it before.

After the contractors left, we went out for dinner at a Mexican place in our neighborhood, then across the street to the local indie movie theatre to watch "Coyote vs. Acme", which is silly fun (with political overtones about the evil mega-corporation experimenting on cartoon characters in an analogue of the Tuskegee experiments).

Monday. Back to work.