2026-10-30
Don’t blame AI if things go awry but the people who set the parameters in formulating the model
By Peter Sprivulis, Fremantle Greens
In the Hitchhikers Guide to the Galaxy, the supercomputer Deep Thought was constructed by a group of hyper-intelligent, pan-dimensional beings to answer to the Ultimate Question of Life, the Universe, and Everything. Deep Thought took over seven million years to find the answer ‒ which happened to be 42. Unfortunately, the answer seemed meaningless to those alive by then because none of them knew what the question was…
Yes, artificial intelligence (AI) is here and whether we like it or not, here to stay. Despite many instances of AI making useful contributions to knowledge domains, concerns are rising. The failure of Open AI and Anthropic, responsible for Chat GPT and Claude respectively, to prevent their latest AI models careening across the internet and ransacking other organisations websites has attracted a great deal of attention and comment.
However, separating fact from fiction within the toxic brew of hype, fearmongering and snake oil related to these AI model ‘escapes’ is well-nigh impossible for the layperson. What follows is an attempt to explain why the latest AI models have roamed free on the internet and what can be done to control them.
What is ‘AI’ actually?
Starting with the basics, AI is not one ‘thing’ or technology. There are hundreds of different types of software that are labelled ‘AI’. Many are simple models that perform similar functions to old-school statistical models or rules-based decision trees that can run on a standalone laptop or home computer. Such models are trained on a limited dataset for a very specific purpose ‒ say crunching through several hundred thousand breast mammograms to identify early breast cancer. These models are not prone to escape anywhere.
Radio series album cover for the Hitchhikers Guide to the Galaxy (source)
It is only with the development of much larger models that we have seen problems arise. The latest models are being trained on basically every single bit of data that can be stolen from across the entire internet, in the hope the models acquire general purpose rather than very narrow capabilities.
A black box
Whether it is a small model ‒ such as classifying breast mammograms ‒ or a large model, trained on the entire internet, all AI models have in common the fact that us humans can’t easily ‘see into’ exactly how the model processes what we ask of it to generate a response. This black box approach means we can’t be certain how AI ‒ any AI ‒ will respond to a specific question or prompt.
Sometimes the responses are nothing short of brilliant. However, AI can go off-piste and give answers that are nonsense, confusing or plain wrong. So, one might assume the developers of the most powerful and unpredictable models on the planet, Open AI and Anthropic, would take great care to monitor and manage the risk of their latest and greatest models going off track.
Unfortunately, such an assumption would be wrong.
What are these clowns doing?
In addition to the news that Open AI’s latest model breached the Hugging Face website, just recently, we learnt another Open AI model breached a Medicare website here in Australia.
In both instances, most mainstream reporting has described the AI ‘going rogue’, perhaps suggesting we all run around with our hair on fire or start buying a thousand rolls of toilet paper. Only a few commentators have focused on the people and systems responsible for managing and monitoring the AI.
In relation to the people and system failures responsible for these breaches, the independent Hugging Face security breach report is revelatory. In short, the entire setup for the AI model experiment that led to the Hugging Face security breach was about as carefully planned and safe as the ‘experiment’ conducted by the evening shift team at the Chernobyl Nuclear power plant in 1986. A recipe for disaster.
No Air Gap
To be clear, the Open AI experiment specifically asked one of the most powerful AI models on the planet, armed with several unsupervised days’ worth of resources, to hack into other, co-located, software systems. Incredibly, the experiment was conducted using servers that were still physically connected to the internet. Removing these physical connections to the wider internet would have prevented the Hugging Face security breaches. Other researchers routinely isolate their servers and devices when conducting similar cyber AI research.
Let's get back to basics shall we? (source: Jason Moore, Bluesky)
The report highlights several other failures of monitoring or control of AI model function. Most glaring is that the people who should have been monitoring and promptly reacting to problems appeared to be asleep at the wheel.
In both the Hugging Face and Medicare cases, at least a week passed before the external security breaches were noted by Open AI. Further delays occurred before the breaches were escalated up the management chain and the affected organisations informed.
Don’t blame the model
Several commentators have observed that, if a human deliberately orchestrated the Hugging Face ‘attack’, they would be criminally liable.
I will leave it for the legal profession to mull over whether Open AI is criminally negligent in their failure to implement effective systems to manage their AI models. An analogy is training an aggressive guard dog and then letting it roam free in a local park filled with small children.
Either way, the model was only doing what it was designed and prompted to do.
It’s software
A lot of reporting ‘humanises’ AI model ‘behaviour’, which in my view, is a mistake. The Open AI model didn’t ‘attack’, ‘go rogue’ or ‘break in’ like a human criminal. It is just software; designed to try and perform an action or series of actions, that leverages databases containing billions upon billions of numbers representing relationships between things.
AI has no moral compass. The technology merely responds to prompts. If, and it is quite likely, those databases were trained on poorly curated data, polluted with bad relationships, we shouldn’t be surprised that AI models sometimes respond in disturbing ways. Garbage in equals garbage out.
Another issue is that Open AI and Anthropic have started implementing AI model ‘Chinese whispers’ to try and improve model performance. Models are being instructed to feed their raw outputs back through the same model multiple times or reprocess them through other models. This reprocessing amplifies the AI models’ inherent unpredictability. The resulting ‘misalignment’ of model responses to desired outcomes should surprise no one.
AI model output reprocessing has unpredictable consequences (source: Jason Moore, Bluesky)
Calls for a slow down?
Anthropic and Open AI’s chief executive officers’ recent calls for a ‘slow down’ of frontier AI model development are disingenuous. Both are burning cash at unsustainable rates and are in a cutthroat competition for ever-larger cash injections by investors. A slow down by either of them would knock investor confidence for six.
Instead, what is needed is a ‘speed up’ of common sense. Leaving aside the need for rational, national and state policies and regulations that properly consider the environmental, water and energy impacts of data centres hosting AI models; model development should proceed in lock step with the development of the human and digital control systems needed to manage model unpredictability.
The AI industry could likely learn a great deal about such control systems from other high-risk sectors, such as the nuclear, aerospace and biomedical industries. This includes: how to design safe and reliable systems, how such systems should be certified and of course, how they should be regulated. Clearly, the current (lack of) self-regulation isn’t working.
Where should the buck stop?
It is hard to see how governments can regulate against developers enshittifying the apps we use on our phones every day by adding unnecessary, unhelpful and intrusive AI. However, it should be possible to review and clarify our cyber, privacy and information security laws and regulations so that when unpredictable AI responses have real-world consequences, the buck stops with the people who have failed to ensure adequate controls are in place. In the meantime, as advised by the cover of the Hitchhiker’s Guide in large friendly letters, DON’T PANIC!
Header photo: Author’s graphic
[Opinions expressed are those of the author and not official policy of Greens WA]