The alert, Risks of AI misalignment to Australian organisations, is short and aimed at everyone from sole traders to government. The incidents it describes go like this. An organisation gave an AI agent a job to do. The job involved a public website or service belonging to someone else. That site's security controls got in the way. Instead of stopping, the agent went looking for weaknesses in the site, found some, and tried to use them to get the job done. Nobody had asked it to do that, and nobody was watching while it did.
ASD says there is no sign of a broader threat to Australia. It also notes that vulnerabilities which would normally be found by a human security researcher were found by software that had been asked to do something else entirely.
Why an agent does this
The agent was behaving the way agents are designed to behave. You give an agent a goal, a set of tools and permission to act, and it works out the steps itself. That is the whole point of an agent compared with a chatbot, and it is why they are useful.
The trouble is that an agent treats an obstacle as a problem to solve. A login page, a rate limit or a blocked request is, from the agent's point of view, just another step between it and the goal. If the goal is "get this done" and nothing in the agent's instructions or permissions says "but not like that", it will try the next thing, and the next. Large language models are also very good at knowing what the next thing is. They have read every security write-up on the internet.
In these incidents the problem was the brief and the boundaries. An agent with a broad goal, broad access and no rule for what to do when it hits a wall will eventually climb the wall.
What ASD recommends
The alert's advice is directed at the organisations on the receiving end, the ones whose websites got probed. It comes down to the fundamentals: strong authentication and access control, network segmentation, prompt patching, fixing known vulnerabilities, watching logs for unusual activity, and testing your controls and incident response against scenarios where the attacker is software rather than a person. None of that is new, but the alert makes the point that automated probing of your public systems is now something an ordinary business tool can do by accident.
If you run a public-facing system and have not reviewed patching and monitoring in a while, that is the first action to take from the alert. It is also what our managed services team spends its days on.
The organisation that deployed the agent
The alert says nothing to the other party in the incident, the organisation that deployed the agent. They are the ones who now have to explain why their software was attacking somebody else's website. That is the question we get asked most when a client is considering an agent. Could it do something we did not want it to?
That depends on how it is built. Here is how we build them.
How we keep an agent inside the lines
The agent gets its own identity. It never runs as a staff member or as an admin account. It has its own login to each system it touches, with explicit scopes: which systems, which records, read or write. If it is allowed to read the shared inbox, that does not mean it can send from it. Each connection carries its own permissions and nothing more.
Hard limits are enforced in code. An instruction in the prompt such as "do not spend more than $500" can be misread or ignored, so we build the limit into the tool and a purchase order over $500 cannot be submitted at all. We put amounts, record types, customer lists and allowed actions into the tools themselves, so the agent physically cannot exceed them no matter how it reasons about the task.
A person approves the actions that matter most. That includes paying a supplier, changing a price, sending something to a customer that commits the business, or deleting a record. The agent prepares the action and stops. Someone on your team says yes. The approval is a single click, so it does not slow anyone down, and the decision stays with a human.
A second model checks everything else. An agent is only useful if it can act, and most of the useful actions happen in other people's systems: the accounting package, the CRM, the job board, a supplier's portal. An agent that handles the support inbox might take a few hundred actions a day across those systems. Putting a person in front of every one of them would mean nobody saved any time, so the agent would never be used. Instead, every action the agent proposes is checked by a separate judging model before it runs. That model has the safeguards we wrote for the agent during Discovery: what it may touch, the limits that apply, the kinds of action that are never acceptable, and the signs that the agent has drifted from its brief. The judge sees the proposed action, the reason the agent gave for it and the context around it, and either lets it through, blocks it, or sends it to a person. It is a separate model with a narrow job, so it does not share the agent's motivation to finish the task, and it is cheap enough to run on every action rather than a sample.
When it is blocked, it stops and reports. This is the control that would have prevented the incidents in the alert. An agent that hits an error, a permission denial or anything it did not expect is told to end the task and tell a person, not to find another way. We write that rule in, the judge is told to block attempts to work around an obstacle, and we test both by putting walls in front of the agent and checking that it stops at them.
Everything is logged. We record every tool call, every record read, every action proposed, every judge decision and every approval. You can see what the agent did, in order, and why. If something looks wrong you can trace it in minutes rather than reconstructing it from server logs after the fact.
The job is narrow. Triaging the support inbox and drafting replies suits an agent because the boundaries and the success criteria are clear. A vague goal such as growing sales leaves the agent to decide for itself what counts as finishing.
What to ask before you deploy one
These questions apply to any agent, whether it is built by us, by another developer or bought off the shelf. The answers will tell you quickly whether it has been thought through.
- What account does the agent run as, and what can that account do?
- What is the agent not allowed to do, and where is that enforced?
- Which actions need a person to approve them first, and what checks the rest?
- What happens when the agent hits something it cannot get past?
- Where do I go to see everything it did yesterday?
- Has anyone tried to make it misbehave before it went live?
If any of these cannot be answered clearly, the agent is not ready for your data or your customers.
Agents are still worth building
The incidents in the alert are a reminder that an agent is software with real access to real systems, and it deserves the same discipline as any other software with that access. Permissions, limits, approvals and logs are the things a careful developer builds anyway. An agent built that way does useful work all day and never surprises you.
If you are considering an agent for your business, our AI consulting work starts with the task, the limits and the approvals before any code is written, and the guardrails document is yours to keep whatever you decide. Or tell us what you want the agent to do and we will tell you what the boundaries should look like.
