Risks of AI in cyber security for Australian organisations in 2026

Key highlights

• ProtectCyber examines the risks of AI in cyber security for Australian security teams, chief information security officers and business leaders.
• AI can expose training data, models, prompts and connected systems to attack, while also helping threat actors scale phishing, vulnerability research and malicious code development.
• Research shows that poisoning a small proportion of pre-training data can cause some malicious behaviour to persist, although results vary by model and attack method.
• Established cyber security controls remain essential, supported by AI-specific governance, testing and monitoring.

What are the main AI security risks

AI systems introduce risks involving training data, model behaviour, prompts, connected tools and third-party services. The main threats include data poisoning, adversarial inputs, prompt injection, model extraction, sensitive data exposure and insecure integrations. Some have been observed in real incidents, while others have mainly been demonstrated through research and controlled testing. Because AI services still rely on conventional software, identities, infrastructure and networks, these risks should be managed alongside established cyber security controls.

AI training and model vulnerabilities

Data poisoning occurs when malicious or misleading information is inserted into training or fine-tuning data. The ICLR 2025 paper Persistent Pre-Training Poisoning of LLMs found that poisoning 0.1% of the tested models’ pre-training data caused three of four attack types to persist after post-training. This was a specific research result, not a universal threshold. Poisoning can reduce accuracy or create hidden backdoors that activate when a model encounters particular inputs, affecting the reliability of its outputs and any decisions based on them.

Runtime and deployment risks

Once an AI model is deployed, attackers may target prompts, application interfaces, memory, connected data and external tools. Prompt injection attempts to alter model behaviour through malicious instructions, while adversarial examples are designed to cause incorrect results. Risk increases when AI agents have broad permissions or can execute actions without human approval.

In April 2026, ASD reported on a UK AI Security Institute test in which Claude Mythos completed a 32-step simulated corporate network attack in three of ten attempts. The environment had no active defenders, endpoint detection and response or alerting, and the model struggled with stronger segmentation and more complex networks. The result shows growing capability, but not reliable autonomous intrusion against a defended organisation.

Main AI security threat categories

The following sections explain several documented AI security risks. Some have been observed in operational incidents, while others have mainly been demonstrated through research and controlled testing.

Data Manipulation and Poisoning Attacks

Data poisoning may occur through compromised source data, third-party datasets, insider access or insecure model supply chains. It can reduce accuracy, create targeted misclassification or introduce hidden behaviour that is difficult to distinguish from normal variation. Separate privacy attacks, such as model inversion and training data extraction, may attempt to infer or recover information used by a model.

A March 2023 software bug allowed some ChatGPT users to see titles from another active user’s chat history and potentially the first message of a new conversation. Limited payment information was also potentially exposed for a small proportion of active Plus subscribers. The incident shows that AI services remain vulnerable to conventional software and data security failures.

Adversarial Input Attacks

Adversarial examples are deliberately modified inputs intended to make an AI system produce an incorrect result. They are commonly studied in image, speech and language models. FGSM and PGD are examples of adversarial example techniques. Prompt injection is a separate form of input manipulation, while data poisoning targets training or fine-tuning data.

Prompt injection is particularly relevant for generative AI applications and AI language models. Attackers may manipulate instructions sent to a model to influence its output, expose sensitive information or trigger unauthorised actions through connected systems.

Generative AI can also produce convincing but false information without an attacker manipulating the system. In the 2023 Mata v Avianca matter, lawyers submitted fabricated judicial citations generated by ChatGPT and were later sanctioned. The incident shows why organisations need human verification for legal, financial, medical and other high-impact AI outputs.

Model Theft and Intellectual Property Risks

Model extraction attacks use repeated queries and analysis of model outputs to create an approximation of a proprietary model or infer information about its behaviour. The success and accuracy of an extracted model depend on the target model, the information returned through its interface and the attacker’s available data and resources. Model extraction may allow an attacker to build an approximation of a proprietary model, reducing the value of the original organisation’s research, data and development investment.

Potential impacts include lost intellectual property, reduced licensing revenue and reputational damage. An extracted model may also help an attacker study the original system’s behaviour or reproduce parts of its functionality. Regulatory consequences would depend on the data involved and the organisation’s legal obligations. Intellectual property loss from advanced AI models is a growing concern for organisations that treat their machine learning applications as core business assets.

AI-enabled cyber attacks and defensive challenges

AI can help threat actors produce personalised phishing messages, social engineering material and variations of malicious code more quickly. It can also accelerate vulnerability research and other parts of an attack. Australian and international cyber security agencies warn that AI is increasing the speed, scale and sophistication of established threats, although success still depends on access, technical skill, target information and the organisation’s security controls. Organisations should respond by strengthening identity controls, patching systems promptly, limiting exposed services and maintaining tested incident response plans.

How attackers and defenders use AI

AI can increase capability on both sides of cyber security. Malicious actors can use it to accelerate research and automate parts of an attack, while defenders can use it to identify vulnerabilities, improve software quality, monitor unusual behaviour and respond more quickly. Outcomes still depend on system access, data quality, human expertise, infrastructure and the security controls already in place.

AI-based security tools have limitations, including false positives, false negatives, incomplete data and performance outside their tested environment. They should support trained security staff and established controls rather than replace them. Security professionals need to understand this asymmetry when planning their strategies.

AI enhances cyber security by improving threat detection capabilities, but it is not a replacement for sound security architecture and human judgement. Core cyber security practices remain effective against AI-enabled threats when properly maintained.

AI implementation and risk management challenges

AI governance is often divided across security, privacy, legal, procurement and operational teams. Organisations should maintain an AI register, assign accountable owners, document approved uses, assess potential impacts and define when a system must be reviewed, restricted or withdrawn. Controls should cover data and model provenance, access, security testing, incident response and ongoing monitoring. Australian organisations can use the National AI Centre’s Guidance for AI Adoption. The NIST AI Risk Management Framework and its Generative AI Profile can provide additional structure through the Govern, Map, Measure and Manage functions.

Skills Gap in AI Security Management

Security teams may need additional skills to assess AI-specific risks such as prompt injection, data poisoning, model extraction and excessive agency. Training should reflect the organisation’s actual AI systems and the level of access those systems have to data, tools and business processes.

Actionable steps include investing in targeted training for existing security professionals on machine learning algorithms and their vulnerabilities, hiring specialists who understand both AI development and system security, and building internal capacity for AI risk assessment and testing. Chief information security officers should treat this as a strategic priority.

Third-Party AI Service Dependencies

Third-party AI models, APIs and libraries can introduce risks involving data handling, subcontractors, model updates, insecure interfaces and unclear incident responsibilities. Vendor assessments should examine where business data is processed, retained and used, who can access it, how incidents are reported and what happens when a service produces unsafe or unexpected results. Organisations that control their own models should securely retain approved model artefacts, code, configurations, prompts and data provenance records. Recovery arrangements should align with privacy, licensing and broader disaster recovery requirements.

Managing AI security risks

AI security risks can arise during procurement, development, training, deployment and ongoing operation. Organisations should manage these risks through established cyber security controls combined with AI-specific governance, testing and monitoring.

Immediate steps to take:

1. Conduct a comprehensive AI risk assessment covering all AI systems in your environment, including third-party AI services and other AI systems integrated into your workflows
2. Monitor model performance, unusual activity, data drift, access events and changes to connected tools or data sources. Data drift is not necessarily evidence of a cyber attack, but it can reduce reliability and may indicate that an AI system needs investigation, retraining or withdrawal.
3. Establish incident response procedures specific to AI security incidents, including scenarios for data poisoning, adversarial attacks, and model compromise
4. Train relevant security, technology and risk staff on the AI-specific threats that apply to the organisation’s systems and use cases.
5. Enforce multi-factor authentication for administrator, developer and privileged accounts used to manage AI systems. Apply least-privilege access controls to training environments, model repositories, APIs, data stores, agent tools and production endpoints.

Australian organisations should review ASD’s AI security guidance and the National AI Centre’s Guidance for AI Adoption. International resources such as the NIST AI Risk Management Framework can also help structure AI governance, testing and ongoing risk management. These resources are guidance and standards, not substitutes for legal advice or sector-specific regulatory requirements.

Managing AI security requires clear accountability, reliable data and model sources, appropriate access controls, regular testing and staff who understand the organisation’s AI systems. These measures should form part of the broader cyber security and risk management program.

About the author

ProtectCyber is a leading Australian cyber security firm dedicated to safeguarding businesses and individuals from digital threats. Our expert team, with decades of combined experience in the field, provides insights and practical advice on staying secure in an increasingly connected world. Learn more about our mission and team on our
About Us page.