AI software supply chain risks prompt new corporate diligence

AI software supply chain risks prompt new corporate diligence

Share:
Christopher Jablonski

Christopher Jablonski

Contributor

Zscaler

Jan 13, 2025

AI code is already a challenge to secure, but the arrival of autonomous or agentic AI will mean even greater difficulty.

If your organization is using generative AI, you may want to widen your cybersecurity purview.

Like the risks of any cloud software supply chain, it is often hard to understand or predict downstream threats to cybersecurity that originate in the AI software supply chain. Many vulnerabilities can originate in your second or third-tier software supplier and go undetected until it is too late.

The mass enterprise adoption of AI/ML, LLMs, virtual agents, and chatbots over the last few years can reveal risk fault lines in shared code and infrastructure pushed through devops pipelines. Even with proper vendor risk management and due diligence, ‌your tech stack ‌could‌ be eventually compromised. 

CISOs are on the look out for novel attack vectors or future breaches of the upstream assets partners use to deliver their cloud-delivered AI services directly or to engineering teams for further tooling. These include code libraries, training data, and open source infrastructure as code (IaC) templates. Risks include manipulated or poisoned data sets and adversarial attacks.

Here’s an industry insider’s take on the scope of the challenges we face this year:

“By 2025, supply chain security will demand a whole new layer of vigilance, where even the datasets and AI models feeding into our applications are analysed for adversarial tampering. A secure supply chain won’t just be about code but curating safe and verifiable AI training sources,” Cache Merrill, founder of software development company Zibtek, recently told TechRepublic.

Even if your SaaS vendor mgmt and AI supply chain is about as airtight as it can be, security problems could arise from errors in the outputs of “trusted” GenAI tools, such as those used to generate code. 

Perils of AI-generated code

Using GenAI to code new software is not new and it’s turning developers into productivity superheroes. Savvy software development teams are augmenting their coding practices with AI, such as in quality assurance and testing. There’s an entire marketplace of AI coding solutions ready to take your business to new heights.

However, without proper testing, inspection, and overall adherence to an organization’s official software development life cycle (SDLC), AI-created code could introduce weaknesses and even cause outages. When bad code is made, put in a public repository, and others use it in their projects, the problems can quickly spread into the software supply chain. 

AI code generation models and the code they output can have exploitable vulnerabilities

Cybersecurity Risks of AI-Generated Code

 

The arrival of autonomous or agentic AI, which lets AI create its own code, gives AI a lot of power, but also makes the challenge even harder. AI agents can become multi-agent systems that are shared over the internet.

Zscaler Chief AI Officer, Clauadionor Coelho explains: “AI agents will cause a tremendous increase in the attack surface. On top of LLM-based attacks, we can add leaking of internal confidential data and model integrations and runtime data, and application code attacks, to name a few.”

All of this brings a new formidable risk to the AI ‌software supply chain, especially since they’re compounded by potential vulnerabilities in the training data and models used to make the code generators as well.

“By enabling tools to dynamically change or even enabling LLMs to generate new tools whenever an AI agent detects that it cannot perform a task with the current toolset, we open the door to completely new types of attacks that have never been considered,” Coelho warns. 

The stealth of data-poisoning attacks  

Another type of vulnerability that can lurk in the AI software supply chain is data poisoning. Here, an attacker contaminates or manipulates a model’s training data for an intended goal. Attacking AI this way involves no hacking in the traditional sense, making it hard to detect.

An attacker can manipulate an LLM’s training data to produce code that imports a malicious package for instance, or a library that can carry out phishing attacks or cause unexpected behaviors. Hugging Face, a community for AI developers, serves as a prime example of an attempted data poisoning attack when researchers identified a hundred malicious models on the platform. 

Lessons learned from cloud software supply chain risks

After software supply chain breaches at Target and Home Depot ten years ago, software supply chain risk made headlines again in 2020 with the SolarWinds data breach. The hack embedded malware code in the company’s network and monitoring platform, gaining backdoor access to all the systems connected to all the SolarWinds customer machines running the Orion platform. In the process, it exposed 30,000 public and private organizations–including public-sector agencies. Other supply chain attacks like MOVEit and Log4J have underscored the need for third party and software supply chain risk mitigation, and enterprise security teams have responded.

With AI at the center of IT innovation today, it will likely overtake low-code and no-code software development. The scale of code generated and shared openly this year and into the future will be staggering. 

Mitigating risks to the AI software supply chain

There are many things security and engineering leaders can do to help mitigate against the risks of the AI software supply chain.

Start with a cybersecurity program consisting of multi-factor authentication, endpoint detection and response, data encryption, data backups, and regular system updates. The next step is modern hygiene that includes a zero trust architecture and advanced capabilities like app-to-user segmentation, and honeypots and decoys.

To further reduce and mitigate the unique AI software supply chain risks covered in this article, consider how the various steps below can have a positive impact, including on your DevSecOps cycle:  

  • Customize your third-party risk assessments for AI - Adapt your protocols and provisions for GenAI and LLMs. Check for certifications like ISO/IEC 27001 and your vendor risk assessment reports are conducted by a reputable Governance, Risk, and Compliance organization. When building AI in-house, ensure the chosen platform(s) fits your data governance, intellectual property, and privacy policies. Extend your procurement due diligence to cover the AI software supply chain and inquire about validation and data cleansing mechanisms.
  • Consult guidance, such as secure software development practices and the NIST Cybersecurity Framework - Integrate AI software supply chain security measures into your CI/CD pipelines and expand secure-by-design principles to include code generation models and other AI systems.  Research and implement advanced defense strategies such as adversarial training and data input validation.  
  • Use continuous threat exposure management (CTEM) to scan for hidden vulnerabilities across your software supply chain - Define a scope that encompasses the software development process, including third-party APIs, microservices, code libraries, and AI vendors involved in creating software for your organization.

By making your entire ecosystem and supply chains that underpin GenAI, LLMs, etc., part of your overall threat universe, you can better safeguard your enterprise against emerging threats in the AI age.  

Resources:
CSET: Cybersecurity Risks of AI Generated Code 
NIST: Integrating Software Supply Chain Security in DevSecOps CI/CD Pipelines
Cloud Security Alliance (CSA): AI Controls Matrix (in peer review) 
 

Recommended