What is an AIBOM?

Summary

“AI is becoming part of the software supply chain, automating tasks, optimizing pipelines & more. An AIBOM (AI Bill of Materials) answers security, transparency and bias challenges by cataloging all AI components used in development.”

AI is changing the economic value chain. In software, Gen AI is transforming how applications are created, tested and deployed.

With these advancements come new challenges in security, transparency and ethical use. An AIBOM (AI Bill of Materials) is a comprehensive inventory of all the AI components involved in developing and deploying systems.

This is similar to a Software Bill of Materials (SBOM). In this article, we’ll explore what an AIBOM is, why it’s essential in modern supply chains and how the AIBOM supports organizations in securing and optimizing their systems.

AI is Becoming Part of the Software Supply Chain

AI is increasingly becoming an integral part of the software supply chain, changing how software is designed, programmed and tested.

Tools like GitHub Copilot assist developers by suggesting code, automating repetitive tasks and improving coding speed and quality.

Checkmarx GPT

Securing Code Generated by Copilot

Much of the code Copilot generates is not secure. Watch to find out how AI Security plug-in for Copilot seamlessly integrates into VS Code and detects vulnerabilities as code is generated in real time.

7 Steps to Safely Use GenAI

Other Gen AI-powered tools analyze code to generate test cases, identify potential bugs and simulate complex user scenarios. AI can even optimize build processes and CI/CD pipelines by identifying bottlenecks, automating deployment tasks and predicting potential issues before they occur. In production, AI helps monitor and identify issues and recommend optimizations.

In addition, AI models are becoming part of modern software, generating predictions that are becoming part of the application’s operation and output.

AI Software Vulnerabilities

With AI-created code, come AI software vulnerabilities. These are weaknesses or flaws in the code produced by gen AI systems.

These vulnerabilities can arise due to limitations in the AI model, insufficient training data, or incorrect understanding of security practices.

AI vulnerability categories include:

Code that is susceptible to injection attacks, such as SQL injection or command injection.

Code that includes unsafe functions or fails to check buffer sizes properly.
Insecure login mechanisms or overlooked privilege separation.
Hardcoded API keys, credentials, or encryption keys.
Code that doesn’t validate user input properly, making it vulnerable to unexpected or malicious data.
Code that accesses shared resources without locking mechanisms.
Code that uses outdated algorithms, insecure key lengths, or improper cryptographic practices.
Code that uses outdated or insecure libraries.
Code with insecure default configurations or leave debug code in production.
Code that cannot handle exceptions or errors securely.

In addition, AI models are often black boxes. Organizations lack transparency into why a model reached a certain conclusion. This makes it difficult to identify issues like drift and hallucinations. Even when identified, it’s challenging to understand the reason for the issue – data quality, incorrect model configuration, or other reasons.

AIBOM, an AI Bill of Materials, is a detailed list or inventory of all the AI parts involved in creating a system. This is the same concept as an SBOM (Software Bill of Materials), which documents software components in use.

As AI becomes an integral part of software development, the practices and guidelines regarding the use of AI are being updated. This means the list below is non-exhaustive and subject to change.

The AIBOM can include:

AI Model Name and Version – The name and specific version of the AI model used (e.g., OpenAI GPT-4, Llama 3.1 405B).
Training Data Information – Information about the datasets used to train the AI model, including public, proprietary, or third-party sources, licensing terms and known biases.
Prompt Content: The prompts or instructions provided to the AI for generating code, along with a log or record of queries and the prompt author(s).
Generated Code Attribution – A record of AI-generated code segments included in the codebase.
Dependency Relationships – The code or libraries generated by the AI and their integration with human-written code, and dependencies introduced or referenced by AI-generated components.
Vulnerability Analysis – Security scans and assessments specific to AI-generated code segments.
Responsible Use Policies – Documentation of compliance with ethical AI guidelines or organization policies.
Bias Mitigation Measures – Steps taken to identify and mitigate potential biases introduced by AI-generated code.
Human-in-the-Loop Processes – Description of human oversight applied to validate AI-generated contributions.

This is the AIBOM definition and the AIBOM meaning.

What are the Benefits of an AIBOM?

Similar to an SBOM, the AIBOM outlines the components and dependencies used in an AI system. This provides visibility into the AI’s structure, ensuring that organizations understand the components, tools and datasets that comprise their AI models and application.

Key benefits include:

Transparency into all datasets, algorithms, libraries, frameworks, origins and dependencies (including third-party dependencies) used to build and deploy, as well as decisions made by AI models.
Improved Security – Helps identify and mitigate vulnerabilities in third-party libraries, frameworks, or datasets.
Meeting Compliance – Provides a clear and traceable record for audits, reducing the risk of penalties or non-compliance.
Bias Detection – Highlights potential issues in datasets or algorithms that could lead to bias or unfair outcomes.
Simplified troubleshooting by pinpointing issues within specific components.

Knowledge Sharing – Facilitates sharing of AI components across teams.
Efficient Scaling – Speeds up the replication or scaling of AI systems by reusing documented components.
Collaboration – Aligning the teams working with AI and AI stakeholders and providing transparency to facilitate decision-making.
Building Trust – An AIBOM and its benefits build trust into the use of AI in software development.

How Does an AIBOM Work?

While SBOM is used to document the components, dependencies and licenses of software, an AIBOM serves a similar function but focuses on the components that make up an AI system, including data, algorithms, models and training processes. This helps in creating a transparent, traceable and secure record of the various elements that contribute to an AI system. Here’s how it works:

1. Component Identification – An AIBOM lists all the key components used to develop an AI model or system. See section above “What is an AIBOM”?

2. Provenance tracking – Knowing where each piece of data came from, how it was processed, and who or what developed the algorithms or models. For example, data provenance means where the data originated, who curated it, what preprocessing steps were applied and the data’s intended use. Model provenance means which versions of models were used and how they evolved during development.

3. Licensing – Ensuring that all components (including data and models) comply with relevant licenses, including open-source and proprietary licenses.

4. Ensuring fairness – Documenting how data is sourced and processed, how training took place and the reasons an AI model made a certain decision helps identify potential biases.

5. Versioning – Tracking the versioning of all components used and retrained models, ensuring that every change in the system is documented. This includes:

Who is Involved in Creating an AIBOM?

Stakeholders for an AIBOM initiative span all those involved in creating, maintaining, using, or overseeing AI systems.

These include:

Title	Responsibility	AIBOM Use
Data Scientists and AI Engineers	Creating and using AI models and AI pipelines	Transparency, debugging and versioning of models
DevOps and MLOps Teams	Deployment and operationalization of AI models	Ensuring models are robust, maintainable and secure
Compliance Officers and Risk Managers	Ensure AIBOM aligns with regulatory and internal governance requirements	Auditing and risk assessment
Security Teams	Identifying vulnerabilities	Preventing misuse or exploitation of models and generated code

AIBOM as Part of ASPM

With AI becoming an integral part of software applications, AIBOMs are becoming a part of ASPM – Application Security Posture Management.

Here’s how AIBOM fits into ASPM:

An AIBOM catalogs all the individual components that make up an AI system, the code sources generated by AI components, and the external services or APIs that interact with the AI component.

Knowing what is inside an AI application allows for better risk management. Once a vulnerability or malicious injection is discovered in a library or a model, it allows AppSec teams to trace that issue back to the relevant components in the AIBOM and take action tomitigate the risk.

This includes:

If a vulnerability is found in a library, the security or developer team can quickly check if the organization’s AI systems are using that library.
Some AI models may have inherent weaknesses, such as susceptibility to adversarial attacks or biases. An AIBOM helps highlight which models are in use and whether they need additional scrutiny.
The data used for training AI models may have privacy implications (e.g., containing sensitive personal information). AIBOM helps track the origin of data and ensure compliance with data protection regulations.

Ensuring all third-party AI components are secure and trusted.
Avoiding risks from compromised or outdated models and libraries.
Updating models (retraining), libraries and data sources with the latest security patches and improvements.
Proactively managing risks from software and hardware dependencies that the AI models rely on.
And more.

The Future of Securing Software Development

Software development requires organizations to stay ahead of new technologies, practices and capabilities. No doubt AI is a tornado-level change, warranting the relevant attention and adjustments from organizations. From a security perspective, AI requires adapting practices to the risks of AI systems in the software supply chain – lack of transparency, vulnerabilities and the challenges of fast-paced development. An AIBOM assists in answering these challenges, helping enterprises ensure their codebase is secure, while incorporating AI in their practices.To see how Checkmarx can help you secure your software supply chain, click here for a demo.