ArchitectureCyber Defense
AI supply chain security: model signing, AI bills of materials, safe formats and tool vetting
The AI supply chain covers models, datasets, libraries and the tools agents call. A sound design verifies each artifact on the way in, signs what you produce, records contents in a bill of materials and prefers file formats that cannot run code on load. This page sets out a reference path from external source to production, with the standards and documentation that support each control.
On this page
- What the AI supply chain includes
- A verified path from external source to production
- Model file formats and the risk each carries
- Provenance and signing for models
- Two bill of materials formats for AI
- Dataset risks and how to contain them
- Vetting third-party tools and MCP servers
- Questions and answers
- Sources
What the AI supply chain includes
OWASP's 2025 Top 10 for LLM Applications lists supply chain as LLM03, and data and model poisoning as LLM041. Together they cover pre-trained models, fine-tuning adapters, datasets, libraries and the plug-ins and servers that give an agent its tools.
NIST has published a community profile for developers. SP 800-218A, issued in July 2024, augments the Secure Software Development Framework (SP 800-218, version 1.1) with practices for generative AI and dual-use foundation models2. It is a useful anchor for internal policy, but it does not replace the artifact-level controls described below.
The principle is the same as for conventional software: know what you run, verify it came from where you think and keep a record. What differs is that models are large binary files, often loaded by code that may execute whatever they contain.
A verified path from external source to production
- Select and approve source
Approve publishers and registries, and record owner, license and intended use before anything is downloaded.
- Fetch into quarantine
Download to an isolated environment with no production credentials and no outbound access beyond the source.
- Inspect and convert
Scan files, prefer tensor-only formats and convert unsafe formats before use.
- Record in an AI BOM
Capture hashes, versions, datasets and training details in an ML-BOM or SPDX AI package.
- Sign and publish internally
Sign the approved artifact and store it in an internal registry.
- Verify at deploy time
Deployment refuses artifacts without a valid signature from an expected identity.
Model file formats and the risk each carries
| Format or approach | What the documentation says | Practical stance |
|---|---|---|
| Pickle (default for many PyTorch weights) | Python's documentation states that the pickle module is not secure and that you should only unpickle data you trust; unpickling can execute arbitrary code3. | Load only from trusted sources, or convert to a safer format in quarantine. |
| Hub scanning of pickle files | Hugging Face scans imports with ClamAV and a pickle import scan, and says scanning is not 100% foolproof; the dangerous opcodes are STACK_GLOBAL, GLOBAL and REDUCE4. | Treat scanning as a filter, not a guarantee. |
| Safetensors | A simple format for storing tensors safely, as opposed to pickle, while remaining fast5. | Prefer for weights; note it stores tensors only, not code or arbitrary objects. |
| JSON for configuration and data | Python's pickle documentation points to json as the safer choice for untrusted data3. | Use for metadata and settings. |
| Restricted unpickling | It is possible to restrict which globals can be loaded by overriding find_class, but the documentation says to be careful3. | Use only as a last resort, with expert review. |
Hugging Face notes that signed commits guarantee the origin of a file, not its safety4.
Provenance and signing for models
The Sigstore model-transparency project provides tooling for signing and verifying machine learning models. Signing with Sigstore is the default and needs no long-term key management; key-based and PKCS#11 options also exist6. The tool treats a model as a directory and writes a Sigstore bundle containing a signed statement of the file hashes, and signing events go to an append-only transparency log6.
Verification re-hashes the files and checks them against the signed statement, so a changed weight file fails the check6. The project builds on OpenSSF work. In your pipeline, sign each approved artifact at the end of the quarantine stage, then have deployment verify the signature and the expected signer identity before loading.
Be precise about what this proves. A signature shows that a particular identity vouched for specific bytes and that they have not changed. It does not show that the model is free of backdoors or unsafe behavior, which is why signing sits beside testing and the controls in the AI red teaming guide.
Two bill of materials formats for AI
| Aspect | CycloneDX ML-BOM | SPDX 3.0.1 AI profile |
|---|---|---|
| What it represents | Models, datasets and dependencies, including dataset provenance, training methodologies and framework configuration7. | A standardized way to document AI software packages8. |
| Notable fields | Provenance of datasets and training approach, expressed in the CycloneDX model7. | AIPackage properties include autonomyType, domain, hyperparameter, informationAboutTraining, limitation, metric, safetyRiskAssessment, standardCompliance, typeOfModel and useSensitivePersonalInformation8. |
| Dataset coverage | Datasets are components in the same bill of materials7. | A Dataset profile defines DatasetPackage with dataCollectionProcess, datasetType, knownBias, intendedUse and hasSensitivePersonalInformation, among others8. |
| Practical choice | A natural fit if your software supply chain already produces CycloneDX documents. | A natural fit if your tooling and customers already use SPDX. |
The last row is a practical judgment, not a statement from either specification. Pick one format, generate it in the pipeline and keep it with the signed artifact.
Dataset risks and how to contain them
Poisoned or backdoored training data
Early signalUnusual behavior on a specific trigger phrase after fine-tuning on a new dataset.
MitigationRecord dataset sources and hashes, review additions and test for triggers. Anthropic, the UK AI Security Institute and the Alan Turing Institute reported on 9 October 2025 that 250 poisoned documents were enough to backdoor models of 600 million to 13 billion parameters, while 100 were not reliably enough; the backdoor tested was narrow, and it is unclear whether the result holds for larger models or more harmful behaviors9.
Unknown provenance or license
Early signalA dataset or model cannot be traced to an owner or terms of use.
MitigationRequire provenance fields in the bill of materials before approval, and refuse artifacts without them.
Sensitive data embedded in datasets
Early signalPersonal data or secrets appear in samples or model outputs.
MitigationCapture sensitive-data flags such as SPDX's hasSensitivePersonalInformation property and review them before training or release8.
Retrieval stores filled from unvetted sources
Early signalDocuments from public or third-party sources enter the knowledge base without review.
MitigationTreat ingested text as untrusted and apply the isolation layers in prompt injection defense.
Vetting third-party tools and MCP servers
Tools give agents their reach, so each one is a dependency with its own credentials and trust boundary.
Questions and answers
Why are pickle model files risky?
Python's documentation says the pickle module is not secure and that unpickling can execute arbitrary code, so you should only unpickle data you trust3. Because pickle is the default for many PyTorch weights, a downloaded model can run code when loaded4. Convert to a tensor-only format in a quarantined environment where you can.
Does a signed model mean a safe model?
What is an AI bill of materials?
It is a structured inventory of the components of an AI system. CycloneDX ML-BOM represents models, datasets and dependencies, including dataset provenance and training methodologies7. The SPDX 3.0.1 AI profile provides a standardized way to document AI software packages, with fields such as informationAboutTraining and safetyRiskAssessment8.
How many poisoned documents does it take to backdoor a model?
One study found 250 poisoned documents were enough to backdoor models from 600 million to 13 billion parameters, while 100 were not reliably enough9. The backdoor tested was narrow, and the authors note it is unclear whether the result holds for larger models or more harmful behaviors, so treat it as a reason to track data provenance, not as a general threshold.
How should we assess an MCP server before connecting it?
Treat it as third-party code. Review the exact command it runs, sandbox local servers, avoid token passthrough, grant narrow scopes and restrict outbound requests, following the Model Context Protocol's security guidance10. Record the owner and version in your inventory and retest after upgrades.
Sources
- OWASP Top 10 for LLM Applications 2025 — OWASP GenAI Security Project · checked 19 November 2026
- SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models — NIST · checked 19 November 2026
- pickle: Python object serialization — Python Software Foundation · checked 19 November 2026
- Pickle scanning — Hugging Face · checked 19 November 2026
- Safetensors documentation — Hugging Face · checked 19 November 2026
- model-transparency: signing and verifying machine learning models — Sigstore · checked 19 November 2026
- Machine learning bill of materials (ML-BOM) — CycloneDX · checked 19 November 2026
- SPDX 3.0.1 specification: AI profile — SPDX · checked 19 November 2026
- A small number of samples can poison LLMs of any size — Anthropic · checked 19 November 2026
- Security best practices — Model Context Protocol · checked 19 November 2026
More in Cyber Defense
Back to Cyber DefenseYour starting point
What would you like to move forward?
Start with the outcome that feels closest. We'll help give the next step a useful shape.
Let's find your next move
One conversation. A clearer direction.
Tell us what you want to improve. A few sentences are enough to start.
shayan@coldai.org