Engineering teams have spent years moving security checks earlier in the delivery process. Code is scanned on commit, and dependencies are checked against known vulnerabilities before a build moves forward.
Machine learning (ML) pipelines tend to sit outside that routine; training datasets and model files often move from experimentation to production without passing through the same gates. Kiteworks’ 2026 forecast found that 78% of organizations cannot validate data before it enters training pipelines, and 77% cannot trace where their training data came from.
So what would it take to give an ML pipeline the same scrutiny as the application code wrapped around it?
DevSecOps for Machine Learning: Why ML Pipelines Need Checkpoints
Answering that question starts with the assets themselves: an application pipeline mostly handles source code and its dependencies, while an ML pipeline adds large datasets and trained model files to the mix, and standard code scanners weren’t built to inspect either one.
Both asset types carry risks that are easy to underestimate.
Model Files Can Carry Hidden Code
A trained model can do more than make predictions. The risk comes from how certain model formats are serialized, which allows code to run during loading. For engineering teams, this means a model pulled from a public hub deserves the same scrutiny as an unvetted software package.
A Small Amount of Bad Data Goes a Long Way
Training data carries its own exposure. A joint study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that injecting 250 malicious documents into pretraining data was enough to backdoor LLMs ranging from 600M to 13B parameters. The researchers concluded that attack success depends on the absolute number of poisoned documents, regardless of what share of the training data they represent.
The study used a deliberately narrow backdoor that produced gibberish text, which the researchers note is unlikely to pose significant risks in frontier models. Its broader lesson still applies to any team training or fine-tuning models: larger datasets don’t dilute the threat.
Both risks trace back to what enters the pipeline and who has the ability to change it. That makes training data the logical place to set the first checkpoint.
Any AI tool that can access sensitive data needs certain security parameters: Setting AI Infrastructure Security Foundations in Regulated Environments
A Starting Point: Who Can Access the Data?
Training data shapes how a model behaves. It deserves the same access discipline that teams already apply to a customer database or a production secrets store.
In many organizations, training data lives in shared storage buckets and notebook environments. Permissions are usually granted broadly during early prototyping, and rarely revisited once a project moves toward production.
Least Privilege
Access control for ML starts with mapping who and what touches each dataset. Pipeline jobs need the same identity discipline as human users.
- Separate access by data stage: Raw ingested data and curated training sets should sit behind different permissions. Write access to curated sets should be limited to the pipeline jobs that produce them.
- Give training jobs scoped, short-lived credentials: Shared service accounts with long-lived keys are hard to audit and easy to misuse.
- Review notebook and experimentation environments: These often hold wide read access that was granted for speed and never tightened.
- Log every read and write: Joint guidance from CISA, the NSA, and the FBI recommends that teams log all data access and model inference activity, and implement role-based access controls and immutable logging.
Track Provenance
Provenance answers two questions about every dataset. Where did it come from, and what has happened to it since?
The same joint guidance from CISA, the NSA, and the FBI mentioned in the previous section advises that data used in AI training should come from trusted, reliable sources, with provenance tracking so that data can be reliably traced as it is used or modified. In practice, that means a few habits applied consistently.
- Version datasets with content hashes: Every training run should reference an exact, reproducible snapshot of its data.
- Record the source and license of external data: This matters for security reviews and for legal questions that surface later.
- Keep transformation steps as versioned code: Each change to the data then has a commit that someone can review.
- Sign trusted revisions: CISA, the NSA, and the FBI also recommend using digital signatures to authenticate and verify trusted revisions during training, fine-tuning, and post-training processes.
Validation Gates
In application pipelines, a failing test stops a merge. Training data can be held to the same standard, with automated checks that block a run when something is off.
- Schema checks: Confirm that fields, types and formats match what the training code expects.
- Distribution checks: Compare a new data version against the last approved one and flag unexpected shifts.
- Anomaly detection: Looks for outliers and suspicious patterns. The same guidance from CISA, the NSA, and the FBI proposes that organizations employ data sanitization and anomaly detection tools to identify outliers or suspicious patterns in training data.
- Clear failure reports: Tell the data team which check failed and why, so fixes happen quickly.
Treat these checks as pipeline steps that can fail a build. A gate that only produces warnings tends to get ignored under deadline pressure.
If you’re required to meet CMMC, NIST, or FedRAMP requirements, you may want to take a look here before adopting AI tools: Compliant AI Adoption Planning for Government Contractors
Next Steps: Verify Model Artifacts Before Deployment
Most platform teams already require container images to come from a trusted registry with a verified signature. A trained model deserves the same treatment; it’s a file produced by a build process, and it can be swapped or altered at any point between training and deployment.
Sign and Verify Models
Signing gives each model a verifiable record of who produced it and confirms it hasn’t changed since. The OpenSSF model-signing project has made this practical for ML teams.
With its 1.0 release, the project’s maintainers recommended that every ML pipeline sign models using Sigstore as the default approach to ensuring model integrity, though the library also supports public key infrastructure and self-signed certificates.
To build this into the pipeline:
- Sign as the final training step using the pipeline’s workload identity.
- Verify on upload to the model registry and reject anything unsigned.
- Verify again at deployment and fail closed if verification doesn’t pass.
- Hold third-party models to the same rule. Verify the publisher’s signature where one exists. Where none exists, review the model internally before signing it with your own identity.
Scan Serialized Models
Some model formats can run code the moment they are loaded. According to Hugging Face, Pickle is the default format for PyTorch model weights, and loading a pickle file can enable dangerous arbitrary code execution attacks.
Scanning helps, but it has limits. ReversingLabs points out that Picklescan relies on a blacklist of dangerous functions, and Checkmarx researchers found other functions that could trigger code execution but were missing from that list. A layered approach works better.
- Scan statically in CI: Tools such as ModelAudit detect malicious code and security issues in ML model files without loading or executing them.
- Prefer safetensors for weights: Hugging Face developed safetensors as a simple serialization format for weights, and many frameworks now support it.
- Quarantine pickle-based files from external sources until they have been reviewed.
- Load untrusted models only in isolated environments with no network egress and no access to production credentials.
Record Lineage
A signature confirms who produced a model and that it hasn’t been altered. Lineage records what went into it. NIST SP 800-218A covers this directly, with recommendations on securing code storage, managing model versioning and lineage, and clarifying shared responsibilities among organizations.
Store these details in the registry with each model version:
- The hash of the exact dataset snapshot used for training
- The code commit and dependency lockfile for the training job
- Hyperparameters and evaluation results
- The identity that signed the artifact
Lineage pays off during an incident. If a dataset turns out to be compromised, the team can quickly find every model trained on it and roll back to a verified version. Signed, traceable models also help teams automate much of the incident response work that follows a compromise, which is a stated goal of the model-signing project.
Monitoring: Drift and Tampering in Production
Deployment shifts the focus of security work. Once a model is live, the checks move from the artifact itself to how the model behaves under real traffic.
Watch for Drift
Data drift happens when the data a model sees in production starts to differ from the data it was trained on. The joint guidance from CISA, the NSA and the FBI names data drift as one of its three primary risks, along with data supply chain vulnerabilities and maliciously modified data.
Drift monitoring works best when it starts from a clear baseline:
- Capture input distributions at training time and compare live traffic against them on a regular schedule.
- Track prediction patterns and output confidence: Sudden changes in either can signal a problem before accuracy metrics catch up.
- Compare predictions against ground truth as labeled outcomes become available.
- Define retraining thresholds in advance: Guidance from CISA, the NSA, and the FBI mentioned in earlier the section A Starting Point: Who Can Access the Data? suggest implementing feedback loops to periodically retrain models and defining thresholds for when a model must be refreshed.
Unexplained Changes Could Be Tampering
Natural drift and deliberate manipulation can produce similar signals. Investigate before assuming that a shift is harmless.
Feedback loops deserve particular attention; deployed AI systems are attractive targets for exploitation, especially if feedback data can be manipulated to degrade model performance over time.
- Validate feedback data before it enters retraining: Apply the same gates used for the original training data.
- Re-verify model hashes when a model loads and at scheduled intervals while it runs.
- Watch for unusual query patterns, such as high volumes of near-identical requests that may indicate probing.
- Send model and inference alerts to your existing SIEM so they reach the on-call team through established channels.
Fold ML Security Into Your Existing Workflow
None of these controls require a separate security stack. Most can run on the CI/CD platforms, policy engines and logging tools that teams already maintain.
Extend Tools and Standards
- Use existing CI runners for data validation and model scanning jobs.
- Enforce signature checks with policy-as-code, using the same admission controls that already govern container images.
- Build on the frameworks your organization already follows: NIST designed SP 800-218A (mentioned in the earlier section Next Steps: Verify Model Artifacts Before Deployment) to be used in conjunction with the Secure Software Development Framework many software teams already reference.
Start With One Pipeline
A full rollout across every model at once rarely succeeds. A staged approach gives teams a working pattern they can repeat.
- Inventory current models and datasets, including where each is stored and who can access it.
- Add access controls, validation gates and signing to that pipeline and confirm they hold under normal delivery pressure.
- Turn the result into a template that other teams can adopt.
Strengthen Your ML Pipeline Security
The organizations that secure AI well tend to make one shift in thinking: they stop treating models as research outputs and start treating each one as a release, with the accountability that comes with it.
Securing an ML pipeline draws on several disciplines at once, including identity and access design, CI/CD hardening, compliance mapping, and production monitoring. SMS has spent decades doing this work in government and highly regulated environments, where every system has to stand up to audit.
Talk to our engineers about securing your AI workloads with the same governance as the rest of your production environment.
