Most organizations in regulated industries already run AI somewhere in the business. Models sort insurance claims, summarize clinical notes, flag suspicious transactions, and answer staff questions using internal documents. Many of these workloads started as pilots in sandbox environments with broad permissions and light oversight. They then moved into production with the same settings.
The cost of this security and oversight lack is starting to show. In IBM’s 2026 Cost of a Data Breach Report, 21% of organizations reported a breach involving their own AI models or applications, up from 13% the year before. Of those organizations, 92% lacked proper AI access controls.
Most of the controls that would close this gap already exist inside regulated organizations. What’s missing is their application to AI.
If you’re planning an AI rollout and are required to meet CMMC, NIST, or FedRAMP requirements, you may want to take a look here first: Compliant AI Adoption Planning for Government Contractors
Why AI Workloads Belong in the Production Category
An AI system that touches sensitive data is a production system, and as such, it deserves the same scrutiny as a core banking platform or an electronic health record. Many organizations still manage AI as an innovation project, with lighter controls and fewer reviews. That approach made sense for early experiments, but it carries greater risk once a model is running against live data.
What Makes an AI Workload Different?
AI systems reach across the organization in ways most applications don’t. A single model may draw on data from finance, operations, customer service, and HR. Those business units usually keep their data separate for good reasons.
AI workloads also create data stores that didn’t exist before. These include:
- Feature stores that hold prepared data for training and inference
- Vector databases that store documents in a searchable numerical form
- Embeddings that encode the meaning of source text and can sometimes be reversed to recover it
- Fine-tuned model weights that may retain fragments of training data
- Prompt and response logs that capture whatever users typed and whatever the model returned
Each of these can hold sensitive information, and therefore need a data classification and an access policy, the same as any database in the environment.
Where Governance Tends to Slip
Gaps usually appear in the move from experiment to production. Common patterns include:
- Sandbox permissions that follow the workload into production without review
- AI systems that sit outside the change management and access review processes every other production system goes through
- Teams adopting unapproved AI tools to fill gaps in what the organization provides
That last pattern is growing quickly. IBM’s 2026 report found that security incidents involving shadow AI more than doubled to 43%, and 68% of breached organizations lacked AI governance to manage AI use or detect shadow AI.
Applying Production Discipline to AI
The fix starts with classification. Once an AI workload is formally recognized as a production system, it falls under the processes the organization already uses. Security and compliance teams don’t need a new playbook; they need AI included in the one they already run.
Data Pipeline Access Controls
Access control for AI involves more than simply deciding who can use a chatbot. Data passes through several stages between its source and the model’s output. Every stage is a point where the wrong person or process could reach information it shouldn’t.
Map the Pipeline Before You Secure It
A typical AI pipeline moves through these stages:
- Ingestion: Data is pulled from source systems such as databases, file shares, and SaaS applications.
- Preparation: Data is cleaned, labeled, and transformed for use.
- Storage: Prepared data lands in feature stores or vector databases.
- Training or fine-tuning: The model learns from the prepared data.
- Model registry: Approved model versions are stored and tracked.
- Inference: The model receives requests and returns outputs.
- Logging: Prompts, outputs, and system events are recorded.
Each stage needs its own permissions. A common weakness is a single service account with access across every stage. If that one credential is compromised, an attacker can reach the entire pipeline.
Least Privilege for People and Machines
Most access in an AI pipeline belongs to machines. Service accounts, API keys, and automated agents move data between stages around the clock. These non-human identities often hold broader permissions than any employee would be granted.
Practical steps for tightening machine access include:
- Scoping each service identity to a single pipeline stage
- Issuing short-lived credentials in place of long-lived keys
- Storing secrets in a central vault with access logging
- Reviewing machine identities on the same schedule as human accounts
Retrieval Systems Need Permission Checks
Many generative AI tools answer questions by searching an internal knowledge base and passing relevant documents to the model. This approach is called retrieval-augmented generation, and it’s one of the most common places for data to leak.
The risk comes from how documents are stored: when files are converted into a vector database, their original permissions often don’t come with them. A user who could never open a confidential HR file might still receive its contents in a chatbot answer.
Classification Must Travel With Data
Sensitivity labels applied at the source system should follow data through every transformation. When a record is cleaned, chunked, embedded, or logged, its label should come along. If the label is lost during preparation, every downstream control loses its reference point. A vector store can’t enforce a restriction it doesn’t know exists.
Network Architecture for Training and Inference
AI changes how data moves across the network. Large datasets travel to compute clusters and models call out to external services. Network design determines how much of that movement is controlled and visible.
Training moves large volumes of data to compute clusters and depends on fast, synchronized communication between GPUs. Bandwidth and predictable performance matter most here.
Inference depends on low latency and high availability. Its security priority is controlling who and what can reach the model endpoint.
Segmentation and Isolation
Training clusters and the sensitive data stores that feed them should sit in their own network segments, apart from general corporate traffic. Some key practices include:
- Isolating training environments from user-facing networks
- Connecting to model endpoints and managed AI services through private connectivity, away from the public internet
- Restricting which internal systems can call inference endpoints
- Logging traffic between segments so unusual data movement is visible
These practices follow the Zero Trust principles set out in NIST SP 800-207.
Egress Control is a Data Protection Control
Outbound traffic deserves as much attention as inbound. Every connection to an external AI API is a path for sensitive data to leave the environment. This is how many shadow AI incidents begin: an employee pastes regulated data into a public tool, and it leaves the network without anyone knowing.
Network teams can reduce this exposure by:
- Allowing outbound connections only to approved AI services
- Inspecting traffic to AI endpoints for sensitive data patterns
- Logging what data is sent to which service
These controls also give compliance teams a record they can produce when auditors ask where regulated data has gone.
Data Residency and Location
In regulated sectors, where AI processing happens can carry as much weight as how it’s secured. Training jobs and inference requests may route through regions the organization never approved, especially when they use managed cloud AI services. Network architecture should confirm that data stays within approved locations at every stage.
Take a deeper dive into the NIST’s Zero Trust Principles: Zero Trust in Practice: Real Identity and Access Controls
Data Lineage Tracking
Data lineage is the record of where data came from and what happened to it along the way. For AI, that record extends past the data itself, to the models it trained and the outputs those models produced.
What Lineage Records for AI Systems
A complete lineage record for AI connects these points:
- The original source of each dataset and the terms under which it was collected
- Every transformation applied, including cleaning, labeling, chunking, and embedding
- The dataset versions used to train or fine-tune each model version
- The model version that was serving when a given output was produced
With these links in place, a team can trace any output back to its source data. They can also trace forward from a single dataset to every model it shaped.
Provenance and Integrity
Lineage also protects the integrity of training data. Training data should be found from trusted, reliable sources, with provenance tracked as it is used or modified.
This is particularly important because training data can be tampered with, and models degrade as real-world data drifts from what they learned. Integrity checks and provenance records let a team see when data changed, who changed it, and which models were affected.
Lineage Capture Can Be Automated
Lineage kept in spreadsheets or wiki pages falls out of date as soon as a pipeline changes. The more reliable approach is to capture lineage inside the orchestration and data tools themselves, so each pipeline run writes its own record.
Those records deserve the same protection as audit logs. Start with new models, then work backward to the existing models that handle the most sensitive data.
What Happens When the Foundations Are Skipped?
Gaps in AI infrastructure tend to surface months after launch, usually at an inconvenient moment. Missing controls can turn into compliance and exposure problems:
The audit question has no answer: An auditor asks which data trained a model that supports credit or claims decisions. Without lineage, the team has to piece together an answer from old notebooks, emails, and memory. That can take weeks, and the result may not satisfy the auditor.
The breach has no boundary: A dataset turns out to be compromised, and no-one can say which models trained on it or which vector indexes still hold it. The incident response team has to assume the widest possible impact, which expands notification obligations and recovery work.
The model that has to be retired: A customer exercises a right to deletion, or a data source turns out to lack proper consent. If the team can’t locate that data across training sets and embeddings, the safest option may be to retire the model and rebuild it from scratch.
The endpoint anyone can reach: An inference endpoint left open to the wider network becomes a target for attacks on the model itself. In IBM’s 2026 findings, model inversion attacks averaged $6.07 million per breach and prompt injection attacks averaged $5.89 million.
The project stalls before launch: Data problems discovered late can halt a project entirely. Gartner predicts that through 2026, organizations will abandon 60% of AI projects that aren’t supported by AI-ready data.
Where You Should Start
Few organizations can fix every gap at once, so the order of the work matters. These steps give a practical starting point.
- Build an AI inventory: List every model, AI tool, and AI-related data store in use, including tools teams adopted without formal approval. Network logs and SaaS discovery tools can help surface them.
- Rank workloads by data sensitivity: Start with systems that touch regulated data, such as patient records, financial data, or controlled unclassified information.
- Assign an owner to every AI system: Each system needs one named person who is accountable for its access decisions and its records.
- Threat model new initiatives at the outset: Run data security threat modeling and privacy impact assessments at the start of any AI initiative.
- Test retrieval systems against your permission model: Have users with different access levels ask the same questions, then compare the answers. Any difference that exposes restricted content points to a gap.
Strong Foundations Make AI Easier to Trust
Organizations that put these foundations in place early tend to find each new AI use case easier to launch. Data arrives already classified and traced, and security reviews move faster because the groundwork is done.
From there, the questions go deeper. Models and training pipelines need protection from tampering in their own right. Sector rules shape how data can be handled, while capacity planning also becomes pressing once training and inference begin to scale.
SMS has delivered IT and infrastructure to U.S. organizations since 1976. Our engineers have designed and operated environments for federal agencies, defense organizations, power and energy providers, and commercial companies working toward FedRAMP authorization.
That work calls for the same disciplines AI tools now need: tightly scoped access and reliable, up-to-date compliance records. If your AI workloads run on regulated data, we can help you assess the foundations underneath them and close the gaps that carry the most risk.
