Air Gapped AI: A Security Team's Deployment Guide

Air-gapped AI means running an AI system inside a physically isolated enclave where no routable path to the public internet exists. Models, weights, data, and inference outputs never cross that boundary. If your workload touches classified information, Controlled Unclassified Information (CUI), or any environment governed by FedRAMP High, DoD IL5/IL6, or CMMC Level 3, you are almost certainly required to consider it.

Three quick decision checks before you read further:

• Regulated classification: Does your data carry a CUI, SECRET, or higher marking, or does it live inside a SCIF? A true air gap is likely mandatory, not optional.
• Contractual or accreditation requirements: Does your contract reference DoD IL5/IL6, FedRAMP High, or CMMC? Those frameworks expect evidence of enforced isolation, not just a policy statement.
• Outbound telemetry prohibition: Does your security policy or data-handling agreement forbid any outbound call from the AI runtime? Many commercial LLM deployments make silent telemetry calls that violate this on day one.

For small teams or individuals who need provable privacy without enterprise-scale infrastructure, Greencube offers a fully offline AI deployment that runs entirely on a single machine with no cloud connection required. For enterprise accreditation, the rest of this guide covers the full architecture.

Table of Contents

• What does "air-gapped AI" actually mean, and how is it different from just "offline"?
• What core components must live inside an air-gapped enclave?
• How does an air-gapped AI deployment actually run day to day?
• Which deployment shape matches your regulatory tier?
• What are the most common ways air-gapped AI projects fail?
• What are the real operational tradeoffs of running AI in an air gap?
• Step-by-step implementation checklist for an air-gapped AI deployment
• What do practitioners and auditors actually look for in air-gapped AI?
• Key Takeaways
• The part most guides skip: air-gapping is a lifecycle commitment, not a one-time build
• Private offline AI without the enterprise overhead
• Useful sources for accreditation and further reading
• FAQ

What does "air-gapped AI" actually mean, and how is it different from just "offline"?

The phrase gets misused constantly. Teams say "isolated" when they mean "on-prem." They say "air-gapped" when they mean "behind a VPN." Auditors know the difference, and so should you.

True air-gapped AI means the enclave has no physical or logical path to any external network. No DNS resolution to external hosts. No external certificate authority trust chains. No outbound NTP to public time servers. No vendor license callbacks. No NAT gateway that could be re-enabled. The system is, in the literal sense, off the institutional network graph, which removes entire attack classes including lateral movement, botnet enrollment, and network-mediated compromise.

How on-prem and VPC "isolation" fall short

Many teams mistake VPC isolation or a private cloud deployment for an air gap. They are not the same thing, and auditors treat those as differentiators. Here is what makes on-prem and VPC patterns weaker:

• NAT gateways and allowlists: Traffic can still leave the environment via permitted routes. An allowlist is a policy control, not a physical one.
• Vendor control-plane access: Many on-prem AI platforms phone home for license validation, model updates, or telemetry. That call breaks any air-gap claim.
• External CA trust chains: If the system trusts a public certificate authority, an attacker who compromises that CA can issue certificates your enclave will accept.
• Shared DNS resolvers: Any DNS query that reaches an external resolver is an outbound channel, even if the query itself seems harmless.

The "no routable path" test

Assessors and accrediting authorities use a simple test: can any process on any machine in the enclave initiate a connection that reaches a host outside the boundary? If the answer is yes under any condition, including maintenance windows, the environment does not qualify as air-gapped. Run a controlled inference job and capture all network traffic at the host, hypervisor, and physical switch layers simultaneously. Zero outbound packets is the only passing result.

What core components must live inside an air-gapped enclave?

Nothing in the AI stack can depend on an external network at runtime. That sounds obvious, but the dependency list is longer than most architects expect on first pass.

The full component inventory

• Local model registry: Signed model weights stored in an internal registry with provenance metadata. No pulling from Hugging Face or any external hub at runtime.
• Inference engine: Local serving stacks such as vLLM or Text Generation Inference (TGI) run entirely on enclave hardware. No external API calls during inference.
• Vector store and embedding models: The retrieval-augmented generation (RAG) stack needs a local vector database (Qdrant, Weaviate, or Milvus deployed internally) and local embedding models. Calling an external embedding API is one of the most common ways an air gap silently breaks.
• Container and image registry: A mirrored registry (Harbor is a common choice) holds all container images. No pulls from Docker Hub or any public registry.
• Package mirrors: Python packages, OS packages, and any other runtime dependencies must come from an internal mirror. pip install from PyPI is not available.
• Internal PKI and CA: The enclave runs its own certificate authority. No external CA trust chains.
• SIEM and log storage: Observability data stays inside the boundary. Logs ship to an internal SIEM; nothing goes to a cloud-hosted logging service.
• Orchestration layer: Kubernetes with Helm, or a container runtime like Podman, deployed and managed entirely inside the enclave.
• Local identity provider: User authentication runs against an internal IdP (Active Directory, FreeIPA, or a FIPS-validated equivalent). No external OAuth or SAML federation to a public IdP.

Hardware and cryptography notes

GPU sizing depends on the model you are serving. A 7-billion-parameter model quantized to 4-bit can run on a single A100 or equivalent; a 70-billion-parameter model at full precision needs multiple high-memory GPUs. Storage must use encryption validated under FIPS 140-2 or FIPS 140-3 where DoD or FedRAMP High compliance is required. CPU selection matters for inference on smaller models: high core counts and fast NVMe I/O reduce time-to-first-token on CPU-only deployments. Plan for redundant storage and local backup capacity since no cloud backup path exists.

Software Bill of Materials (SBOM) generation is not optional in regulated environments. Every container image and model artifact entering the enclave needs an SBOM that auditors can inspect. Artifact provenance, including the signing key used and the chain-of-custody record, belongs in the ATO evidence package.

Pro Tip: Wire your local observability stack to alert on any process attempting a network connection outside the enclave's internal CIDR range. In Kubernetes, a NetworkPolicy that defaults to deny-all egress and permits only intra-cluster traffic is the minimum starting point. Pair it with a host-level eBPF sensor (Falco works well here) to catch connections that bypass the CNI layer.

How does an air-gapped AI deployment actually run day to day?

At runtime, inference and retrieval run entirely on local hardware. A user query hits the local inference engine, which calls the local vector store if RAG is configured, and returns a response. No packet leaves the enclave during that entire cycle. That is the design goal, and it is also where most deployments quietly fail.

The update and patch path

Since you cannot pull updates from the internet, every artifact update follows a staged transfer process. Successful deployments rely on one-way ingest pipelines where containers, model weights, and libraries are scanned, signed, and checksummed on a staging machine before physical transfer into the enclave.

Data diodes (hardware-enforced one-way transfer devices) are the preferred path for high-classification environments. Removable media (a "sneakernet" transfer) is acceptable at lower tiers when the media is encrypted, tracked by serial number, and handled under a written transfer SOP. The SOP must specify who is authorized to carry media, what scanning happens before and after transfer, and how the media is sanitized or destroyed afterward.

Which deployment shape matches your regulatory tier?

Not every workload needs a full air gap. The right isolation level depends on the classification of the data, the accreditation framework governing the system, and the contractual obligations in your agreement.

The isolation continuum

Connected (standard cloud or SaaS): Suitable for public or low-sensitivity data. No isolation controls beyond standard access management. Appropriate for FedRAMP Low or unclassified commercial workloads.

BYOC (Bring Your Own Cloud / VPC deployment): The model runs in your cloud tenant, but the provider's control plane still has some access. Appropriate for FedRAMP Moderate in many cases, but auditors will scrutinize vendor access paths. Isolated AI networks at this tier still permit outbound DNS and CA calls, which matters for your threat model.

Air-gapped with data diode ingest: A physically isolated enclave with a hardware-enforced one-way ingest channel. Appropriate for FedRAMP High, DoD IL5, HIPAA workloads with strict outbound prohibitions, and CMMC Level 3 environments handling CUI.

Fully air-gapped (no ingest channel at runtime): Updates arrive only via vetted removable media under strict SOP. Required for DoD IL6 (SECRET) and above, SCIF deployments, and any system where even a one-way hardware channel is considered an unacceptable risk surface.

U.S. regulatory mapping

Practical signals that you need to move up a tier: your contract contains a clause prohibiting data from leaving a specific boundary, your data carries a CUI or higher marking, your auditor's questionnaire asks for evidence of enforced isolation rather than a policy statement, or your ATO package requires proof that no data exfiltrated during a controlled test.

What are the most common ways air-gapped AI projects fail?

The gap between "we have an air-gapped policy" and "our enclave is actually air-gapped" is where most projects break. The most common failure is a misconfigured RAG pipeline where a local model attempts to call a remote embedding or search API, silently breaking the enclave boundary.

The failure list practitioners actually see

• RAG pipelines calling external embedding APIs: A developer configures LangChain or LlamaIndex with a default embedding provider (OpenAI, Cohere) and deploys it inside the enclave. The first query triggers an outbound HTTPS call. The model appears to work, the enclave is broken, and no one notices until an audit.
• Implicit DNS leaks: A container resolves an external hostname during startup for a health check or a license validation. The DNS query leaves the enclave even if the TCP connection is blocked.
• External CA trust chains: The system ships with the default OS trust store, which includes public CAs. A compromised or mis-issued certificate from any of those CAs could be accepted by the enclave.
• Unvetted removable-media imports: A well-meaning operator plugs in a personal USB drive to "just copy one file." Without a written SOP and enforced scanning, that is an uncontrolled ingest path.
• Telemetry baked into model serving frameworks: Some inference servers phone home for usage metrics or version checks by default. Check every framework's default configuration before deployment.
• Supply-chain risks in container images: A base image pulled from a public registry before transfer may contain a dependency with a hidden outbound call. SBOM diffing catches this; skipping the diff does not.

Mitigations that actually hold up under audit

Default-deny egress at both the host (iptables/nftables or eBPF) and the orchestration layer (Kubernetes NetworkPolicy) is the baseline. Treat compliance as a runtime property and enforce controls in the kernel and orchestration layers so compromised processes cannot reach the network even if they try. Artifact signing with a key held offline and checksum verification at ingest catches supply-chain tampering. Least-privilege identity policies limit the blast radius if a process is compromised.

Pro Tip: Before any accreditation review, run a "silence test": start a full inference workload, including a RAG query, and capture all network traffic at the physical switch for 15 minutes. Any packet with a destination outside the enclave's internal CIDR is a finding. This test is cheap to run and catches hidden dependencies that policy reviews miss entirely.

What are the real operational tradeoffs of running AI in an air gap?

Air-gapping solves a specific threat model. It does not solve every problem, and it creates several new ones. Decision-makers who treat it as a universal answer end up with expensive, hard-to-maintain systems that underdeliver.

The tradeoff summary

Update cadence: Cloud-hosted models get continuous updates. An air-gapped deployment updates on a controlled cycle, typically monthly or quarterly, depending on the transfer SOP and the accreditation body's requirements. That lag means your model may be several versions behind the current state of the art. For most regulated workloads, that is an acceptable tradeoff. For workloads that depend on real-time knowledge, it is a serious constraint.

Model freshness vs. provable isolation: You cannot have both simultaneously. Every update cycle is a risk event that requires signing, scanning, and chain-of-custody documentation. Teams that try to update too frequently create audit fatigue and increase the chance of a procedural error during transfer.

Hardware costs: You own the hardware. A GPU cluster capable of serving a large high-parameter model costs significantly more than a cloud API subscription for equivalent throughput. Factor in redundancy, physical security, power, and cooling.

Audit overhead: Every artifact transfer generates documentation. Every access event generates a log entry. The operational cost of maintaining an auditable air-gapped enclave is real and ongoing.

Capacity planning

Quantization (4-bit or 8-bit) reduces GPU memory requirements substantially and is the standard approach for fitting large models onto available hardware. The quality tradeoff is measurable but acceptable for most document analysis and question-answering tasks.

When air-gapping is cost-effective: The data carries a classification or regulatory designation that makes any outbound path legally or contractually prohibited. The threat model includes nation-state adversaries or insider threats with network access. The ATO requires evidence of enforced isolation.

When BYOC or connected tiers suffice: The data is unclassified and the primary concern is vendor data use policies. A strong BYOC deployment with strict egress controls and a reviewed vendor agreement may satisfy the requirement at lower operational cost.

Step-by-step implementation checklist for an air-gapped AI deployment

This checklist covers the three phases every deployment goes through: pre-deployment planning, staging and ingest, and ongoing operations. Adapt it to your specific accreditation framework; the sequence holds across FedRAMP, DoD, and HIPAA contexts.

Phase 1: Pre-deployment planning

1. Map your threat model. Identify the adversary types (insider, nation-state, supply chain), the data classification, and the specific controls your accreditation framework requires. Document this before touching hardware.
2. Capture accreditation requirements. Pull the specific control families from your framework (NIST SP 800-53 for FedRAMP, DISA STIGs for DoD, NIST SP 800-171 for CMMC). Map each AI component to the controls it must satisfy.
3. Define your SBOM policy. Decide which SBOM format you will use (SPDX or CycloneDX), who generates it, and how it is stored and compared at each ingest cycle.
4. Procure hardware. Size GPUs based on the model and QPS targets above. Confirm FIPS 140-2/140-3 validated storage encryption. Plan physical security for the enclave room.
5. Establish the internal PKI. Stand up an offline root CA and an intermediate CA for the enclave. Document the key ceremony and store root keys offline in hardware security modules (HSMs).
6. Define transfer authority roles. Assign named individuals as transfer authorities. Document their responsibilities, the media they are authorized to handle, and the escalation path if a transfer is rejected.

Phase 2: Staging and ingest runbook

1. Build the staging zone. Set up an internet-connected staging host that is physically and logically separate from the enclave. This machine downloads artifacts and performs initial scanning.
2. Download and verify artifacts. Pull model weights, container images, and packages. Verify each against the vendor-published checksum before proceeding.
3. Sign artifacts. Use an offline signing key held by the transfer authority. Record the signing key ID and timestamp in the chain-of-custody log.
4. Run malware scanning and SBOM diff. Scan every artifact. Compare the new SBOM against the previous version and document any new dependencies. Reject artifacts with unsigned or unexplained new dependencies.
5. Create the transfer bundle. Package signed artifacts with their checksums and SBOM into an encrypted transfer bundle. Record the bundle hash.
6. Execute the transfer. Move the bundle via data diode or vetted removable media per the written SOP. Log the media serial number, operator identity, and timestamp.
7. Verify inside the enclave. Re-verify the bundle hash and each artifact checksum after transfer. Confirm signatures against the enclave's trusted public key.
8. Record chain-of-custody evidence. File the complete transfer record in the ATO evidence package. This record is what auditors examine during an IL or FedRAMP review.

Pro Tip: Never activate a newly transferred artifact on the same day it arrives. Impose a mandatory 24-hour hold period during which a second operator independently verifies the checksums and SBOM diff. This two-person integrity check is a standard control in classified environments and catches errors that a single reviewer under time pressure will miss.

Phase 3: Operational runbook

1. Daily health checks. Verify that all enclave services are running, that the SIEM is collecting logs, and that no unexpected processes are active. Run the silence test (network capture during inference) weekly.
2. Access review. Review user access logs weekly. Flag any access outside normal working hours or from unexpected endpoints. Revoke access for departed personnel within one business day.
3. Incident response. Because the enclave has no outbound comms, incident response relies on local forensics. Maintain a local copy of your IR playbook. Isolate affected nodes by powering them down or disconnecting them from the internal enclave network. Preserve disk images before any remediation.
4. Disaster recovery. Maintain encrypted local backups of model weights, configuration, and data on separate storage within the enclave. Test restoration quarterly. Document the recovery time objective (RTO) and recovery point objective (RPO) in your DR plan.
5. Accreditation evidence collection. Before each periodic review, compile transfer records, access logs, SBOM diffs, and silence test results into the evidence package. Assign a named owner for this task.

For a deeper look at hardening the operational layer, the offline AI security guide covers practical controls for maintaining and updating isolated AI systems.

What do practitioners and auditors actually look for in air-gapped AI?

The gap between a compliant-looking deployment and one that passes an accreditation review is almost always in the runtime evidence, not the architecture diagrams.

Pre-review diagnostic checks

Before an accreditation review, run these checks yourself so the auditor does not find them first:

SBOM matching: Compare the SBOM of every running container against the SBOM that entered the enclave at last ingest. Any new package that appeared after ingest is a finding. This catches post-transfer modifications and supply-chain insertions.

CA chain inspection: Enumerate every certificate in the enclave's trust store. Any public CA (DigiCert, Let's Encrypt, or any commercial root) that is not explicitly required and documented is a finding. Auditors for classified workloads expect a trust store containing only the internal CA.

DNS and NTP sink tests: Configure a local DNS sinkhole and NTP server. Run a full inference workload and check whether any query reached the sinkhole or whether any process attempted to contact an external NTP server. Either result is a finding.

Egress enforcement validation: Attempt to initiate an outbound connection from inside the enclave to a known external IP. The connection must fail at the network layer, not just at the application layer. Application-layer controls alone do not satisfy auditors for high-classification environments.

Patterns that hold up under scrutiny

Treating compliance as an enforceable runtime property rather than a policy document is the single most important shift in mindset. Default-deny egress enforced at the kernel (iptables, nftables, or eBPF via Falco or Tetragon) means a misconfigured application cannot phone home even if it tries. One-way ingest via data diode enforces the same principle at the physical layer.

Air-gapping eliminates whole attack classes but transfers the hardest problems to lifecycle management: packaging, signing, verifying, and moving artifacts securely into the enclave on a controlled cadence. Teams that underestimate this cost end up with stale models, broken ingest procedures, and audit findings that could have been avoided with upfront planning. Assign a named owner for lifecycle management before the first artifact enters the enclave.

Operational failures often come from hidden dependencies such as DNS, CA, and NTP calls, and from egress holes quietly punched to solve update or telemetry problems. The silence test described in the risks section is the objective check that validates runtime behavior rather than relying on configuration review alone.

Key Takeaways

A true air-gapped AI deployment requires physical network isolation, signed artifact ingest, enforced default-deny egress at the kernel layer, and continuous audit evidence collection to satisfy FedRAMP High, DoD IL5/IL6, and CMMC accreditation requirements.

The part most guides skip: air-gapping is a lifecycle commitment, not a one-time build

Most articles about air-gapped AI focus on the architecture diagram and the compliance checklist. Both matter. But the thing that actually determines whether a deployment survives its first annual review is something less glamorous: who owns the update cycle, and whether they have the time and authority to run it properly.

The architecture is the easy part. You can design a technically correct air-gapped enclave in a week. The hard part is the 36th month, when the team that built it has turned over, the transfer SOP is three versions out of date, and the ISSO who signed off on the original ATO package has moved to a different agency. That is when auditors find findings, and that is when the real cost of air-gapping becomes visible.

There is also a tendency in the industry to treat air-gapping as a binary choice: either you have a full enterprise enclave or you are using a cloud API with no privacy controls. That framing misses a large population of users who need provable local privacy without the operational burden of a classified enclave. A researcher handling sensitive interview data, a lawyer reviewing privileged documents, a small defense contractor who needs offline AI for CUI but does not have a dedicated ISSO. For those users, a lightweight offline desktop tool that runs entirely on their own machine is not a compromise. It is the right tool for the job.

The lesson from practitioners who have run air-gapped AI through multiple accreditation cycles is consistent: plan the lifecycle before you plan the architecture. Assign owners, define cadence, write the transfer SOP, and run the silence test before you call the environment production-ready. The auditor will ask for all of it.

Private offline AI without the enterprise overhead

For individuals and small teams who need AI data protection without standing up a full enterprise enclave, Greencube is a direct path to provable privacy. It runs entirely on your own Windows machine, with no cloud connection, no account creation, and no subscription. You get chat, PDF analysis, and image understanding, all processed locally, with nothing ever transmitted externally.

The difference from a cloud AI tool is concrete: your documents never leave your device. There is no vendor with access to your queries, no telemetry, and no dependency on an internet connection. For a lawyer reviewing privileged documents, a researcher handling sensitive data, or a contractor working with CUI who does not need a full DoD-accredited enclave, Greencube covers the privacy requirement at a fraction of the operational cost.

If your workload requires FedRAMP High, DoD IL5/IL6, or CMMC Level 3 accreditation, you need the full enterprise air-gapped architecture described in this guide. But if your requirement is simply that AI stays on your machine and never touches the cloud, Greencube is a one-time purchase that works immediately after installation, no model setup or technical configuration required. Visit greencube.app to see what it does and get started.

Useful sources for accreditation and further reading

These are the primary and practitioner sources most relevant for U.S. regulated environments. NIST publications and ISOO guidance are prescriptive mandates; the practitioner guides are useful for evidence package construction and architecture validation.

• The Strategic Necessity of Air-Gapped AI Systems (MIT DSpace) — Academic treatment of air-gapped AI as a strategic control; useful for framing in ATO narratives and executive briefings.
• NIST SP 800-53 (Security and Privacy Controls) — The prescriptive control catalog for FedRAMP and federal systems; map every enclave component to the relevant control families here.
• NIST SP 800-171 (Protecting CUI in Nonfederal Systems) — Required reading for CMMC Level 2 and Level 3 compliance; directly applicable to CUI handling in air-gapped AI deployments.
• FedRAMP Authorization Boundary Guidance — Defines what constitutes the authorization boundary; essential for scoping an air-gapped AI ATO package correctly.
• DISA Cloud Computing Security Requirements Guide (CC SRG) — Governs DoD IL tiers; specifies the physical and logical isolation requirements for IL5 and IL6 deployments.
• ISOO CUI Registry and Notices — The authoritative source for CUI categories and handling requirements; determines whether your data classification triggers an air-gap requirement.
• AI Index Report 2025 (Stanford HAI) — Broad context on AI adoption in government and regulated sectors; useful for executive briefings on why air-gapped AI is becoming a standard requirement.

NIST SP 800-53 and the DISA CC SRG are prescriptive mandates with specific control requirements. The MIT and Stanford sources are guidance and context, not mandates, but they carry weight in ATO narratives. The FedRAMP boundary guidance and ISOO CUI registry are the two sources most directly relevant to scoping and classification decisions.

FAQ

What is air gapping in AI?

Air gapping in AI means running an AI system inside a physically isolated network with no routable path to the public internet, so that models, data, and outputs never leave the organization's boundary. It removes entire attack classes by taking the system off the institutional network graph.

What does "air-gapped" mean in tech?

An air-gapped system has no physical or logical connection to any external network, including no external DNS resolution, no public certificate authority trust chains, and no outbound connections of any kind. The term comes from the literal "air gap" between the isolated system and any connected network.

What is the difference between air-gapped and offline?

"Offline" typically means a device or application is not currently connected to the internet but could be reconnected. "Air-gapped" means the system is physically and permanently isolated with no connection path available, a much stronger and auditable guarantee that accrediting authorities require for classified and CUI workloads.

Does Greencube qualify as air-gapped AI?

Greencube runs fully offline on a single machine with no cloud connection or telemetry, making it a practical privacy solution for individuals and small teams. It does not constitute an enterprise air-gapped enclave with the full accreditation controls (signed artifact ingest, internal PKI, SIEM) that FedRAMP High or DoD IL5 require.

When is a full enterprise air gap legally required?

A full enterprise air gap is typically required when data carries a CUI, SECRET, or higher classification, when a contract references FedRAMP High, DoD IL5/IL6, or CMMC Level 3, or when a security policy explicitly prohibits any outbound connectivity from the AI runtime. Consult your ISSO and the relevant accreditation body for a definitive determination.

Recommended

• Offline AI Security: A Practical Hardening Guide
• Local AI Models Explained: Privacy, Use Cases, How to Start
• GDPR Compliant AI: Your 2026 Compliance Roadmap
• Top 3 Offline AI Solutions: Ollama Alternatives 2026