Designing enterprise on-premises client RAG with 1M tokens using DeepSeek-V4 in 2026

"We want to adopt ChatGPT Enterprise, but we cannot send confidential data outside our organization." "We want to run an LLM capable of handling 1 million tokens on premise." Since early 2026, inquiries seeking to run LLMs within internal environments rather than via cloud APIs have surged, particularly in finance, manufacturing, and the public sector. Driving this trend is DeepSeek's release of DeepSeek-V4, featuring a 1-million-token context that agents can utilize at a production-ready level. With its weights distributed as open source, organizations can now run it on their own GPUs in addition to accessing it via the web and APIs.

This article details design guidelines for custom enterprise RAG projects adopting DeepSeek-V4 and outlines phased migration steps from existing OpenAI and Anthropic API configurations.

Why "1-million-token context on premise" is proving effective for enterprises today

Previously, on-premise LLM options centered on Llama, Qwen, and Japanese models (such as LLM-jp), where context lengths of 32K to 128K were realistic. Meanwhile, enterprise RAG engagements frequently surface requirements like these:

DeepSeek-V4 emerged as a model designed to bridge this gap. In particular, it features long-context processing (with optimized positional encodings and attention mechanisms) ready for practical use by agents. What sets it apart from other models is its ability not just to pack in long contexts, but to reason across their contents.

Overall architecture — reference configuration for on-premise RAG

The architecture we standardize across client projects consists of the following five layers:

1. Data source layer — SharePoint / Confluence / core DBs / file servers
2. Ingestion layer — document parsing, metadata extraction, PII masking
3. Vector / search layer — pgvector / Weaviate / Elasticsearch, hybrid search
4. LLM layer (on-premise DeepSeek-V4) — vLLM / TensorRT-LLM, GPU clusters
5. Agent / UI layer — internal chat, Slack, operational web UI

Notably, supporting 1 million tokens shifts the boundary of responsibility between Layer 3 (search) and Layer 4 (LLM). Previously, the standard practice was to narrow results down to five items via search and feed them to the LLM. With 1 million tokens available, a design that broadly retrieves 100 items via search and lets the LLM make the evaluation becomes viable. This dramatically cuts search tuning costs.

5 design guidelines to master in client development

1. Size GPU clusters based on peak concurrent connections

Size production deployments of DeepSeek-V4 based on peak concurrent connections rather than average concurrent connections. Because LLMs hold GPU memory throughout the entire request, handling N concurrent requests requires N times the VRAM.

Enabling continuous batching in vLLM boosts perceived throughput by 3x to 5x.

2. Avoid designs that always fill the 1-million-token window

Using the full 1-million-token context on every request causes per-request latency and GPU occupancy time to explode. For custom development projects, we recommend the following layering:

• Standard queries: up to 32K
• Long-document summarization queries: up to 128K
• Large-scale cross-sectional analysis: 1 million tokens (queued as explicit background jobs)

The practical UX approach is to determine on the application side whether using 1 million tokens is necessary, notify the user that processing will take several minutes, and then execute the job.

3. Implement PII masking in two tiers: at ingestion and prior to LLM handover

LLMs protect PII only probabilistically. Mask data in two tiers: prior to vectorization and prior to LLM input.

def ingest_document(doc: Document):
    masked_text = pii_masker.mask(doc.text)  # 第1段
    chunks = chunker.split(masked_text)
    for chunk in chunks:
        vector = embedder.embed(chunk)
        vector_store.upsert(chunk, vector, metadata=doc.metadata)

def query_llm(question: str, retrieved_chunks: list[str]):
    safe_chunks = [pii_masker.mask(c) for c in retrieved_chunks]  # 第2段
    return llm.generate(prompt=build_prompt(question, safe_chunks))

While two tiers may appear redundant, the downstream tier is necessary to handle PII whose masking criteria change dynamically after search (e.g., adjusting masking rigor based on user role permissions).

4. Structure audit logs across four tiers: question, search results, LLM output, and source citation

To satisfy enterprise requirements, retain structured logs so the basis of every answer can be traced back to source document IDs.

- conversation_id ─┬─ question
                   ├─ retrieval (chunk_ids[], scores[])
                   ├─ llm_call (prompt_tokens, output_tokens)
                   └─ answer (citations[chunk_id, doc_id])

Establishing a structure that allows teams to trace why an answer reached a specific conclusion is a mandatory prerequisite when expanding from internal use to customer-facing applications.

5. Build evaluation sets across three axes: factuality, confidentiality, and refusal behavior

Evaluate on-premise LLMs beyond basic answer quality across the following three axes:

Integrate these evaluations into CI and run them without fail whenever updating models.

Phased migration from existing OpenAI / Anthropic configurations

For projects already running ChatGPT Enterprise or Claude for Work, a hybrid operational model is more practical than an immediate, wholesale replacement.

Step 1: Use case triage

• Retain cloud APIs: Low-confidentiality, latency-sensitive tasks (general Q&A, drafting)
• On-premise DeepSeek-V4: RAG containing confidential data, long contract analysis, source code review

Step 2: Parallel operational phase (3 to 6 months)

Allow both backends to be called from the same interface (internal chat UI) and route by use case.

Step 3: Operational measurement and cost comparison

Compare 3-month operational costs (API billing vs. GPU amortization) and response quality.

Step 4: Allocation optimization

Determine the optimal cloud-versus-on-premise allocation for each workflow and document it in governance policies.

Combining this with the operational design principles we detailed in Guide to Building Private LLMs with LLM-jp-4 makes evaluating a transition to DeepSeek-V4 straightforward.

Licensing and geopolitical considerations

When utilizing DeepSeek, always confirm the following items during initial client discovery:

• Model weight licensing terms: Commercial scope, modification rights, and redistribution permissions
• Training data provenance: Whether data provenance documentation is required in regulated industries
• Geopolitical risk: Client internal policies regarding the jurisdiction of the model provider
• Switchability to alternative models: Abstraction layers to prevent single-model lock-in

Particularly in large financial institutions and public sector entities, clients care more about whether the architecture can accommodate a different model in the future than about the specific model itself. In custom development, introduce a model abstraction layer early on to ensure seamless interchangeability with Llama, Qwen, or Japanese models.

The vendor selection criteria outlined in Enterprise MCP Governance Design apply directly to LLM vendor evaluations as well.

Conclusion

With the arrival of DeepSeek-V4, a 1-million-token context on premise has become a practical option. Here are five key takeaways for custom enterprise RAG projects:

1. Size GPUs based on peak concurrent connections
2. Treat 1-million-token requests as explicit background jobs
3. Mask PII in two tiers: at ingestion and before LLM handover
4. Structure audit logs across four tiers
5. Evaluate against three axes: factuality, confidentiality, and refusal behavior

At GleamHub, we are actively delivering enterprise RAG projects using open-source LLMs, including DeepSeek-V4. If you face confidential data constraints that rule out cloud LLMs or require 1-million-token-class document analysis, please feel free to reach out to us.