In brief
Running an open model such as DeepSeek, Qwen, GLM, or Llama on premises requires five decisions in order: the exact checkpoint, serving precision, GPU-memory budget, serving framework, and acceptance test. Hardware follows from those choices. The capacity examples are point-in-time, weight-only arithmetic; procurement still requires the selected checkpoint, software release, and production request profile to be measured together.
Key takeaways
- Choose the exact model checkpoint before choosing hardware; family names alone do not define a memory requirement.
- Treat precision as a quality, capacity, and speed decision that must be evaluated on the organisation's own tasks.
- Size weights, KV cache, runtime overhead, and operational headroom together rather than checking weight size alone.
- Pin the serving framework and version because scheduler and memory-management changes can move measured results.
- Accept the system on sustained throughput and p95 latency under a representative request profile, not a generic token-rate claim.
Running an open-weight model locally is not a single hardware question. It is a sequence of model, quality, capacity, software, and service-level decisions. Starting at the server catalogue reverses that sequence and often produces either expensive unused headroom or a system that cannot sustain the required context length and concurrency.
The five steps below are deliberately checkpoint-led. Model families evolve, vendors release new variants, framework support changes, and published parameter counts can be revised. Every number in a planning worksheet should therefore carry a checkpoint identifier, an observation date, and a validation owner.
Step 1: Pick the Exact Model Checkpoint
“We will run DeepSeek” or “we will self-host Llama” is a direction, not a deployable specification. Each family can contain flagship mixture-of-experts models, dense mid-sized models, distilled checkpoints, instruction-tuned variants, and multiple context configurations.
Start with the workload:
- Gather representative prompts, documents, expected outputs, and failure cases.
- Define quality criteria for factuality, retrieval grounding, coding, structured output, safety, and latency.
- Shortlist two or three exact checkpoints whose published capability appears relevant.
- Test those checkpoints on the organisation’s tasks.
- Record the model-card revision, licence, allowed use, and any redistribution or scale conditions.
The smallest checkpoint that clears the quality bar can reduce accelerator, power, and operational requirements substantially. If only the largest checkpoint clears it, the team should know before procurement.
Open weights do not imply one common licence. Some checkpoints use permissive terms, others use community or bespoke licences, and terms can vary within a family. Licence review belongs beside task evaluation in the decision record. This guide is not legal advice; the accountable team should review the actual terms for the selected checkpoint.
Step 2: Choose Serving Precision Deliberately
Precision connects output quality, memory footprint, memory traffic, and throughput. As a first-order planning approximation, BF16 weights use about two bytes per parameter and FP8 weights about one byte per parameter. Real runtime use is higher because formats, metadata, temporary buffers, framework allocation, and cache design differ.
Lower-bit quantisation can reduce the footprint further, but quality impact is task-dependent. Reasoning, code, extraction, and high-stakes drafting may react differently from casual chat. Hardware support also does not guarantee that a particular framework release implements a format efficiently for a particular checkpoint.
A defensible precision decision records:
| Decision input | Evidence to retain |
|---|---|
| Candidate precision | BF16, FP8, or the exact quantisation method and format |
| Quality comparison | Results on representative tasks, including known hard cases |
| Memory result | Measured loaded weights, cache allocation, and runtime overhead |
| Framework support | Hardware, kernel, and serving-framework versions tested |
| Refresh condition | Trigger for revalidation after model or software upgrades |
FP8 can be a useful starting point for large-model serving when the checkpoint, accelerator, and framework support it, but it is not an unconditional default. Use the highest-capacity format that satisfies both the quality requirement and the measured service target.
Step 3: Build the Complete Memory Budget
With checkpoint and precision fixed, estimate four memory pools:
- Model weights, based on the exact checkpoint and representation.
- KV cache, based on model architecture, context lengths, concurrency, and cache datatype.
- Runtime overhead, including framework buffers, graph capture, temporary tensors, and fragmentation.
- Operating headroom, including co-resident models, rolling replacement, unexpected context growth, and safe failure behaviour.
Mixture-of-experts checkpoints need special care. Only a subset of experts may be active for one token, but the serving system normally must keep all required expert weights resident or provide an explicitly designed offload path. Active parameter counts therefore cannot be substituted casually for resident weight capacity.
The point-in-time screen below was verified against official model cards and AMD specifications on 5 August 2026. It uses decimal units and a deliberately simple approximation of one byte per declared parameter for checkpoints published in FP8. The subtraction is only weight arithmetic: checkpoint files, non-FP8 tensors, KV cache, framework buffers, graph capture, fragmentation, and operating headroom all consume additional memory.
| Exact checkpoint | Official disclosure and licence | Weight-only arithmetic | 8 × MI355X: 2,304 GB aggregate HBM | 8 × MI300X: 1,536 GB aggregate HBM |
|---|---|---|---|---|
| Qwen/Qwen3.5-397B-A17B-FP8 | 397B total, 17B active; FP8; Apache-2.0 | ≈397 GB | 2,304 − 397 = 1,907 GB before all other allocations | 1,536 − 397 = 1,139 GB before all other allocations |
| meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 | 400B total, 17B active; FP8; Llama 4 Community License | ≈400 GB | 2,304 − 400 = 1,904 GB before all other allocations | 1,536 − 400 = 1,136 GB before all other allocations |
| zai-org/GLM-5-FP8 | 744B total, 40B active; FP8; MIT | ≈744 GB | 2,304 − 744 = 1,560 GB before all other allocations | 1,536 − 744 = 792 GB before all other allocations |
| deepseek-ai/DeepSeek-V4-Pro-Base | 1.6T total, 49B active; FP8-mixed base checkpoint; MIT | ≈1,600 GB first-order screen | 2,304 − 1,600 = 704 GB before all other allocations | 1,536 − 1,600 = −64 GB before all other allocations |
DeepSeek’s instruction checkpoint is a separate sizing case: the official
DeepSeek-V4-Pro card describes FP4 expert parameters with most other
parameters in FP8. Its resident footprint cannot be derived by reusing the
FP8-base row; use the exact checkpoint shards and a measured runtime load.
Likewise, a positive subtraction above is not a claim that the checkpoint will
load or meet a service target.
Aggregate HBM across eight GPUs is not automatically one transparent memory pool. Tensor or expert parallelism, collective communication, placement, and framework support determine whether a checkpoint can use it effectively. Treat the table as a shortlist screen. A configuration becomes a fit only after the target checkpoint loads, retains required headroom, and meets the service target under test.
For the underlying weight and KV-cache method, see the GPU memory requirements guide. Use the inference benchmark guide to turn the shortlisted configuration into a reproducible latency and throughput test, and the private AI infrastructure guide to define the operational and governance boundary around it.
Step 4: Pin the Serving Framework and Version
Raw model weights do not serve users. The serving framework determines request scheduling, continuous batching, KV-cache management, tensor parallelism, quantisation kernels, metrics, and failure behaviour.
vLLM is a common production candidate. Its continuous batching and PagedAttention design address utilisation and KV-cache fragmentation. Other frameworks, including SGLang and vendor-specific stacks, may be relevant for a particular checkpoint or operational model.
Framework choice should pass four gates:
| Gate | Validation question |
|---|---|
| Compatibility | Does the exact release support the accelerator, checkpoint architecture, precision, and parallelism plan? |
| Reproducibility | Are framework, ROCm, driver, firmware, kernel, and key serving parameters pinned? |
| Observability | Can operators see queueing, cache use, request errors, GPU health, and model-load events? |
| Operations | Are upgrade, rollback, restart, model replacement, and support paths documented? |
On AMD Instinct platforms, confirm the supported ROCm and framework combination from current documentation. Performance and compatibility can move between releases, so an unversioned claim is not a reproducible deployment result.
Step 5: Validate on the Intended System
The first four steps produce a paper design. Validation turns it into an operational one.
Use the exact checkpoint, precision, framework, and hardware configuration. Replay a request profile that represents expected input length, output length, concurrency, arrival pattern, and retrieval behaviour. Fix the service target before optimising batching; otherwise a higher aggregate number can simply mean each user waited longer.
The minimum result set is:
- prompt-processing throughput and time to first token;
- sustained output token rate;
- p50 and p95 end-to-end latency;
- throughput and latency at planned and stress concurrency;
- error, timeout, and queue behaviour;
- accelerator memory, utilisation, thermals, and health;
- storage and network behaviour during model load and RAG traffic;
- exact software and test conditions needed to reproduce the run.
Run longer than a single warm burst. Repeat tests, report ranges, and include a stress point above planned concurrency. Refresh the result when the checkpoint, precision, framework, driver, firmware, or material request distribution changes.
For broader infrastructure dependencies, see the network and storage planning guide and the private inference infrastructure analysis.
Reference Single-Node Architecture
The reference architecture joins the five sizing decisions to the surrounding network, storage, ingress, observability, and site responsibilities.
Figure 1 — A deployable on-premises model service includes the compute node and the infrastructure required to load, expose, observe, and sustain it.
| Layer | Reference component | Responsibility | Acceptance evidence |
|---|---|---|---|
| Compute | One 8-GPU inference node sized from the selected checkpoint | Load and serve models with required cache and runtime headroom | Model loads cleanly; memory budget and stress behaviour recorded |
| Network | Workload-appropriate Ethernet between compute, storage, ingress, and management | Carry model loading, retrieval, service, and telemetry traffic without hidden contention | Link, latency, loss, congestion, and failover results |
| Storage | NVMe tiers for model files, vector data, documents, logs, and evaluation data | Meet capacity, throughput, IOPS, endurance, protection, and recovery needs | Load-time and retrieval tests plus restore evidence |
| Ingress | Authenticated API gateway or service endpoint | Provide stable access, routing, policy, rate control, and audit boundaries | Access-control, rate-limit, timeout, and error-path tests |
| Observability | Request, model, GPU, network, and storage telemetry | Explain performance and support capacity and incident decisions | Dashboards, alerts, ownership, retention, and escalation tested |
| Site | Rack, power, cooling, cabling, and service access | Sustain the selected configuration safely | Approved site survey and operating limits |
Link speed and cooling method must follow the validated configuration. A high-density design may justify 100G, 200G, or 400G connectivity and may require direct liquid cooling, while a smaller deployment may not. Likewise, total system power cannot be inferred safely from accelerator count alone. Use the current integrated-system specification and a site survey rather than a generic article value.
An 8-GPU platform such as the xSONiC AI Inference Server can form the compute layer, but the deployable service also needs the remaining rows of the matrix.
Where This Guide Stops
If evaluation shows that only a closed frontier model meets the quality requirement, an on-premises open-model design is not a substitute. That workload may belong on an external API or in a hybrid architecture.
If demand remains exploratory or highly variable, rented capacity can be the more responsible way to learn before purchasing hardware. If there is no operational owner for patching, monitoring, capacity, security, and recovery, the programme is not ready merely because a checkpoint loaded successfully.
The durable sequence is checkpoint, precision, memory, framework, and validation. Each step constrains the next, and time-sensitive model, licence, software, and performance evidence must be revalidated when any selected component changes.
How to
How to size an on-premises open-LLM deployment
A five-step sequence from checkpoint selection to measured acceptance.
Pick the exact model checkpoint
Shortlist checkpoints, test task quality, and review the applicable licence before sizing hardware.
Choose serving precision deliberately
Compare quality, memory footprint, and speed at the precisions supported by the checkpoint and serving stack.
Build the complete memory budget
Estimate weights, KV cache, runtime overhead, co-resident services, and operating headroom.
Pin the serving framework and version
Confirm hardware support and record the framework release and material scheduling and memory settings.
Validate on the intended system
Measure sustained throughput and p95 latency with representative prompts, outputs, concurrency, and failure conditions.
Product fit
Where xSONiC Fits
xSONiC combines an 8-GPU AI Inference Server with switching, NVMe storage, service ingress, observability, site-readiness review, and workload-led validation. The configuration is selected from the target checkpoints and request profile, then measured on the intended hardware before acceptance.
FAQ
Frequently asked questions
Which open model should an enterprise start with?
Start with the smallest checkpoint that passes evaluation on the organisation's own documents, prompts, and quality criteria. Shortlist two or three candidates, record their licences and serving requirements, and let task-level results select the model rather than assuming the largest checkpoint is necessary.
Do I need Kubernetes to run an on-premises LLM?
Not necessarily. A single inference node running a pinned serving framework behind an authenticated API gateway can be a valid production design. Orchestration becomes useful when multiple nodes, multiple models, placement constraints, or automated failover justify its operational complexity.
Can one server host multiple models?
Yes, when the aggregate weights, KV caches, runtime overhead, and failover headroom fit across the available accelerator memory. The serving layer can route by endpoint, but each combination should be load-tested because co-resident models compete for memory and bandwidth.
What does validated mean before go-live?
It means the exact checkpoint and precision were tested on the intended hardware and software stack using representative prompt lengths, output lengths, and concurrency. The report should include sustained throughput, time to first token, p50 and p95 latency, error behaviour, and the full test conditions.
Continue reading
Related articles
Buyer GuideGPU Memory Requirements for LLMs: A Practical Sizing GuideEstimate GPU memory for LLM inference from model weights, precision, KV cache, concurrency, serving overhead, and single-node versus multi-node deployment.Read article
Technical ArticleWhy Tokens-per-Second Benchmarks Mislead LLM Infrastructure BuyersA token-rate headline changes with model, precision, context, batching, framework, and latency. Learn how to benchmark an inference service honestly.Read article
Technical ArticleWhat Is Private AI Infrastructure? A Guide for EnterprisesPrivate AI infrastructure keeps models, prompts, documents, embeddings, and logs inside a controlled environment across compute, network, storage, and operations.Read article
Technical ArticleNetwork and Storage Planning for Private AI Inference Servers: A Practical Buyer GuideA practical buyer guide to network and storage planning for private AI inference servers in Australian data centers.Read article
Technical ArticlePrivate AI Inference Is Reshaping Data Center Networking - And Australian Enterprises Are Starting to Pay AttentionEngineering guide to private AI inference networking for Australian enterprises, covering SONiC, RoCE, NVMe-oF, GPUDirect Storage, telemetry, and sovereignty.Read article
Technical ArticleWhat a GPU Actually Does and Why Your Network Fabric Determines Its Real PerformanceEngineering guidance on a GPU Actually Does and Why Your Network Fabric Determines Its Real Performance for Australian network and security teams, covering traffic.Read articleRelevant hardware
Related products
nvme ssdXS-SSD-8118Z-U2Enterprise U.2 NVMe SSD with PCIe Gen4 x4, NVMe 1.4, up to 7.1 GB/s sequential read, and 1,000K IOPS for data center storage workloads.View product
datacenter aiXS-DC-32X400-SP-G232-port 400G spine/core switch for high-capacity data center fabrics and AI-ready backbones.View product