512GB for Local AI: VRAM, KV Cache and Full-Context Inference

The central engineering problem is not “How do I buy 512GB?” It is “What combination of model, precision, context, concurrency, and latency can I support?” A 512GB unified-memory computer, four networked nodes, and sixteen older accelerators are different systems even if their advertised capacities add up to the same number. Memory location and movement can matter as much as capacity.

1. “512GB” is not a usable inference budget

First distinguish units. Decimal GB means 10⁹ bytes; binary GiB means 2³⁰ bytes. A hypothetical capacity specified as exactly 512 billion bytes is about 476.84GiB. Vendors and operating systems do not always use labels consistently, so check the runtime's reported bytes rather than assuming either interpretation of a marketing label.

M_required = M_weights + M_KV + M_workspace
           + M_runtime + M_communication + M_other_resident

M_required must fit the memory actually available to the process,
on every participating device or node, with operating headroom.

Workspace includes temporary tensors, attention kernels, and graph-capture allocations. Runtime overhead includes allocators and framework state. Distributed jobs need communication buffers; unified-memory systems also share capacity with the CPU, operating system, and other applications. Reserved cache blocks and fragmentation can make allocation fail before an elementary byte sum reaches the installed capacity.

A fixed “use 90%” rule is not portable. One configuration may reserve workspace up front while another grows it during prefill. Measure the peak during startup, long-prompt ingestion, concurrent decode, and cancellation or cache eviction. Do not treat swap as equivalent to additional fast model memory.

2. Model weights: stored parameters are not active parameters

A first approximation for weight storage is parameters × bits_per_weight / 8. A hypothetical 400-billion-parameter checkpoint at exactly four bits per parameter has a raw payload of 200GB, or 186.26GiB. That excludes quantization scales, zero points, metadata, tensors retained at higher precision, padding, and runtime-specific packing. Inspect the actual converted artifact and loaded allocation rather than using the nominal quantization name as an exact compression ratio.

A mixture-of-experts (MoE) model routes each token through a subset of experts. Its active parameter count helps explain compute cost, not automatically storage cost: unselected experts still need to be available when routing changes. A model with many stored parameters and relatively few active parameters is not a model whose inactive weights disappear. CPU, SSD, or distributed expert placement requires explicit runtime support and can introduce significant transfer latency.

Precision changes also affect quality and kernel compatibility. Weight-only four-bit quantization does not imply a four-bit KV cache; those are separate settings. A format that saves disk space may be unpacked or converted on load. Compare the exact model revision, tokenizer, quantizer, runtime, and kernel path, and include draft-model memory if speculative decoding is enabled.

For the mathematical background on quantization, see our SimpleCNN and CIFAR-10 walkthrough. Vision-model PTQ and large-language-model serving have different operator and quality considerations, but both require validation of the deployed representation.

3. KV cache: the memory that grows with the conversation

During autoregressive inference, attention reuses previously computed keys and values instead of rebuilding them for every new token. The key-value cache persists across the sequence. For an ordinary decoder with uniform full-attention layers, its logical size is approximately:

KV_bytes = 2 × layers × KV_heads × head_dimension
           × bytes_per_cache_element × sum(sequence_lengths)

For equal-length requests:
sum(sequence_lengths) = concurrent_sequences × tokens_per_sequence

The factor of two represents keys and values. Use KV heads, not automatically query heads: grouped-query attention (GQA) shares keys and values across multiple query heads. Multi-query attention can reduce that count further. FP16 and BF16 ordinarily use two bytes per element; a supported FP8 cache uses one byte per element before scale and allocation overhead. Beam search, duplicated cache state, and implementation-specific layout can alter physical memory.

A worked example: 80 layers, 8 KV heads, head size 128

Bytes per token = 2 × 80 × 8 × 128 × 2
                = 327,680 bytes = 320 KiB

One sequence at 131,072 tokens:
327,680 × 131,072 / 2^30 = 40 GiB

Four independent sequences at that length:
4 × 40 GiB = 160 GiB

One sequence at 1,000,000 tokens:
327,680 × 1,000,000 / 2^30 ≈ 305.18 GiB

This is an illustrative architecture, not a claim about every large model. The difference between one million tokens and 1,048,576 tokens also matters: “1M” and “1Mi” are not interchangeable. Budget prompt tokens plus generated tokens; if a runtime's context limit is 131,072 tokens, a prompt of that length leaves no room for a nonempty answer under that limit.

Sliding-window layers retain only a bounded history in some implementations; hybrid models mix attention types and state-space layers. Multi-head latent attention (MLA) can cache a compressed latent representation rather than the full conventional K/V tensors. These require their own formulas and runtime inspection. Do not paste the standard GQA calculation onto every architecture.

Apply the formula to the actual model

Establish the effective bytes per token from the model's architecture and serving implementation rather than borrowing a number from another model. Recompute the weight format, cache format, units, concurrent sequences, and overhead from revision-pinned sources. A context limit in a model card specifies a supported configuration, not a promise that your hardware can serve it at useful latency or quality.

4. A runnable memory-planning lab

Save the following as memory_budget.py and run python memory_budget.py. It uses only the Python standard library. The example assumes 512GiB actually available in the chosen memory pool, a 300GiB loaded weight allocation, 32GiB of measured or estimated overhead, and a separate 32GiB safety reserve. These are hypothetical inputs, not a benchmark or a vendor's default allocation policy.

from dataclasses import dataclass
from math import isfinite

GiB = 2 ** 30

@dataclass(frozen=True)
class Attention:
    layers: int = 80
    kv_heads: int = 8
    head_dim: int = 128
    element_bytes: int = 2

    def bytes_per_token(self):
        values = (self.layers, self.kv_heads,
                  self.head_dim, self.element_bytes)
        if any(type(value) is not int or value <= 0 for value in values):
            raise ValueError("Attention dimensions must be positive integers")
        return 2 * self.layers * self.kv_heads * self.head_dim * self.element_bytes

def budget(attention, *, total_gib, weights_gib, overhead_gib,
           reserve_gib, tokens, sequences):
    sizes = (total_gib, weights_gib, overhead_gib, reserve_gib)
    if any(not isfinite(value) or value < 0 for value in sizes):
        raise ValueError("Memory sizes must be finite and nonnegative")
    if total_gib <= 0:
        raise ValueError("Total memory must be positive")
    if any(type(value) is not int or value <= 0
           for value in (tokens, sequences)):
        raise ValueError("Tokens and sequences must be positive integers")

    per_token = attention.bytes_per_token()
    fixed = (weights_gib + overhead_gib + reserve_gib) * GiB
    cache_bytes = per_token * tokens * sequences
    available = total_gib * GiB
    return {
        "cache_gib": cache_bytes / GiB,
        "required_gib": (fixed + cache_bytes) / GiB,
        "fits": fixed + cache_bytes <= available,
        "max_equal_length_sequences": max(
            0, int((available - fixed) // (per_token * tokens))
        ),
    }

if __name__ == "__main__":
    attention = Attention()
    settings = dict(
        total_gib=512, weights_gib=300,
        overhead_gib=32, reserve_gib=32, tokens=131072,
    )
    print("bytes_per_token:", attention.bytes_per_token())
    for count in (1, 3, 4):
        print(count, budget(attention, sequences=count, **settings))

bytes_per_token: 327680
1 {'cache_gib': 40.0, 'required_gib': 404.0, 'fits': True, 'max_equal_length_sequences': 3}
3 {'cache_gib': 120.0, 'required_gib': 484.0, 'fits': True, 'max_equal_length_sequences': 3}
4 {'cache_gib': 160.0, 'required_gib': 524.0, 'fits': False, 'max_equal_length_sequences': 3}

Change element_bytes to 1 to model the idealized storage payload of an eight-bit cache. In this example the logical cache halves to 20GiB per sequence and the arithmetic ceiling becomes seven equal-length sequences. That is a sizing scenario, not a throughput guarantee or a claim of equal model quality. The calculator omits block padding and architecture-specific states; for a cluster, repeat the budget per rank using the actual sharding or replication of weights and cache.

5. Compressed attention is not the same as putting a model on an SSD

Three mechanisms are easily confused: lower-precision cache storage, architecture-level compressed attention, and moving model-related data to a slower storage tier. They solve different problems and have different correctness and latency implications.

The linked Mia'a AI Lab deployment report describes native low-precision weights and NVMe-resident Engram lookup tables for DeepSeek V4.1 Flash. It reports approximately 189GiB of tables on NVMe and about 305GiB of GPU-resident data in that configuration. These are the maintainer's measurements, not figures independently reproduced by PSC. The report also distinguishes checkpoint size, resident allocations, and layouts across different rank counts; those are not necessarily directly subtractable quantities.

An architecture-specific lookup table can have a sparse access pattern suitable for that runtime's SSD-backed implementation. That does not mean that an arbitrary 189GiB of dense transformer weights or KV cache can be moved to SSD at negligible cost. Dense per-token weight access, random lookup reads, and cache paging have different working sets. Validate SSD latency, IOPS, page-cache behavior, and sustained operation under memory pressure.

The same report warns that putting the tables in host RAM can evict the model on unified-memory systems. “Offload to RAM” is not a capacity win when RAM and GPU allocations consume the same physical pool. Record what is actually resident, replicated, mapped, and pageable rather than merely listing where the file lives.

6. Six hardware approaches, six different bottlenecks

The following compares six hardware approaches, not an endorsement or a universal ranking. Vendor specifications were checked on 20 October 2026 where accessible; stock, regional pricing, supported configurations, and runtimes still need a fresh check before purchase.

Mac Studio: one large pool is simpler, but not unrestricted

Apple's current specification page lists M5 Ultra configurations up to 512GB and 1.2TB/s memory bandwidth; its configuration rules attach the highest memory option to the higher-core-count variant. Those are vendor specifications, not measured LLM speeds. A spec listing does not itself confirm a purchasable configuration in your region; verify availability and pricing directly.

Do not hard-code a universal macOS percentage as usable GPU memory. Query what the chosen framework can allocate, observe memory pressure during long-context runs, and leave capacity for the OS. An unsupported limit override can destabilize the machine and does not increase physical bandwidth. Two Macs connected by Thunderbolt still require a distributed implementation; the cable does not merge their memory spaces.

DGX Spark: coherent memory within a node, networking between nodes

NVIDIA's Spark hardware documentation lists a 128GB configuration, 273GB/s memory bandwidth, and ConnectX-7 networking. The product page also lists other memory options, so verify the SKU. Four 128GB nodes make the headline sum, but memory occupied by the head node's API server, tokenizer, communication buffers, and CPU tasks is unavailable to its model shard.

A vendor's advertised FP4 compute figure is not tokens per second. Precision, sparsity assumptions, attention kernels, memory traffic, and communication determine the application result. Confirm the exact model and runtime support before treating an aggregate capacity claim as a working configuration.

RTX PRO 6000: faster local memory may beat a larger pool

NVIDIA lists 96GB GDDR7 for RTX PRO 6000 Blackwell. Four such devices add up to 384GB. Distinguish Max-Q and Workstation Edition variants; do not apply one variant's power and cooling requirements to every board. A four-card workstation also needs appropriate electrical service, PSU capacity, airflow, slot spacing, and enough usable PCIe lanes. Do not assume a particular interconnect or peer-to-peer feature from the family name.

EPYC, MI50, and mini-PC builds: cheap capacity can charge in latency

A many-channel EPYC system can offer substantial DDR capacity, but population matters: filling only a few channels or accessing a remote NUMA node can waste much of the theoretical bandwidth. A historical build price is not a current bill of materials; reprice compatible DIMMs, motherboard, CPUs, cooling, storage, and any accelerators.

Sixteen older MI50 cards add compatibility and integration risks. Check the exact GPU generation against the runtime, driver, and kernel support matrix; successful community workarounds are not equivalent to current vendor support. Include risers, PCIe switches, power distribution, fans, spare parts, and failure diagnosis. Two compact Strix Halo systems reduce chassis complexity but still face inter-node bandwidth and supported distributed-execution constraints. Neither approach earns a recommendation just by reaching a memory total.

7. Distributed inference: memory does not add itself

Tensor parallelism divides computations within layers and typically needs frequent collectives. Pipeline parallelism assigns stages to devices and transfers intermediate activations between them; bubbles and uneven stages affect utilization. Expert parallelism distributes MoE experts and adds token-routing traffic. Data parallelism ordinarily replicates a complete serving instance, increasing throughput rather than making one oversized instance fit.

Check the backend's sharding rules. Fewer KV heads than tensor-parallel ranks may require replication, and some tensors or stages remain uneven. Adding ranks can therefore reduce weight pressure without dividing all cache or overhead equally. vLLM's parallelism guide discusses supported layouts and the runtime's cache-capacity logs; those measured allocations are more useful than assuming perfect division.

Keep bandwidth units consistent. A 200Gbit/s link has a raw arithmetic ceiling of 25GB/s before protocol and software overhead. It is not a 200GB/s memory bus. The 273GB/s Spark memory specification and a network link describe different operations; neither can simply replace the other. Check topology, message sizes, latency, link negotiation, and actual collective performance.

8. Prefill, decode, prefix caching, and quantized cache

Prefill processes the input prompt and constructs its cache. Decode generates new tokens while reusing cached state. Long prompts stress attention, compute, and cache allocation; low-concurrency decoding often puts more pressure on weight movement. Architecture, batching, speculative decoding, and kernels can change which bottleneck dominates.

Approximate cold prompt processing time = input_tokens / measured_prefill_tokens_per_second
Approximate output time = output_tokens / measured_decode_tokens_per_second

End-to-end time also includes queueing, tokenization, transfers,
startup or compilation, and any cache misses or storage reads.

At a sustained 5,000 input tokens/s, a one-million-token cold prompt takes about 200 seconds to process. At 18,000 input tokens/s, the corresponding arithmetic is about 55.6 seconds. These are conditional examples, not extrapolated benchmark promises: attention cost, chunking, cache capacity, and throttling can change throughput at extreme lengths. A short-prompt decode number cannot predict long-context time to first token.

Prefix caching can reuse an already-computed, matching token prefix. It does not eliminate the first cold ingestion, and changed tokens earlier in the prefix can reduce reuse. Cache eviction, tokenizer or template differences, model revision, and tenant-isolation rules affect hit rate. vLLM documents APC as a prefill optimization, not a general acceleration of every newly generated token.

KV-cache quantization changes cache storage precision. It can increase capacity, but speed and quality depend on hardware and kernels. vLLM's documentation describes supported formats and calibration schemes. Do not assume all architectures, layers, or devices support the same mode. Compare retrieval, long-horizon reasoning, tool use, and coding tasks with a higher-precision baseline before approving a smaller cache.

A defensible cache-quality comparison holds checkpoint, prompts, tool environment, decoding settings, and evaluation protocol fixed and changes only the cache setting. Report the task-level results and their variation rather than assuming a universal quality penalty for FP8.

9. Separate measured evidence from shopping numbers

The linked Mia'a AI Lab report gives a four-Spark single-stream decode result of 87.7 tokens/s and a 16-stream aggregate result of 342.7 tokens/s for its setup. Those are community-maintained deployment results using particular low-precision kernels and speculative decoding, not vendor guarantees. Aggregate throughput does not mean each of 16 users gets 342.7 tokens/s; individual rates and scheduling fairness must be measured separately.

The report also describes a verified prompt of roughly 1.01 million tokens and cold-prefill measurements at several shorter lengths. A successful long-context needle test demonstrates something useful but narrower than complete long-context reliability. It does not establish that every workload, every query position, and every concurrent session behaves equally well.

The local-inference-lab four-RTX-PRO deployment report describes a separate configuration. Compare its exact memory tiering, speculative settings, kernels, and benchmark revision before ranking it against Sparks. A headline result with tables in RAM cannot silently be assigned to a disk-backed deployment. Do not copy permissive container settings such as host networking or a disabled seccomp profile without reviewing why they are required and how the environment is isolated.

A useful measurement matrix

1. Pin model and tokenizer revisions, quantization format, draft model, runtime commit, drivers, kernels, and hardware topology.
2. Test cold inputs at 4K, 32K, and 128K, then higher lengths only if required. Count tokens with the deployed tokenizer; reserve output capacity.
3. Repeat at concurrency 1, 4, and the expected peak. Use both independent prompts and realistic shared-prefix traffic.
4. Record p50/p95 time to first token, inter-token latency, completed-request latency, per-request output rate, total throughput, errors, and cancellations.
5. Track peak per-rank memory, CPU pressure, network throughput, SSD reads, wall power, and thermals after warm-up and at sustained load.
6. Evaluate answer quality at multiple context positions and lengths. Keep quality failures visible alongside speed and capacity.

“Full context” should mean the complete tested workload fits and meets its quality and latency targets, not merely that a maximum-length command-line option was accepted.

10. Local versus API: a transparent break-even model

Hardware and API pricing change over time; the worked scenario below uses illustrative assumptions, not current quotes. Region, tax, model availability, cache discounts, and contractual data terms must be checked separately. A useful comparison starts with the same task quality and an actual utilization trace.

Monthly_local_cost =
    (hardware_cost - expected_resale_value) / useful_life_months
    + electricity + cooling + administration + maintenance + network/storage

Monthly_API_cost =
    uncached_input_millions × uncached_input_price
    + cached_input_millions × cached_input_price
    + output_millions × output_price
    + other provider or platform charges

As an illustrative scenario only, suppose hardware costs $28,000, useful life is 36 months with zero assumed resale, wall power averages 0.6kW over 720 hours/month, electricity is $0.25/kWh, and administration plus maintenance is $150/month. Amortized hardware is $777.78/month, electricity is $108/month, and the subtotal is $1,035.78/month before cooling and other costs.

For a hypothetical API at $0.50 per million input tokens and $2.00 per million output tokens, 100 million input plus 20 million output tokens cost $90/month. With all else held fixed, roughly 11.51 times that token volume matches the illustrative local monthly subtotal. That does not prove the local box can produce the corresponding workload at acceptable latency.

For a separate cash-payback view, divide upfront hardware cost by monthly net avoided API cost after recurring local expenses. If net savings are zero or negative, there is no positive payback period under those assumptions. Do not count hardware amortization again in that denominator. Tokens per second multiplied by every second of the year is installed capacity, not demonstrated demand.

Self-hosting can still be justified by offline availability, predictable access, data governance, or workflow control. An API can be preferable for bursty workloads or rapidly changing models. Include licensing, regional availability, redundancy, spare capacity, and operator time. Cheap hardware with expensive integration can lose to a smaller supported deployment.

11. Validate and secure the final system

Local inference keeps data on infrastructure you control only if the surrounding system is configured that way. Runtime downloads, telemetry, connectors, tools, logs, and remote management can still transmit data. The model endpoint is also a resource-intensive service that needs ordinary production controls.

1. Inventory before loading. Record model source, license, artifact digests, runtime and container versions, driver support, and conversion provenance. Review custom model code rather than enabling trust flags automatically.
2. Isolate management and fabric. Keep model-serving control planes, distributed-worker ports, RDMA networks, dashboards, and storage services on restricted networks. An OpenAI-compatible API shape does not provide authentication by itself.
3. Authenticate and bound inference. Enforce caller identity, authorization, per-tenant quotas, maximum input and output tokens, concurrency, queue limits, and timeouts. Reject oversized workloads before expensive prefill.
4. Protect cache and logs. Treat cached prompts, KV state, spill files, prefixes, and debug traces as potentially sensitive. Apply tenant isolation, retention limits, encryption where appropriate, and supported deletion procedures.
5. Contain tool use. A locally hosted model is not an authorization engine. Tool adapters need least privilege, network egress controls, explicit permissions, and independent validation of model-proposed actions.
6. Exercise failures. Test cancellation, one rank failing, SSD unavailability, cache eviction, OOM, restart, and backend fallback. A degraded CPU path should be observable rather than silently violating service targets.
7. Promote only measured configurations. Archive the actual benchmark report and peak memory, then use controlled rollout, health checks, capacity alerts, and a tested rollback path.

12. Sources, verification limits, and final checklist

Source review date: 20 October 2026. Product pages and community repositories are mutable. Primary documentation supports the architectural discussion; externally reported benchmarks are attributed to their maintainers. This article is not an independent hardware benchmark, a live price list, or a claim of affiliation with the vendors.

1. Apple Mac Studio specifications — memory configurations and advertised bandwidth, not an application benchmark.
2. NVIDIA DGX Spark hardware overview and product specifications — distinguish the selected memory SKU and network from the application result.
3. NVIDIA RTX PRO 6000 Blackwell — verify the exact board variant and installation requirements.
4. Mia'a AI Lab: DeepSeek V4.1 Flash on DGX Sparks — maintainer-reported deployment, memory placement, context tests, and throughput.
5. local-inference-lab: four-RTX-PRO deployment — a separate runtime configuration; benchmark storage mode must match deployment mode.
6. vLLM parallelism and scaling, automatic prefix caching, and quantized KV cache — select documentation matching your installed release; the last link is the developer-preview guide.

Before buying: confirm units and usable memory; calculate weight and cache allocations; validate per-rank fit; price the complete system; measure cold and warm workloads; test full-context quality; and review the deployment's data and administrative boundaries. Memory is one line in that checklist, not its conclusion.