Know what your AI factory will cost before you buy it.

AI Engineer uses a first-order roofline model: the reasoning an infrastructure architect would do on a whiteboard, applied the same way to every platform so the comparison is fair.

Memory

Weights take parameters times bytes per parameter (2 for BF16, 1 for FP8, about 0.56 for FP4 including scales). Each request holds a KV cache of 2 times layers times KV heads times head size times bytes per value, for every token in its context. Models with compressed attention, such as DeepSeek V3 and Kimi K2, use their latent width instead, and sliding-window layers only keep their window. A replica uses 90% of GPU memory, keeps 2 GB per GPU for activations, and gives the rest to the KV cache.

Speed

Each decode step reads the weights once plus the KV cache of every request in the batch, and does about 2 FLOPs per active parameter per token. Step time is the larger of memory time and compute time, divided by a parallelism efficiency and, where one is set, the platform's software efficiency derived from measured throughput, plus a fixed scheduling overhead. For mixture-of-experts models the share of experts read in a step grows with the batch. Prompt processing is compute bound and interrupts decoding, so the speed each request sees is:

Time to first token assumes an idle server: the larger of prompt compute and one read of the weights per 8,192-token prefill chunk, so short prompts on large mixture-of-experts models are bound by memory bandwidth.

Sizing

For each tensor-parallel width AI Engineer finds the largest batch that fits in memory and still meets your speed target. To own, it keeps the width with the lowest yearly fixed cost of the servers or racks you would buy (depreciation, colocation and support), then fewer units, fewer GPUs and a narrower width. Energy is left out of that choice because this first-order model derives utilisation from capacity, which would make wider and less efficient layouts look cheaper. Renting is sized separately on the GPUs in use, so the rented and owned layouts can differ. Servers and NVL72 racks are bought in whole units, so spare GPUs show as headroom.

Cost

Owning is hardware plus networking, storage and integration, amortised straight line, plus colocation per kW of IT capacity, metered energy times PUE, support, and any staff cost you enter. Renting is the GPUs in use times the hourly rate for every hour of the year, less any commitment discount. The API figure prices your yearly input and output tokens at list price. Yearly output tokens are peak concurrency times average load times target speed times seconds in a year.

Limits

This is a planning estimate, not a benchmark. Real throughput depends on the serving stack, scheduler settings, quantisation, prompt caching and network design, and can differ from these figures by 30% or more. It is calibrated against MLPerf Inference results for Llama 70B only, so treat mixture-of-experts and multi-server results as indicative. For MI300X, MI325X and MI355X the model applies a software efficiency of 68%, 67% and 60% respectively, derived from their throughput measured against NVIDIA on Llama 3.3 70B; the same factor is used on every serving profile. Other models and serving software can differ. The three software profiles (conservative, typical, optimised) assume 50%, 65% and 75% of peak memory bandwidth. Benchmark the shortlisted platform with your own prompts, for example with NVIDIA AIPerf or GenAI-Perf, before committing capital.