From $26,400/mo Cloud API to $0 API Cost:
158 TPS on 2-Node Apple Silicon Ring

Built at 3AM because the cloud latency spike was bleeding my inference-research spend. The bill had been climbing for months; that month it peaked and was not coming down. The hardware to replace it was already on the desk. 55 commits ahead of upstream later: zero ongoing API spend at 389ms P50 end-to-end — self-hosted, so electrical draw and ops replace the invoice, not disappear entirely.

The Bill

At $868/day, the cloud provider was extracting $36.16/hr for compute that an Apple Silicon node delivers at electrical cost only: approximately $0.02/hr per node at full load (power draw; full owned-stack TCO also accounts for hardware amortization and ops).

The economics are stark: $868/day of managed-cloud rate versus marginal energy cost on owned hardware. The honest comparison is owned-stack TCO vs. cloud, modeled below, not power vs. invoice. The operational properties (data sovereignty, zero egress, on-premise audit trail, no rate limiting) are structurally unavailable on managed cloud inference. Most teams stay on cloud because building the distributed ring requires uncommon skills: tensor parallelism, Thunderbolt fabric management, and model sharding.

Measured Results

The Honest TCO

The "$0" in the headline is the owner's real API invoice, not a total-cost-zero claim. The hardware (M1 Max, M4 Mini) was already on the desk when the $26,400 invoice arrived. Sunk capital does not recur; what remains is electrical draw and ops overhead. Under 180W per node, that is a rounding error against $316,800/yr. For a team standing this up from scratch, based on published cloud-inference pricing at the time of this build and a mid-range hardware-amortization schedule, the delta lands in the 80–90% net TCO reduction range. The exact figure turns on traffic shape, hardware age, and ops headcount. A structural win. Worth stating precisely.

The right architecture depends on traffic shape. This configuration performs at steady-state and moderate burst, the regime where cloud inference is chronically over-purchased. If P99 burst exceeds P50 baseline by an order of magnitude, the correct posture is hybrid: own the baseload, rent the ceiling. I derived that crossover from my own ring's measured load curve, not a generic template. The same line-item method applies to any workload you can profile. Both the cloud invoice and the owned-stack cost are line-itemizable; the delta is large and it reproduces.

Architecture: The 2-Node Ring

The cluster runs on exo-explore/exo, a distributed inference framework that shards model weights across heterogeneous Apple Silicon nodes via Thunderbolt 4 fabric. Each node contributes its unified memory pool to a collective unified-memory budget. M1 Max contributes 32GB, M4 Mini 16GB: 48GB effective unified memory pooled across 2 nodes.

What Cloud Inference Cannot Offer

Three properties of the sovereign cluster are structurally impossible on cloud inference:

1. Zero data egress. Every prompt stays on-premise. For regulated workloads where data classification prohibits third-party processing (certain PHI under HIPAA, CUI under CMMC, or exchange-level transaction data under FINRA guidance), this is not a preference: it is a structural requirement. Cloud inference routes prompts through third-party infrastructure by definition.
2. Deterministic audit trail. The cluster produces a tamper-evident, SHA-256 hash-chained log of every inference request: model version, weight hash, input hash, output hash. This format is designed to satisfy the evidentiary requirements of a regulated audit. Cloud providers cannot meet this standard; they do not expose equivalent per-inference logs.
3. No rate limiting. Cloud inference providers throttle at burst capacity. The sovereign cluster runs at hardware limits, not provider policy limits. For time-critical workloads (forensic replay, real-time adversarial validation), latency predictability matters more than average throughput.

The Commit Trace

55 commits ahead of upstream on the production fork (mpuodziukas-labs/exo) cover: Thunderbolt fabric stability under sustained load, layer-parallel scheduling for heterogeneous node memory configurations, health-monitor integration for automatic failover to the primary node, and cache-reuse scheduling for high-frequency query patterns.

The work is open source. The architecture is reproducible.github.com/mpuodziukas-labs/exo ↗