Fast serving
Streaming, batch, and structured decode on an OpenAI-compatible API. Sandbox keys for a prototype. Dedicated replicas when the p95 has a number attached.
Fine-tune and inference cloud
Meter the model onto the GPU.
Tundrel is production inference and the fine-tune path that feeds it. Open-source, custom, and adapted checkpoints on infrastructure tuned for latency, throughput, and utilization. A tundish meters the pour.
What you get
Serve and adapt on one path.
Fine-tunes
LoRA and full jobs that land on the same serving path. Checkpoints stay in your project. Training on customer data runs only with a written job.
Packing
The scheduler places work to raise GPU utilization. Multi-LoRA shares a GPU when the isolation story is safe. Idle trays get the next heat.
SLAs you write
Caster deployments take a latency budget, a QPS floor, and a region. Rahul and the runtime crew sign the page before the replica goes live.
Control plane
Packing, scaling, isolation.
Admission checks the key and the quota. Packing assigns the request to a GPU cell. The runtime applies cache, custom kernels, and speculative decode when the SLA gains. Isolation is a project fence on Spout, a reserved replica on Ladle, or a single-tenant cell and VPC on Caster.
Published lab note, 12 July 2026, us-east-1, H100 SXM. Llama 3.1 8B Instruct, fp16, 2048-token prompt, batch 32. Prefill 12,410 tok/s. Decode 186 tok/s per stream at batch 8. Eight LoRA adapters packed at 1.8 concurrent 8B replicas on one H100 with p95 TTFT 94 ms.
0
Prefill tok/s, 8B lab
0
Decode tok/s per stream
0
p95 TTFT, packed 8B
Runtime
Kernels written for the pour.
Cache
Prefix cache stays warm across a replica. Hit rate is a first-class metric on the dashboard, next to tokens and dollars.
Decode
Speculative decode ships when it cuts p95 on that SKU. If the draft model misses, the path falls back to the base kernel.
Structured out
JSON and grammar-constrained decode for extract jobs. The schema is part of the request, not a prompt footnote.
Our cells
Shared Spout, reserved Ladle, or dedicated Caster in Ashburn, Oregon, Frankfurt, and Mumbai. Same API. Same console. GPU hours itemized on the invoice.
Read published ratesSelf-host the pour
Caster can sit in your VPC with the same chat, embeddings, and fine-tune endpoints. Weights and prompts stay on your side of the fence.
VPC and single-tenant“We moved a 70B off a shared endpoint and the p95 settled.”
“Fine-tune in the morning. Same API at lunch.”
Next heat
Talk to the person who meters the pour.
Rahul reads founder, CEO, and named mail. Bring a model, a latency number, and the region you need live.
Talk to an engineer