Latency classes
spout-* Shared cell. Cold start is real. Good for batch and burst chat.
ladle-* Reserved replica. Warm prefix cache. Slack on the same thread as paging.
caster-* Dedicated GPUs, written p95, optional VPC. TAM and an FDE on the pour.
Platform
Serve the checkpoint you actually run.
Open-source, custom, and fine-tuned models on one API. Sandbox keys to prototype. Observability for tokens, dollars, and GPU metrics. Forward-deployed engineers on Caster.
Production inference
Streaming, batch, structured decode.
The chat completions path is OpenAI-compatible enough to migrate a working client. Embeddings share the same key. Dedicated endpoints pin a replica to a SKU, a region, and a concurrency cap.
Multi-LoRA sits on one GPU when adapters share a base and the isolation note is green. Regions live today: us-east-1 Ashburn, us-west-2 Oregon, eu-central-1 Frankfurt, ap-south-1 Mumbai.
Open the API notesFine-tunes
LoRA and full jobs, same serving path.
LoRA
Adapters train against a pinned base. The job writes a checkpoint into the project. Promote it to a spout, ladle, or caster SKU without a second export step.
Full fine-tune
Full-weight jobs run on reserved H100 or A100 hours. Data stays in the project bucket. A written job is required before customer records touch a trainer.
Runtime and kernels
Engineering notes, with caveats.
Custom kernels
Attention and fused decode kernels ship per SKU after a lab pass. A kernel that misses the SLA on that shape stays off.
Caching
Prefix and KV cache are on by default for chat SKUs. Batch extract can disable cache when prompts do not repeat.
Speculative decode
Draft-plus-target is enabled when it lowers p95 on the published shape. Acceptance rate is visible on the replica chart.
VPC and self-host
Single-tenant in our cloud, or yours.
Caster can be a locked cell in a Tundrel region or a stack in your VPC. The API surface stays chat, embeddings, fine-tune jobs, and dedicated endpoints. Keys still issue from the console.
Private link, customer-managed keys, and a written network diagram are part of the Caster packet. Rahul walks the first cut on a call.
Same API
Base URL changes. Auth header does not. Quotas become the replica count you bought. Status for a VPC cell reports through the same strip as public regions.
VPC quickstartModel library
Open weights and bring-your-own.
Each card lists size, context, and the serving SKU that is live today. Upload a safetensors bundle to a project when the license is yours.
Llama 3.1 8B Instruct
spout-8b · ladle-8b · caster-8b
Llama 3.3 70B Instruct
spout-70b · ladle-70b · caster-70b
Qwen2.5 32B Instruct
spout-32b · ladle-32b
Mistral Small 24B
spout-32b · ladle-32b
Gemma 2 9B
spout-8b · ladle-8b
Bring your own
project safetensors
Use cases
Latency class per pour.
Voice stacks
STT, LLM, and TTS on a tight hop budget. Pin the LLM hop to a ladle or caster replica so the turn stays under 250 ms p95 on 8B.
Coding agents
Streaming completions with an 80 ms TTFT target on 8B and a longer 32B path for repo-scale edits. Prefix cache keeps the repo head warm.
Document models
32k to 128k context for contracts and filings. Structured decode writes JSON the caller can file. Batch during off-peak hours on Spout.
Chat
Product chat on a dedicated replica when the p95 is in the contract. Shared Spout for internal tools that can wait on a cold start.
Batch extract
Overnight pours across a corpus. Pay per million on Spout. Quotas raise with a note to Rahul if the job crosses 200M tokens in a day.