Should we self-host models to cut costs?

Rarely worth it until you have $10M+ ARR and predictable, high-volume workloads. Hosted APIs (OpenAI, Anthropic) are cheaper than self-hosting at low volume because you don't pay for idle GPU time. Break-even for self-hosting is typically millions of daily requests with steady load.

How much of inference cost should we mark up?

Standard SaaS gross margin logic still applies — target 70-80% gross margin. If inference costs 20% of the price, mark up 5x. Customers accept this as long as pricing is transparent. Trying to price at 2x cost destroys margin; pricing at 10x creates competitive vulnerability.

What about frontier model price cuts?

Frontier model prices have dropped ~90% over two years and continue to fall. Don't build your entire economics assuming today's prices persist, but also don't wait for future price drops to fix broken unit economics now. Optimize aggressively today, benefit again when prices drop.

AI Inference COGS: Structure, Optimization & Pricing (2026)

AI-native SaaS has fundamentally different unit economics than traditional SaaS.

AI COGS: Managing Inference Costs Without Destroying Gross Margin

Traditional SaaS has near-zero marginal cost per user — hosting and support scale sub-linearly with revenue. AI-native SaaS is different: every user action calls an LLM, every LLM call costs real money, and costs scale linearly (sometimes worse than linearly) with usage. Companies that don't manage inference costs actively see gross margin collapse from 80% at launch to 40-50% at scale.

What's in AI COGS

LLM inference (per-token costs from OpenAI, Anthropic, Google, or self-hosted). Vector database costs (Pinecone, Weaviate) for RAG. Embedding costs for indexing new documents. GPU compute for self-hosted models. Data storage costs (usually small vs. inference). Third-party AI APIs used inside the product (transcription, image generation, etc.). These typically run 15-40% of revenue for AI-heavy products — vs. 5-15% for traditional SaaS.

Model routing

The single highest-leverage optimization. Use expensive frontier models (GPT-5, Claude Opus) only for tasks that genuinely need them. Use mid-tier models (GPT-4.5, Claude Sonnet) for standard workloads. Use small local models (open-weight or fine-tuned) for classification and simple tasks. Well-implemented routing cuts inference costs 40-70% with minimal quality impact. Requires infrastructure to detect task type and route accordingly.

Caching and reuse

Semantic caching: identical or near-identical queries return cached responses instead of hitting the LLM. Prompt caching (offered by Anthropic and OpenAI): reuse cached prompt prefixes at reduced cost. Response reuse: for deterministic tasks, cache outputs indefinitely. Typical impact: 20-40% cost reduction with proper cache design. Higher for consumer-facing products with repetitive queries; lower for personalized workflows.

Pricing that passes through costs

Two structures. (1) Usage-based pricing where price scales with inference cost — natural alignment, no margin blowout at scale. (2) Tiered pricing with usage limits and overages — simpler for buyers, requires accurate cost modeling per tier. Avoid: unlimited-usage flat pricing on AI-heavy products — heavy users destroy your unit economics. If you must offer unlimited, cap it operationally (rate limits, model tier restrictions).

Frequently asked questions

Should we self-host models to cut costs?: Rarely worth it until you have $10M+ ARR and predictable, high-volume workloads. Hosted APIs (OpenAI, Anthropic) are cheaper than self-hosting at low volume because you don't pay for idle GPU time. Break-even for self-hosting is typically millions of daily requests with steady load.
How much of inference cost should we mark up?: Standard SaaS gross margin logic still applies — target 70-80% gross margin. If inference costs 20% of the price, mark up 5x. Customers accept this as long as pricing is transparent. Trying to price at 2x cost destroys margin; pricing at 10x creates competitive vulnerability.
What about frontier model price cuts?: Frontier model prices have dropped ~90% over two years and continue to fall. Don't build your entire economics assuming today's prices persist, but also don't wait for future price drops to fix broken unit economics now. Optimize aggressively today, benefit again when prices drop.

Related fundraising guides (40)

Investor directory · Fundraising library · Articles A–Z · Company funding database