Module 4 of 6 · 35 minutes

Deploying AI applications

Where and how you deploy shapes cost, speed, security and flexibility. This module covers the architecture decisions and resilience patterns that matter.

Learning objectives

Compare provider APIs, cloud AI platforms and self-hosting
Explain the benefits of a gateway
Implement timeouts, retries with back-off and fallbacks
Improve latency and plan for scale and rate limits
Apply security and data-residency practices

4.1 Hosting options

Option	Examples	Pros	Cons
Model provider API	Anthropic, OpenAI, Google Gemini APIs	Fastest start; latest models; no infrastructure	Data leaves your environment (check terms); vendor dependency
Cloud AI platform	Amazon Bedrock, Google Vertex AI, Microsoft Azure AI Foundry	Many models behind one platform; enterprise security, networking and compliance; data stays in your cloud account region	Platform-specific features; model availability varies by region
Self-hosted open-weight models	Llama, Mistral, Gemma, Qwen served with open-source servers such as vLLM	Full control and privacy; predictable cost at high volume	You run GPUs, scaling, patching and security

Three hosting options on a spectrum of control versus operational effort.

4.2 Use a gateway

Route all model calls through one internal component — your own code or an off-the-shelf AI gateway. It centralises API keys, retries, timeouts, rate limiting, logging, cost tracking and the ability to switch or route between models without touching every application.

4.3 Resilience: timeouts, retries and fallbacks

Python · retries with exponential back-off and a fallback modelimport random, time

class TransientError(Exception):
    pass

def call_with_retry(call, max_attempts=4, base_delay=1.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except TransientError:                       # e.g. rate limit, overloaded, timeout
            if attempt == max_attempts:
                raise
            delay = base_delay * 2 ** (attempt - 1) + random.uniform(0, 0.5)   # jitter
            time.sleep(delay)

def answer(question):
    try:
        return call_with_retry(lambda: primary_model(question, timeout=20))
    except Exception:
        try:
            return call_with_retry(lambda: fallback_model(question, timeout=20), max_attempts=2)
        except Exception:
            return "Sorry — I can't answer right now. A colleague will follow up."

Retries with exponential back-off (plus random jitter), then a fallback model, then a polite failure message.

4.4 Latency and user experience

Stream responses so users see text immediately.
Choose the smallest model that meets the quality bar; route harder requests to larger models.
Cache: prompt caching for long repeated prefixes; response caching for repeated questions.
Parallelise independent calls (e.g. retrieval and classification).
Keep prompts lean — every input token adds processing time.
Measure time to first token and total time, at the median and the 95th percentile.

4.5 Scaling and limits

Provider APIs enforce rate limits (requests and tokens per minute). Plan capacity for peaks, request higher limits early, queue non-urgent work, use batch APIs for bulk jobs, and apply per-user quotas so one user can't exhaust capacity — or your budget.

4.6 Security and data residency

Keep API keys in a secrets manager, never in code or prompts
Use private networking where available; restrict egress
Choose regions that meet data-residency requirements
Review providers' data retention and training terms
Log securely, with access controls and retention limits

Key takeaways

APIs are fastest to start; cloud platforms add enterprise controls; self-hosting gives control at operational cost.
Route all calls through a gateway.
Use timeouts, exponential back-off with jitter, and fallbacks.
Stream, right-size models, cache and keep prompts lean.
Protect keys, choose regions, and check retention terms.