Sapnil Patel

A distributed LLM serving platform that makes LLaMA-3 & Mistral answer faster, cheaper, and at higher volume.

The problem. Naive LLM serving runs one request at a time in FP32 — GPUs sit idle between requests and memory is wasted, making inference slow and expensive.

• ▸Dynamic batching: concurrent requests merged into a single forward pass within a max_wait_ms window
• ▸Explicit KV-cache decode loop with a memory estimator for OOM-safe admission control
• ▸FP16/BF16 mixed precision with automatic CPU→FP32 fallback
• ▸Tensor parallelism across GPUs via DeepSpeed, with async batching scheduler and admission control for backpressure
• ▸Prometheus + Grafana observability; deployed on GCP via GKE with horizontal pod autoscaling for zero-downtime serving

HTTP ─▶ FastAPI ─▶ InferenceEngine ─▶ DynamicBatcher ─▶ ModelBackend
                     │                   │                  │
                /metrics           async queue +      prefill + KV-cached
               (Prometheus)       max_wait window     decode, FP16/BF16 · TP

Field note: Inference cost is an engineering problem, not a hardware bill you just pay.

A search engine built from first principles — crawler, index, and ranker — answering queries in milliseconds, with the performance numbers to prove it.

The problem. Search frameworks hide the interesting parts. Building the whole pipeline from scratch — crawling, deduplication, indexing, BM25 ranking — exposes every data-structure and memory trade-off that makes real search engines fast.

• ▸Positional inverted index with sorted posting-list intersection, phrase matching, and bounded min-heap top-K selection
• ▸Parallel async crawl pipeline: bounded worker pools, backpressure, a priority frontier enforcing per-host politeness, graceful shutdown
• ▸From-scratch bloom filter for URL dedup — measured 1.02% false-positive rate, within 2% of theory
• ▸215 tests passing in CI

seeds ─▶ frontier (per-host politeness) ─▶ async fetchers (bounded pool)
             ▲                                  │
       bloom-filter dedup ◀── links ─── parser ─▶ positional inverted index
                                                      │
query ─▶ tokenize ─▶ posting-list intersection ─▶ BM25 ─▶ top-K heap

Field note: Inverting the question — look up the word, not the page — is the one trick that makes every search engine fast.