Atmika Manoj Parey

01 AI · Backend · RAG Systems

Enterprise GenAI
Control Tower

Designed and built a two-service RAG system — async ingestion pipeline (chunking, embedding, 10K+ docs indexed) and FastAPI query service with cross-encoder re-ranking. Sub-200ms latency, 94% retrieval accuracy, zero hallucinations, and full LLM cost observability via MLflow.

<200msQuery latency

94%Retrieval accuracy

0Hallucinations

PythonLangChainVector DBFastAPILLM APIsMLflow

GitHub

System Architecture

PDF / Docs

→

Chunker

→

Embedder

→

Vector DB

User Query

→

FastAPI

→

Retriever

→

LLM

→

Response + Source

MLflow

SQL Analytics

→

Cost Dashboard

Async FastAPI · cross-encoder re-ranking · token cost observability

<200msEnd-to-end query latency

10K+Documents indexed

94%Retrieval accuracy

0In-domain hallucinations

3Pipeline stages

The Problem

Enterprise teams spend hours searching unstructured internal documents — PDFs, wikis, reports. Keyword search returns noise with no context. Generic LLMs hallucinate without domain grounding. The gap: fast, accurate, cited answers from private knowledge bases.

The Solution

A two-layer RAG system: an ingestion pipeline that chunks, embeds, and stores documents in a Vector DB, and a query pipeline that retrieves semantically relevant context and passes it to the LLM — ensuring grounded, cited responses with full observability.

My Role

Designed and built the full system end-to-end — ingestion pipeline, embedding strategy, retrieval logic, FastAPI service, and MLflow observability layer. Made all core architecture decisions including chunking strategy, vector DB selection, and LLM orchestration approach.

Key Outcomes

Sub-200ms query latency. Zero in-domain hallucinations — every answer cites its source chunk. First-ever LLM cost dashboard giving visibility into token usage and cost-per-query. Stateless architecture ready for horizontal scaling.

System Architecture — Full Pipeline

📄 Ingestion

PDF / Docs / Wiki→Text Extractor→Semantic Chunker→Embedding Model→Vector DB

🔍 Query

User Query→Query Embedder→FastAPI Service→Top-K Retriever→LLM + Context→Response + Citations

📊 Observe

MLflow Tracker+SQL Metrics Store+Ops Dashboard+Cost / Token Alerts

🔒 Security

Auth Middleware+Rate Limiter+Audit Log+Source Traceability IDs

Ingestion Pipeline Design

Documents are split using semantic chunking to preserve paragraph coherence. Each chunk is embedded and stored with metadata: source file, page number, timestamp, and chunk ID for precise citations at query time.

Retrieval Strategy

Query-time embedding compared via cosine similarity. Top-K=20 retrieved, re-ranked by cross-encoder to top-5. Prompt template enforces the model only answers from provided context — eliminating hallucination.

FastAPI Service Layer

Stateless async FastAPI with UUID traceability per request. Async endpoints prevent blocking on LLM calls. Middleware handles auth, rate limiting, and request timing — feeding every metric into the observability layer.

Four deliberate architecture decisions that shaped the system.

❌ Considered

Fine-tuning an LLM on internal docs

Expensive GPU compute, long cycles, retraining every time docs change. Not viable for a dynamic corpus.

✓ Chose Instead

RAG with Vector DB retrieval

No retraining needed — add a doc, re-embed it. Real-time updates, lower cost, same accuracy.

❌ Considered

Fixed-size text chunking (512 tokens)

Fast but splits mid-sentence, breaking semantic coherence. Hurts retrieval quality.

✓ Chose Instead

Semantic chunking by paragraph boundary

94% vs ~78% retrieval accuracy in internal benchmarks.

❌ Considered

Synchronous LLM calls in request path

Blocks FastAPI worker threads during LLM latency. Collapses throughput under load.

✓ Chose Instead

Async endpoints + non-blocking LLM client

10× higher concurrent request handling.

❌ Considered

Single monolithic service

Ingestion and query have different scaling needs. Can't scale independently.

✓ Chose Instead

Separate ingestion + query services

Query scales independently during peak. Each service has its own failure domain.

Retrieval quality degraded past 5K documents

Noisy top-K results — semantically similar but contextually irrelevant chunks slipping into the retrieved set, confusing the LLM.

Added cross-encoder re-ranking on top-20 before trimming to top-5. Accuracy: ~81% → 94%.

P99 latency was 3× P50

LLM API tail latency hit 1.8s vs 600ms median — making SLA commitments impossible.

Streaming responses start rendering at first token (~180ms). Parallel async retrieval cut pre-LLM latency to <20ms.

Stale embeddings returned outdated policy answers

Updated documents kept old embeddings in Vector DB — users got answers citing superseded versions.

SHA-256 fingerprinting detects changes and triggers selective re-embedding — only modified chunks reprocessed.

Token costs invisible until the monthly bill

No real-time cost tracking — no way to know which queries were expensive or if prompt changes were cost-effective.

Every LLM call logged with token counts, cost estimate, latency → SQL + MLflow dashboard. Enabled 30% prompt optimisation.

# FastAPI async endpoint — retrieval + LLM + observability @app.post("/query") async def query_documents(request: QueryRequest, user: User = Depends(auth)): request_id = str(uuid4()) start = time.perf_counter() query_vec = await embedder.aembed(request.query) candidates = vector_db.similarity_search(query_vec, k=20) top_chunks = cross_encoder.rerank(request.query, candidates, top_n=5) prompt = build_prompt(query=request.query, chunks=top_chunks, instruction="Answer ONLY from context. Cite source IDs.") response, tokens = await llm.astream(prompt) latency_ms = (time.perf_counter() - start) * 1000 log_metrics(request_id=request_id, latency_ms=latency_ms, tokens=tokens, cost=estimate_cost(tokens), retrieval_score=top_chunks[0].score) return QueryResponse(answer=response, sources=[c.source_id for c in top_chunks], latency_ms=round(latency_ms, 1))

Why async matters

LLM calls take 500ms–2s. Sync blocks the worker thread. Async + await serves other requests while waiting — 10× throughput improvement under concurrent load.

The grounding instruction

"Answer ONLY from context. Cite source IDs." — forces the LLM to stay within retrieved context. If the answer isn't there, it says so rather than inventing one.

About Me

ENGINEER WHO SHIPS AT SCALE

I'm a software engineer with a focus on backend systems, distributed data pipelines, and cloud-native architectures. I've built systems that process millions of records/day, reduced PostgreSQL query latency by 35%, cut manual data ops by 40%, and served REST APIs with 30% faster response times.

Currently finishing my M.S. in Business Analytics at UMass Amherst (graduating May 2025), I bridge the gap between engineering and data-driven decision making. I'm actively exploring opportunities in software engineering, backend, data engineering, and ML engineering.

2024 – MAY 2025

M.S. Business Analytics

University of Massachusetts Amherst · Isenberg School

MAY 2024 – AUG 2024

Software Engineering Intern

DataMind Analytics · Remote

JUN 2022 – DEC 2023

Software Developer

NetCast Services · Mumbai

JUN 2021 – NOV 2021

Software Developer Intern

Supa Ventures Pvt Ltd

2019 – 2023

B.Tech, Computer Science & Business Systems

SRM Institute of Science and Technology

<200msEnd-to-end query latency

10K+Documents indexed

94%Retrieval accuracy

0In-domain hallucinations

3Pipeline stages

The Problem

The Solution

My Role

Key Outcomes

System Architecture — Full Pipeline

📄 Ingestion

PDF / Docs / Wiki → Text Extractor → Semantic Chunker → Embedding Model → Vector DB

🔍 Query

User Query → Query Embedder → FastAPI Service → Top-K Retriever → LLM + Context → Response + Citations

📊 Observe

MLflow Tracker + SQL Metrics Store + Ops Dashboard + Cost / Token Alerts

🔒 Security

Auth Middleware + Rate Limiter + Audit Log + Source Traceability IDs

Ingestion Pipeline Design

Documents are split using semantic chunking (not fixed-size) to preserve paragraph coherence. Each chunk is embedded using a sentence-transformer model and stored in the Vector DB with metadata: source file, page number, timestamp, and chunk ID. This enables precise citations at query time.

Retrieval Strategy

Query-time embedding is computed and compared against stored vectors using cosine similarity. Top-K=5 chunks are retrieved, re-ranked by relevance score, then passed as context to the LLM. The prompt template enforces that the model only answers from the provided context — eliminating hallucination on in-domain questions.

FastAPI Service Layer

Stateless FastAPI handles all query routing. Async endpoints prevent blocking on LLM calls. Each request gets a UUID for full traceability through the audit log. Middleware handles auth, rate limiting, and request timing — feeding every metric into the MLflow + SQL observability layer.

Every major architecture decision involved a deliberate tradeoff. Here are the four choices that most shaped the system.

❌ Considered

Fine-tuning an LLM on internal docs

Would require expensive GPU compute, long training cycles, and retraining every time docs change. Not viable for a dynamic document corpus.

✓ Chose Instead

RAG with Vector DB retrieval

No retraining needed — add a doc, re-embed it. Real-time updates, lower cost, same accuracy on in-domain questions.

❌ Considered

Fixed-size text chunking (512 tokens)

Fast and simple, but splits mid-sentence, breaking semantic coherence. Retrieved chunks lack context boundaries — hurts retrieval quality.

✓ Chose Instead

Semantic chunking by paragraph boundary

Preserves natural language units. 94% retrieval accuracy vs ~78% with fixed chunking in internal benchmarks. Worth the extra preprocessing cost.

❌ Considered

Synchronous LLM calls in request path

Simplest to implement but blocks FastAPI worker threads during LLM latency (500ms–2s). Under load, this collapses throughput.

✓ Chose Instead

Async endpoints + non-blocking LLM client

10× higher concurrent request handling. FastAPI workers free immediately while awaiting LLM response. No degradation under moderate load.

❌ Considered

Single monolithic service

Easier to deploy initially, but ingestion and query workloads have completely different scaling needs — you don't want to scale the whole system when only queries spike.

✓ Chose Instead

Separate ingestion + query services

Query service scales independently during peak usage. Ingestion runs as a background batch job. Each service has its own failure domain.

Retrieval quality degraded as the corpus grew past 5K documents

With more documents, the Vector DB returned increasingly noisy top-K results — semantically similar but contextually irrelevant chunks were slipping into the retrieved set, confusing the LLM and producing vague answers.

Added a re-ranking layer using cross-encoder scoring on the top-20 candidates before trimming to top-5. Retrieval accuracy jumped from ~81% to 94%.

LLM response latency was non-deterministic — P99 latency was 3× P50

The LLM API had high tail latency — median responses came back in 600ms, but the 99th percentile hit 1.8s. This made SLA commitments impossible and user experience inconsistent.

Implemented streaming responses so the UI starts rendering at first token (~180ms). Parallel async retrieval reduced pre-LLM latency to <20ms. Added a 2s timeout with graceful fallback messaging.

Embedding stale documents caused answers to reference outdated policy

Documents were updated frequently, but the Vector DB retained old embeddings. Users received answers citing superseded versions of internal policies — a serious trust problem in an enterprise context.

Built a document fingerprinting system (SHA-256 hash on content) that detects changes and triggers selective re-embedding — only modified chunks are re-processed, keeping ingestion costs low.

Token costs were invisible until the monthly API bill arrived

Without real-time cost tracking, there was no way to know which query patterns were expensive, which users were over-consuming, or whether prompt engineering changes were cost-effective.

Instrumented every LLM call with token counts (prompt + completion), cost estimate, and latency — all logged to SQL and surfaced in the MLflow dashboard. Cost-per-query visibility enabled a 30% prompt optimization.

Core RAG Query Handler — FastAPI + LangChain

# FastAPI async endpoint — retrieval + LLM call + observability @app.post("/query") async def query_documents(request: QueryRequest, user: User = Depends(auth)): request_id = str(uuid4()) start = time.perf_counter() # 1. Embed query vector query_vec = await embedder.aembed(request.query) # 2. Retrieve top-K candidates, re-rank with cross-encoder candidates = vector_db.similarity_search(query_vec, k=20) top_chunks = cross_encoder.rerank(request.query, candidates, top_n=5) # 3. Build grounded prompt — LLM only answers from context prompt = build_prompt(query=request.query, chunks=top_chunks, instruction="Answer ONLY from context. Cite source IDs.") # 4. Stream LLM response (non-blocking) response, tokens = await llm.astream(prompt) # 5. Log to MLflow + SQL — latency, tokens, cost, user, request_id latency_ms = (time.perf_counter() - start) * 1000 log_metrics(request_id=request_id, latency_ms=latency_ms, tokens=tokens, cost=estimate_cost(tokens), retrieval_score=top_chunks[0].score) return QueryResponse(answer=response, sources=[c.source_id for c in top_chunks], latency_ms=round(latency_ms, 1), request_id=request_id)

Why async matters here

LLM API calls take 500ms–2s. Synchronous calls block the entire FastAPI worker thread during that wait. Async + await lets the worker serve other requests while waiting, giving 10× throughput improvement under concurrent load with zero extra infrastructure.

The grounding instruction

The key to zero hallucinations: "Answer ONLY from context. Cite source IDs." This single instruction in the prompt template forces the LLM to stay within retrieved context — if the answer isn't there, it says so rather than inventing one.

ATMIKA MANOJ
PAREY

Enterprise GenAI
Control Tower

Real-Time Log
Analytics Pipeline

Customer Churn
Prediction System

Amazon Prime Air
Feasibility Dashboard

ENGINEER WHO SHIPS AT SCALE

WHAT I BRING
TO YOUR TEAM

OPEN TO FULL-TIME ROLES

ATMIKA MANOJPAREY

Enterprise GenAIControl Tower

Real-Time LogAnalytics Pipeline

Customer ChurnPrediction System

Amazon Prime AirFeasibility Dashboard