What this page describes
RAGOps is the discover → measure → advise surface over a live retrieval stack.
The hosted stack at rag.automatico.llc is Cognito-gated (not a public sandbox). On a walkthrough
you can run the UI, research-agent routes (/ask, /resume), and RAG / Ops APIs
(/rag/*, /ingest/*). Architecture on this page is always open.
Edge delivery uses CloudFront and S3; the data plane combines HTTP API Gateway, Cognito, ingestion and agent Lambdas, Qdrant on ECS/Fargate, document storage in S3, and LangGraph checkpoints in DynamoDB.
/rag/search emits stage traces, run stats, and a health score. A corpus census runs without any traffic at all.Reference deployment
Corpus census, pipeline component status, and advisor findings from the live demo environment.
Corpus
| Measure | Value | Why it matters |
|---|---|---|
| Chunks / documents | 307 across 52 sources | Demonstration scale on Qdrant |
| Topics | 11 — procurement, cafeteria ops, legal, medical, retail, terraform, platform ops, RAG foundations, security, software engineering, and a deliberate garbage lane | Genuinely mixed domains, not one clean corpus |
| Modality mix | markdown 164 · code 67 · prose 64 · table 12 | Heterogeneous — score distributions differ per modality, which is what breaks a single global floor |
| Duplicate chunks | 10 (3.3%) | Near-duplicate and contradictory documents are seeded on purpose to exercise the advisor |
Pipeline components
| Component | State | Detail |
|---|---|---|
| Dense retrieval | running | Titan Embed v2 → Qdrant (ECS/Fargate, private VPC) |
| Lexical BM25 + RRF | running | In-process over a cached corpus scan; fuses on rank with k=60 |
| Cross-encoder rerank | running |
amazon.rerank-v1:0 in us-west-2 (cross-region from us-east-1).
Amazon Rerank runs on a shortlist after hybrid fusion.
|
| Per-source calibration | not fitted | No thresholds on disk, so a single global floor of 0.12 applies to every score source |
| Query transformation | off | Disabled in config; vague and multi-hop queries reach the retriever unmodified |
| Confidence floor | soft | Weak hits stay ranked and flagged rather than being dropped |
What the advisor says about it
| Severity | Finding | Fix |
|---|---|---|
| error | uncalibrated_heterogeneous_corpus — mixed modalities share one global score floor |
Fit per-modality thresholds from the evaluation harness |
| info | no_query_transform — contextualization and HyDE are disabled |
Enable query transformation in configuration |
Product loop
Declared profile versus observed discovery → reconcile → instrumented traffic → advice. A person closes the loop—nothing auto-applies.
Kickoff Q&A
Operator declares the stack they think they are running.
Soft discovery
Live settings and pasted configs are read and compared to the declaration.
Conflicts?
Declared profile and observed reality either agree or they do not.
Reconcile
Mark what is observed, declared, or incomplete before the profile goes live.
Skip reconcile
Declaration matches discovery—continue straight to the active profile.
Active profile
The reconciled configuration that search and Ops will measure against.
Search traffic
baseline or hybrid queries hit the live retrieval stack.
Traces · health · run log
Stage timing, empty rates, and scores accumulate from real traffic.
Advisor findings
Recommendations with the evidence that produced them—modality, thresholds, duplicates, latency.
Human applies config
Advice leaves the loop through a person. Nothing silently rewrites the pipeline. Then the active profile is updated and measurement continues.
Corpus-level findings (/rag/ops/audit) skip the traffic stages entirely,
so a brand-new deployment with zero queries still gets an audit.
Public deployment
Live hostname: rag.automatico.llc.
CloudFront serves the SPA from S3 and proxies API paths to HTTP API Gateway.
Vectors live in Qdrant on ECS/Fargate; documents in S3; embeddings and models on Bedrock;
LangGraph checkpoints in DynamoDB. Cognito JWT required for RAG and ingest routes.
Browser
Operator signs in via Cognito, then uses the SPA at rag.automatico.llc.
CloudFront distribution
SPA from S3 · proxies /ask · /resume · /health · /rag/* · /ingest/*
UI bucket
Private S3 bucket — static React SPA.
HTTP API
Cognito JWT authorizer in front of Lambdas.
Ingestion Lambda
Chunk · embed · upsert · /rag/* · /ingest/*.
Agents Lambda
LangGraph research paths · /ask · /resume.
Qdrant · ECS/Fargate
Vector index on EFS in a private VPC.
S3 documents
Raw corpus objects for ingest.
DynamoDB checkpoints
HITL resume state — not the vector store.
Titan Embed v2
Query and document embeddings.
Claude + Amazon Rerank
Agent reasoning / grading; cross-encoder rerank via amazon.rerank-v1:0 in us-west-2.
Runtime variants
| Variant | Hosting | Vectors | When |
|---|---|---|---|
| Public demo (live) | CloudFront + API Gateway + Lambda | Qdrant on ECS/Fargate | rag.automatico.llc · Cognito required |
| Local development | Docker Compose | Qdrant | Offline pipeline development |
The public demo uses Qdrant on ECS/Fargate. A vector-store facade in code supports alternate backends for other deployment targets.
Unauthenticated probes to /rag/* and /ingest/* return 401 Unauthorized.
/health is open and returns {"status":"ok"}.
Retrieval pipeline
Shared by the Search tab and Ops measurement. baseline is dense retrieval only,
kept as an honest control arm; hybrid is the full stack. BM25 and fusion run in
the serving path rather than only in the eval harness, so both measure the same thing.
Query
Search request enters the shared serving path.
Query transform
Heuristic gate first; LLM transform stays off by default.
Dense retrieve
Titan embed → vector store.
Lexical BM25
In-process, cached corpus index.
Reciprocal rank fusion
Merge on rank, not score — cosine and BM25 stay on their own scales.
Phrase pool merge
Shortlist candidates for the expensive second stage.
Cross-encoder rerank
Shortlist only — never first-stage.
Per-source calibration
Annotate by default; never silently filter unless opted in.
Confidence floor (soft)
Flag weak hits; keep them ranked unless a hard floor is enabled.
Finalize
Body prune · sibling expand · package the result.
Trace + Ops + health → UI
Measurement surfaces get the same path Search uses.
Effective configuration
| Knob | Deployed value | Rationale |
|---|---|---|
rag_top_k / candidates | 8 / 20 | Returned hits versus first-stage pool |
rag_hybrid_enabled | true | Dense and lexical both feed fusion |
rag_rrf_k / rag_lexical_pool_k | 60 / 50 | Standard RRF damping; lexical pool depth |
rerank_backend / pool | bedrock / 50 | Cross-encoder over the shortlist only |
rag_min_vector_score | 0.12 (0.08 topic-scoped) | Global floor, in force because calibration is unfitted |
rag_hard_floor | false | Soft floor: flag weak hits, keep them ranked |
calibration_apply | false | Advisory only — annotate, never filter |
query_transform_enabled | false | Off by default; each transform costs an LLM call |
| Chunking | 1000 / 200 overlap, structural | Structural split respects document boundaries |
Design decisions
The choices that are load-bearing, and what each one is defending against.
| Decision | Reasoning |
|---|---|
| Fuse on rank, not score | Cosine similarity and BM25 occupy incomparable scales, so any weighted sum of the two encodes an arbitrary exchange rate. Reciprocal rank fusion only needs the ordering each retriever produces, which is the part both are actually good at. |
| Cross-encoder second, never first | A bi-encoder scores query and passage separately, so it can only measure whether they land near each other in embedding space. A cross-encoder reads both together and scores the real relationship, but its cost is linear in candidates, so it can only ever run over a shortlist. |
| Soft confidence floor | A hard floor returns empty results with no path forward. Flagged-but-ranked keeps weak hits visible so the reranker or reviewer can still act on them. |
| Calibration per score source | One global threshold across vector, lexical and rerank scores is wrong for at least one source in a heterogeneous corpus. Thresholds can be fitted per modality and applied only after review. |
| Ambiguity measured against this corpus | Global word-sense priors do not transfer: in an AWS corpus "apple" is noise, and in a grocery corpus it is the subject. Term senses are detected by clustering the corpus's own embeddings, so priors come from the deployment rather than from the internet. |
| Degrade, never fail | Reranking, calibration and query transformation can each be unavailable. Each returns a reason instead of raising, so the pipeline keeps serving and the missing stage becomes an advisor finding. |
| Qdrant on ECS for the public demo | The deployment uses a dedicated vector database (Qdrant on Fargate + EFS). DynamoDB holds LangGraph checkpoints, not vectors. |
| Same code in serving and eval | BM25 and fusion live in lexical_index.py and fusion.py, imported by both the request path and the benchmark harness, so measured lift cannot drift away from shipped behaviour. |
Request path
CloudFront keeps the SPA on S3 and forwards API path patterns to HTTP API Gateway. Cognito JWT is required on protected routes.
GET /
Browser → CloudFront → S3 SPA.
Sign in
Cognito issues a Bearer token for the signed-in UI session.
POST /rag/search (or /ask)
CloudFront behavior → API Gateway → ingestion or agents Lambda.
Embed + retrieve
Bedrock Titan embed → Qdrant rank (plus hybrid / SWAG / CRAG stages when enabled).
Response
Hits, traces, and Ops fields return to the signed-in browser session.
Ingestion & reseeding
Uploads and URL fetches land in S3 first, then get chunked, classified, embedded and upserted.
POST /api/ingest/file
Browser → CloudFront → Lambda (POST /ingest/file).
Store raw object
Upload lands in the S3 documents bucket first.
Extract → chunk → classify
Chunking plus chunk_kind and chunk_modality labels.
Embed batches
Titan embedding calls over chunk batches.
Upsert vectors + payload
Qdrant on the public demo (ECS) and for local Ops.
Ack
Response returns document_id and chunks_indexed.
| Field | Describes | Used by |
|---|---|---|
chunk_kind | Shape — body text versus navigation chrome | Body-prune during finalize |
chunk_modality | Format — prose, code, table, markdown, pdf_text | Per-source calibration, modality census, advisor heterogeneity findings |
Bulk re-ingest
Corpus replays go through the public ingest API so chunking, modality labels, and embeddings stay on the same path as interactive uploads.
Ops surfaces & APIs
| Endpoint | Role |
|---|---|
GET /features | UI feature flags |
GET /rag/pipeline | Stage map, effective settings, corpus stats |
GET /rag/ops/audit | Corpus and config audit; runs with zero search traffic |
GET /rag/ops/modalities | Format census and duplicate density |
GET /rag/ops/ambiguity | Corpus-local term ambiguity via embedding cluster split |
GET /rag/ops/calibration | Fitted thresholds and what the global floor costs |
GET /rag/ops/reranker | Whether a cross-encoder is actually running, and why not |
GET /rag/ops/runs | Recent searches, summary, traffic advice |
GET /rag/ops/advice | Traffic advisor plus pipeline and profile |
GET /rag/ops/intake | Kickoff Q&A schema |
PUT /rag/ops/profile | Save answers, soft-discover, return conflicts |
POST /rag/ops/discover | Re-run soft discovery |
POST /rag/ops/reconcile | observed · declared · incomplete |
POST /rag/search | Instrumented search; feeds the run buffer |
POST /ingest/file · /ingest/url | Ingest; /ingest/preview dry-runs chunking |
GET /rag/documents · /rag/chunks | Library inspection |
Module map
| Concern | Module |
|---|---|
| Pipeline orchestration | augmentation.py |
| Lexical retrieval / fusion | lexical_index.py · fusion.py |
| Reranking | reranker.py |
| Thresholds | calibration.py · confidence.py |
| Query rewriting | query_transform.py |
| Diagnostics | advisor.py · ambiguity.py · rag_health.py |
| Ops state | pipeline_ops.py · pipeline_discover.py · pipeline_profile.py |
| Storage facade | rag_store.py → qdrant_vector_store.py |
Security posture
| Control | Implementation |
|---|---|
| TLS | CloudFront HTTPS with ACM certificate on rag.automatico.llc |
| Auth | Cognito JWT on API Gateway for /rag/*, /ingest/*, agent routes |
| Network | Qdrant in private subnets; Lambdas in VPC security groups |
| IAM | Least-privilege Lambda roles for S3, Bedrock, DynamoDB checkpoints, Qdrant access path |
| Open probe | /health remains unauthenticated for liveness checks |
Signed-in access is required for search and ingest.