Anatomy Atlas: Interactive 3D Anatomy with Grounded RAG QA

Overview

Retrieval-augmented generation (RAG) answers questions from retrieved passages and cites them, but a citation alone does not make an answer useful. In anatomy, the reader also needs to know which 3D structure the answer describes — a paragraph about "the muscle that flexes the forearm" is hard to act on when the model contains hundreds of muscles. Anatomy Atlas studies answer-to-3D-object linking: every structure named in a streaming answer is extracted, checked against a controlled anatomy vocabulary, and turned into a clickable mesh link in an interactive atlas.

The application has two halves that feed each other. A Three.js viewer renders a full-body anatomy model whose meshes are grouped into nine anatomical systems (regions, fasciae, muscular, joints, skeletal, cardiovascular, lymphoid, nervous, visceral), with per-layer transparency, focus mode, and click-to-select. A RAG chat panel answers medical and anatomy questions over an 18-textbook corpus [4]; as the answer streams in, it marks the structures it names, and the viewer highlights and navigates to those meshes. Answers cite their sources as numbered citations, and questions outside medicine are refused before any model call.

The project is also a research artifact: it ships with a 155-item behavior benchmark, a benchmark harness with LLM-as-a-judge grading and retrieval ablations, and an IEEE-format paper [1]. It is deployed publicly at anatomyedu.online and targets anatomy and medical students who want to connect textbook prose with a navigable 3D model. The interface is deliberately desktop-only — the 3D viewer needs a pointer and a large viewport.

Tech Stack & Rationale

Two decisions shape the security model. The browser only ever holds the end user's own OpenRouter key plus the chat model and answer style, sent in a base64 x-rag-config header; knowledge-base credentials (Pinecone, embedding model) stay in owner-side environment config merged on the server. And the model list comes from OpenRouter's catalog endpoint, so users pick a supported model rather than typing arbitrary model IDs.

Architecture

The system is a single Next.js app with a clear client/server split: the browser owns the 3D viewer and chat UI; the server owns retrieval, prompts, and the streaming pipeline.

• Viewer layer — anatomy.glb (Draco-compressed, meshes keyed by structure), src/data/anatomy.json (name, system, description per structure), layer slider ordered by the nine anatomical systems, focus mode that ghosts same-system neighbors.
• RAG server — lib/rag/ hosts the pipeline: chat.ts (flow control), prompts.ts (prompt contracts), grounding.ts (vocabulary validation), vectorstore.ts (Pinecone), providers.ts (OpenRouter), sse.ts (stream events).
• Knowledge tooling — owner-only /api/knowledge/* routes gated by a bearer token that fails closed: 503 while the token is unset, 403 on a wrong token. The knowledge panel UI was removed; imports run via curl.

The flow runs: browser POST /api/chat (with x-rag-config header) → condense history into a standalone question plus a scope verdict → out-of-scope questions get a deterministic refusal before any embedding or answer call → Pinecone retrieves corpus chunks (skipped for small talk) → structure candidates are resolved, possibly one clarification round → the grounded answer streams over SSE as status, token, objects, citations, done, and error events.

On the client, each streamed answer carries two machine-readable contracts: [n] citation markers, which are rendered as source citations for only the chunks actually cited, and [[obj:key]] structure markers, which are validated against the anatomy vocabulary before display. Focus chips under the answer combine the model's markers with literal structure names found in the text, so a chip appears even when the model forgets the marker format; mesh availability is checked client-side so chips never point at missing meshes. Clicking a chip highlights the mesh and refocuses retrieval for follow-up questions.

Two build-time pipelines feed the runtime. The model pipeline turns the Z-Anatomy source [2] into anatomy.glb: meshes sharing a structure key are joined, welded, simplified, and Draco-compressed, while build-anatomy.mjs regenerates anatomy.json from the GLB plus BodyParts3D metadata [3]. The corpus pipeline turns the MedRAG textbooks [4] into searchable vectors: a single embedding pass upserts chunks into Pinecone and exports the same chunks — vectors included as base64 float32 — into an importable NDJSON pack (the shipped Gray's Anatomy starter pack holds 3,017 chunks).

Folder Structure

anatomy-atlas-qa/
  apps/frontend/
    data/                  # build-time inputs (bodyparts3d.json) — never bundled
    scripts/               # model pipeline, corpus seeding, benchmark harness
    src/
      app/
        api/               # chat, knowledge, models, settings routes
        page.tsx           # viewer + chat layout
      components/
        viewer/            # 3D viewer components
        chatbot/           # chat panel, citations, focus chips
        settings/          # settings dialog
        shared/            # Dialog shell, custom icons
      hooks/               # viewer, chat, session, settings state
      lib/
        rag/               # server RAG pipeline (chat, prompts, grounding, vectorstore)
        layers.ts          # nine-system layer ordering
        structures.ts      # structure-key mapping shared by viewer + RAG
      types/               # anatomy.ts, rag.ts
    public/
      models/anatomy.glb   # optimized full-body model
      starter-pack/        # pre-vectorized Gray's Anatomy NDJSON pack
  research/
    article/paper/         # IEEE LaTeX paper (main.tex, refs.bib)
    benchmark/             # benchmark items, gold audits, review packets
    runs/                  # benchmark run outputs per model/condition
    literature/            # related-work notes
  docker-compose.yml       # single frontend service

apps/frontend/scripts is the research workbench: roughly twenty TypeScript and Node scripts generate benchmark items, run retrieval conditions, score results with a judge model, run ablations and lexical baselines, and emit review packets — none of it ships in the app bundle.

Database Design

The project has no relational database. Persistence is a managed vector index plus static build artifacts.

The split is deliberate: the only user-specific state is the browser's own chat config, so the server stays stateless and the knowledge base is fully reproducible from the corpus pipeline.

Impact & Results

• Deployed publicly at anatomyedu.online; the viewer loads a full-body model with clickable structures and a working RAG chat panel.
• Evaluated on a 155-item behavior benchmark across eight question categories (factual, relational, clinical, myth, antecedent, composition, refusal, smalltalk) over the 18-textbook corpus, with three current-generation chat models under no retrieval (B1), raw retrieval (B2), and the full system (B3), graded by a judge model from a different model family [1].
• Answer-to-object linking depends on retrieval: of 465 answers written without retrieval, only one marks a structure. Raw retrieval reaches must-structure recall 0.44–0.51; the full system raises it to 0.65–0.67 (broad recall 0.59–0.61), a significant gain for every model (p ≤ 2×10⁻⁵) with no significant difference between models.
• The judge's mark-validity check separates the models at 0.86–0.90, and 88 of 90 refusal and smalltalk answers pass the guardrail check; the two failures refuse politely but emit stray structure marks.
• Grounding is exact: no mesh link produced by the system points outside the anatomy vocabulary.
• The research is written up as an IEEE-format conference paper (ICITEE camera-ready) [1].

My Contributions

• Designed and built the entire system as sole author — viewer, RAG pipeline, benchmark harness, deployment, and paper.
• Built the 3D viewer on react-three-fiber: per-layer transparency across nine anatomical systems, focus mode with same-system ghosting, click-to-select with BVH-accelerated picking, and transparent-sorting/lighting fixes for realistic PBR rendering.
• Wrote the asset pipeline that turns the Z-Anatomy source into a deployable model: mesh joining by structure key, weld/simplify/Draco compression, and structure-metadata regeneration from BodyParts3D.
• Implemented the server RAG pipeline: history condensing with scope verdicts, deterministic out-of-scope refusals, Pinecone retrieval, clarification rounds, SSE streaming, and the citation and structure-marker prompt contracts.
• Built the answer-to-3D-object linking layer: controlled-vocabulary validation, literal-name fallback annotation, and client-side mesh-availability checks so focus chips never point at missing meshes.
• Created the benchmark harness: item generation and enrichment, retrieval-condition runs (B1/B2/B3), LLM-as-a-judge grading, retrieval ablations and a lexical baseline, rescoring, and aggregated review packets — iterated through multiple audit eras up to a full-coverage gold audit.
• Packaged the corpus pipeline (PDF/parquet → chunk → embed → Pinecone upsert + NDJSON pack) and the fail-closed owner-only knowledge-admin routes.
• Wrote the IEEE-format paper, designed the grayscale desktop UI, and deployed the public instance behind Docker.

References

[1] A. S. Shina, "Grounded answer-to-3D-object linking for retrieval-augmented anatomy question answering," Dept. of Electrical Engineering and Information Technology, Universitas Gadjah Mada, Yogyakarta, Indonesia, 2026.

[2] Z-Anatomy, "Z-Anatomy: Open-source 3D anatomy atlas." [Online]. Available: https://github.com/Z-Anatomy

[3] Database Center for Life Science (DBCLS), "BodyParts3D: Anatomical structure dictionary." [Online]. Available: https://lifesciencedb.jp/bp3d/

[4] G. Xiong, Q. Jin, Z. Lu, and A. Zhang, "Benchmarking retrieval-augmented generation for medicine," in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 6233–6251.