RAG System Overview

The Retrieval-Augmented Generation (RAG) system is the backbone of Kinetic Email's AI-powered email generation. It combines vector search with Claude AI to produce production-ready kinetic email HTML.

What is RAG?

Instead of relying solely on Claude's general knowledge, we augment each generation request with relevant code examples and best practices from our curated knowledge base. This dramatically improves output quality for kinetic email — a niche domain where general-purpose LLMs lack sufficient training data.

Storage Model (RAG v2)

Content is split across two stores:

Pinecone (v2 index) — holds only the embedding vector (text-embedding-3-large), small metadata, and a document id
Supabase rag_documents — holds the full example HTML (no size cap) plus metadata; service-role only (RLS enabled with no client policies)

After retrieval and re-ranking, the pipeline fetches the full HTML from rag_documents by vector id before assembling the context.

Architecture

User Prompt
    │
    ▼
┌────────────────────────────┐
│  Technique Extraction       │
│  Parse "Kinetic Features"   │
│  Map to: tabs, carousel,    │
│  survey, quiz, etc.         │
└─────────┬──────────────────┘
          │
    ┌─────▼─────┐
    │  Parallel  │
    │  Queries   │
    │            │
    ├────────────┤
    │            │
    ▼            ▼
┌────────┐  ┌────────────┐
│Query A │  │ Query B    │
│Filtered│  │ Unfiltered │
│topK=10 │  │ topK=15    │
│technique│  │ broad      │
│ match  │  │ search     │
└───┬────┘  └─────┬──────┘
    │             │
    └──────┬──────┘
           │
    ┌──────▼──────┐
    │ Merge &     │
    │ Deduplicate │
    └──────┬──────┘
           │
    ┌──────▼──────┐
    │ Re-rank     │
    │ Claude Haiku│
    │ Select 7    │
    └──────┬──────┘
           │
    ┌──────▼──────────────┐
    │ Fetch full HTML     │
    │ from Supabase       │
    │ rag_documents       │
    └──────┬──────────────┘
           │
    ┌──────▼──────┐
    │ Build       │
    │ Context     │
    │ - CODE      │
    │   EXAMPLES  │
    └──────┬──────┘
           │
    ┌──────▼──────────────┐
    │ Blueprint           │
    │ (Haiku generates    │
    │  copy + design      │
    │  tokens)            │
    └──────┬──────────────┘
           │
    ┌──────▼──────────────┐
    │ Parallel Generation │
    │ ┌─────┐  ┌───────┐  │
    │ │Kine-│  │ AMP   │  │
    │ │tic  │  │4Email │  │
    │ │Sonn-│  │Sonnet │  │
    │ │et   │  │       │  │
    │ └─────┘  └───────┘  │
    └─────────────────────┘

Pipeline Steps

1. Technique Extraction

The extractTechniqueFromPrompt() function parses the user's prompt for a **Kinetic Features:** section and maps it to a known technique:

Detected Feature	Pinecone Filter
Tabbed, tabs	`tabs`
Carousel, slider	`carousel`
Survey, poll	`survey`
Quiz, trivia	`quiz`
Accordion, expand	`accordion`
Hamburger, menu	`menu`
Stepper, wizard	`stepper`
Toggle, switch	`toggle`
AMP	`amp4email`

2. Dual-Query Strategy

Two Pinecone queries run in parallel:

Query A (Filtered): Uses the extracted technique as a $eq metadata filter. Returns the top 10 most relevant code examples specifically for that technique.
Query B (Unfiltered): No technique filter. Returns the top 15 most relevant results across all techniques. Captures blog posts, general best practices, and edge case knowledge.

3. Merge, Deduplicate & Filter

Results from both queries are combined and deduplicated, then filtered:

Code only — blog/concept prose is excluded from code generation entirely (a "build me tabs" query must pull the tabs module, never a blog paragraph about tabs)
Score threshold — matches must score above 50%
Positive examples only — negative/anti-pattern examples are dropped

4. Re-ranking with Claude Haiku

Claude Haiku evaluates the filtered results and selects the best mix of up to 7, optimizing for relevance to the user's prompt. If re-ranking fails, the pipeline falls back to the top 7 by score.

5. Fetch Full HTML from Supabase

Pinecone v2 stores only vectors and small metadata. The pipeline fetches the full example HTML from the rag_documents table (service role) by vector id. Examples whose HTML can't be loaded are dropped — empty code is never fed to the generator.

6. Context Assembly

Selected examples are organized into a knowledge-base section injected into Claude's system prompt:

── CODE EXAMPLES (copy these patterns exactly) ──
[Relevant HTML/AMP code snippets demonstrating the technique]

HTML code examples feed the kinetic build; AMP code examples feed the AMP build.

7. Blueprint Generation (staged copy → blueprint → code)

Before code generation, Claude Haiku produces a structural blueprint containing:

Finalized copy — headlines, body text, CTAs used verbatim in all builds
Design system — color palette, typography, spacing, and button styles
Structure — section layout, interactive config, fallback strategy

The Playground stages this explicitly: /api/gen/copy (Haiku) finalizes copy, /api/gen/blueprint (Haiku) builds the blueprint around it, then /api/generate (Sonnet) writes the code using the pre-built blueprint. If no client blueprint is supplied, generate.js builds one inline.

Brand context shapes the blueprint:

Brand design tokens are force-merged — when a brand is selected, its colors, fonts, and button styles always win; the model cannot pick new ones
Style theories (workspace_brands.style_theory: editorial-minimal, bold-brutalist, friendly-rounded, tech-mono) drive construction rules
Product catalogs (workspace_products) inject exact product names, prices, and URLs

The blueprint is the single source of truth for both kinetic and AMP builds, ensuring textual and visual parity.

8. Parallel Generation + QA

Claude Sonnet generates kinetic HTML and (optionally) AMP4Email HTML in parallel, both referencing the same blueprint for consistent copy, colors, fonts, backgrounds, and button styling. Output then passes through 26 deterministic QA checks (api/lib/qa-checks.js) and tiered auto-fixes (api/_validate-kinetic.js, including the Yahoo/AOL .& injection).

Knowledge Base

The RAG knowledge base contains three types of content:

HTML Code Examples (`type: 'html'`)

Production-tested kinetic CSS email HTML demonstrating specific techniques. Used only in kinetic builds. Each example is tagged with:

Technique (tabs, carousel, survey, etc.)
Complexity (beginner, intermediate, advanced)
Purpose (ecommerce, newsletter, transactional)
Example type (positive or negative/anti-pattern)
Best practice tags (responsive, accessible, outlook-compatible)

AMP Code Examples (`type: 'amp'`)

Production-tested AMP4Email HTML demonstrating AMP components. Used only in AMP builds. Each example is tagged with:

Technique (tabs, accordion, carousel, survey, toggle, hybrid, static)
Complexity (beginner, intermediate, advanced)
HTML type (complete template or component module)

Blog/Knowledge Articles (`type: 'blog'`)

Educational articles covering kinetic email best practices and techniques. Note: blog content is no longer injected into code generation context — code queries pull code examples only; concept prose is reserved for education surfaces. Each article is tagged with:

Content Focus — kinetic (kinetic builds only), amp (AMP builds only), or general (both builds)
Blog Topic (kinetic-techniques, email-best-practices, case-studies, tutorials)
Learning Level (beginner, intermediate, advanced)
Techniques Covered (which kinetic techniques the article discusses)
Key Takeaways (summary used for embedding generation)

The curated articles cover:

Purpose & overview of kinetic email
The kinetic declaration pattern
HTML scaffold structure
Component-specific guides (tabs, survey/quiz, accordion, carousel, menu, stepper, hero headers)
Critical rules and anti-patterns
Style strategy (6 separate <style> blocks)
Email client compatibility
Tracking implementation
Variation matrix

Technique Semantics

When content is uploaded to Pinecone, rich structural descriptions are injected into the embedding text. This helps the vector search understand technique-specific patterns:

tabs: "Radio button inputs sharing the same 'name' attribute create mutually
       exclusive tab selections. Active state highlighting via :checked + label
       selectors. Content panels shown/hidden with :checked ~ .content selectors."

These semantic descriptions are maintained in api/admin/submit-content.js and api/admin/update-content.js.

Pinecone Setup

Index (env var)	Model	Role
`PINECONE_INDEX_NAME_V2`	`text-embedding-3-large`	Primary — the only index queried by `/api/generate`; new content is upserted here (full HTML lives in `rag_documents`)
`PINECONE_INDEX_NAME`	`text-embedding-3-small`	Legacy — still used by some admin routes (list/delete/stats)
`PINECONE_INDEX_NAME_LARGE`	`text-embedding-3-large`	Legacy large index — still referenced by admin update/feedback routes

What is RAG?​

Storage Model (RAG v2)​

Architecture​

Pipeline Steps​

1. Technique Extraction​

2. Dual-Query Strategy​

3. Merge, Deduplicate & Filter​

4. Re-ranking with Claude Haiku​

5. Fetch Full HTML from Supabase​

6. Context Assembly​

7. Blueprint Generation (staged copy → blueprint → code)​

8. Parallel Generation + QA​

Knowledge Base​

HTML Code Examples (type: 'html')​

AMP Code Examples (type: 'amp')​

Blog/Knowledge Articles (type: 'blog')​

Technique Semantics​

Pinecone Setup​