Neo4j Full Implementation

[kien-ship-it]

Design Document — Neo4j GraphRAG (Parent Vision Spec)

Overview

This parent design defines the high-level architecture and shared contracts for the Neo4j GraphRAG system — an opt-in graph+vector search engine that enriches the existing Postgres-based semantic search pipeline. Neo4j operates as a completely independent feature set: when NEO4J_URI is set, the entire graph pipeline activates (entity extraction, relationship extraction, direct Neo4j writes); when it is NOT set, none of these steps run and no graph data is stored anywhere — not in Postgres, not anywhere. Neo4j is the ONLY place graph data lives. The existing BM25+Vector ensemble search continues unchanged regardless.

The system is model-agnostic: BERT (default, free, local) handles entity extraction and embeddings; any OpenAI-compatible LLM (Ollama, LM Studio, OpenAI, Azure) handles relationship extraction. Every component degrades gracefully when its dependencies are unavailable.

This design establishes a two-tier contract system:

1. Parent contracts — shared Zod schemas, API interfaces, Neo4j node/relationship shapes, env var conventions that ALL sub-feature specs must conform to
2. Sub-feature contracts — defined later in each sub-spec for workstream-level boundaries

---

Architecture

System Diagram

graph TB
    subgraph "Client"
        UI[Web App]
    end

    subgraph "Next.js App"
        API[API Routes]
        Ingestion[Ingestion Pipeline<br/>Steps A-G+]
        Ensemble[Ensemble Search<br/>BM25 + Vector + Graph]
        Reranker[Reranker Client]
    end

    subgraph "Data Stores"
        PG[(PostgreSQL + pgvector<br/>Source of Truth)]
        Neo4j[(Neo4j CE<br/>Graph + Vector<br/>Optional)]
    end

    subgraph "Sidecar (FastAPI)"
        Embed["/embed<br/>BERT Embeddings"]
        NER["/extract-entities<br/>BERT NER + CLS Embeddings"]
        RelEx["/extract-relationships<br/>Model-Agnostic LLM"]
        RerankModel["/rerank<br/>Cross-Encoder"]
    end

    subgraph "External (Optional)"
        Ollama[Ollama / LM Studio / OpenAI<br/>Any OpenAI-Compatible LLM]
        EmbAPI[External Embedding API<br/>Optional]
    end

    UI --> API
    API --> Ingestion
    API --> Ensemble

    Ingestion -->|Steps A-E: always runs| PG
    Ingestion -->|"Step F: BERT NER (only when NEO4J_URI set)"| NER
    Ingestion -->|"Step F2: Relationship Extraction (only when NEO4J_URI set)"| RelEx
    Ingestion -->|"Step G: Direct Write (only when NEO4J_URI set)"| Neo4j

    Ensemble -->|BM25 + Vector| PG
    Ensemble -->|"Graph Retriever (only when NEO4J_URI set)"| Neo4j
    Ensemble --> Reranker --> RerankModel

    RelEx -->|OpenAI-compatible API| Ollama
    NER -->|Local BERT model| NER

    Neo4j -.->|Section content lookup| PG

    style Neo4j fill:#e1f5fe,stroke:#0288d1
    style PG fill:#e8f5e9,stroke:#388e3c
    style Ollama fill:#fff3e0,stroke:#f57c00

Loading

Data Flow: Ingestion Pipeline

flowchart LR
    A[Upload] --> B[OCR/Normalize]
    B --> C[Chunk]
    C --> D[Embed via OpenAI or Sidecar]
    D --> E[Store Sections in Postgres]
    E --> CHECK{"NEO4J_URI set?"}
    CHECK -->|No| DONE[Pipeline Complete<br/>Postgres-only, no graph data anywhere]
    CHECK -->|Yes| F["Step F: BERT NER → Entities<br/>(+ CLS Embeddings)"]
    F --> F2["Step F2: LLM Relationship Extraction<br/>(gated: EXTRACTION_LLM_BASE_URL)"]
    F2 --> G["Step G: Direct Neo4j Write<br/>Entities + Relationships + Sections"]

    style G fill:#e1f5fe,stroke:#0288d1
    style F fill:#e1f5fe,stroke:#0288d1
    style F2 fill:#fff3e0,stroke:#f57c00
    style DONE fill:#e8f5e9,stroke:#388e3c

Loading

Key architectural decision: NEO4J_URI gates the ENTIRE graph feature set. When NEO4J_URI is NOT set, Steps F, F2, and G do not run at all — no entity extraction, no relationship extraction, no graph data stored anywhere (not in Postgres kg_* tables, not anywhere). The graph pipeline simply does not exist for users who don't enable Neo4j. When NEO4J_URI IS set, entities and relationships are extracted and written directly to Neo4j — Neo4j is the ONLY place graph data lives. The Postgres kg_* tables are a legacy artifact from the old architecture and are not used by this feature.

Data Flow: Query Pipeline

flowchart LR
    Q[User Query] --> E1[BERT NER on Query<br/>Extract Entities + Embedding]
    E1 --> ES{Ensemble Search}
    ES --> BM25[BM25 Retriever<br/>Postgres]
    ES --> Vec[Vector Retriever<br/>pgvector]
    ES --> GR["Graph Retriever<br/>Neo4j (when enabled)"]
    BM25 --> RRF[Reciprocal Rank Fusion]
    Vec --> RRF
    GR --> RRF
    RRF --> Rerank[Sidecar Cross-Encoder Rerank]
    Rerank --> Answer[LLM Answer Generation]

    style GR fill:#e1f5fe,stroke:#0288d1

Loading

Feature Packet Dependency Graph

graph TD
    R1[Req 1: Neo4j Docker Infrastructure] --> R5
    R2[Req 2: Embedding Provider Abstraction] --> R6
    R3[Req 3: BERT Entity CLS Embeddings] --> R5
    R4[Req 4: Relationship Extraction Endpoint] --> R5
    R5[Req 5: Direct Neo4j Write Pipeline] --> R6
    R5 --> R7
    R5 --> R9
    R6[Req 6: Vector Index + Entity Resolution] --> R7
    R6 --> R10
    R7[Req 7: Enhanced Graph Retriever] --> R8
    R7 --> R10
    R8[Req 8: Modular Ensemble Search]
    R9[Req 9: Document Content Graph] --> R10
    R10[Req 10: Graph-Guided Hybrid Retrieval] --> R11
    R10 --> R12
    R11[Req 11: Community Detection]
    R12[Req 12: Text2Cypher]

    R13[Req 13: Postgres Independence<br/>Cross-Cutting] -.-> R5
    R13 -.-> R8
    R14[Req 14: Graceful Degradation<br/>Cross-Cutting] -.-> R5
    R14 -.-> R7
    R14 -.-> R8

    style R1 fill:#c8e6c9
    style R2 fill:#c8e6c9
    style R3 fill:#c8e6c9
    style R4 fill:#c8e6c9
    style R13 fill:#ffecb3
    style R14 fill:#ffecb3

    classDef foundation fill:#c8e6c9,stroke:#388e3c
    classDef crosscut fill:#ffecb3,stroke:#f57c00

Loading

Critical path: R1 → R5 → R6 → R7 → R10 → R11/R12

Parallelizable foundations (no dependencies): R1, R2, R3, R4 can all be built simultaneously.

---

Components and Interfaces

1. Sidecar API Contracts

POST /extract-entities (Enhanced)

Existing endpoint enhanced with optional CLS embeddings. Backward compatible.

// Request (unchanged)
interface ExtractEntitiesRequest {
  chunks: string[];
}

// Response when called WITHOUT ?include_embeddings=true (unchanged)
interface ExtractEntitiesResponse {
  results: { text: string; entities: { text: string; label: string; score: number }[] }[];
  total_entities: number;
}

// Response when called WITH ?include_embeddings=true (enhanced)
interface ExtractEntitiesEnhancedResponse {
  results: {
    text: string;
    entities: {
      text: string;
      label: string;
      score: number;
      embedding: number[]; // 768-dim BERT CLS vector
    }[];
  }[];
  total_entities: number;
}

POST /extract-relationships (New)

Model-agnostic relationship extraction via any OpenAI-compatible LLM.

interface ExtractRelationshipsRequest {
  chunks: string[];
  known_entities?: string[]; // Optional: constrain relationship targets
}

interface ExtractionEntity {
  name: string;
  type: "PERSON" | "ORGANIZATION" | "LOCATION" | "PRODUCT" | "EVENT" | "OTHER";
}

interface ExtractionRelationship {
  source: string;   // Must match an entity name
  target: string;   // Must match an entity name
  type: string;     // SCREAMING_SNAKE_CASE: ^[A-Z][A-Z0-9_]*$
  detail: string;   // Brief evidence description
}

interface ExtractionChunkResult {
  text: string;
  entities: ExtractionEntity[];
  relationships: ExtractionRelationship[];
  dropped_relationships: ExtractionRelationship[]; // Invalid source/target
}

interface ExtractRelationshipsResponse {
  results: ExtractionChunkResult[];
  total_entities: number;
  total_relationships: number;
  total_dropped: number;
}

2. Embedding Provider Interface

/** Model-agnostic embedding provider — all sub-features use this interface */
interface EmbeddingProvider {
  /** Embed a single text string */
  embed(text: string): Promise<number[]>;
  /** Embed multiple texts in a batch */
  embedBatch(texts: string[]): Promise<number[][]>;
  /** The dimensionality of output vectors */
  readonly dimensions: number;
  /** The provider name for logging */
  readonly providerName: string;
}

/** Factory: reads env vars, returns the appropriate provider */
function createEmbeddingProvider(): EmbeddingProvider;
// - No EMBEDDING_PROVIDER or EMBEDDING_PROVIDER=bert → BertEmbeddingProvider (768-dim, calls Sidecar /embed)
// - EMBEDDING_PROVIDER=openai-compatible → OpenAICompatibleEmbeddingProvider (configurable dim)
// - Fallback: if external API unreachable → BertEmbeddingProvider with warning log

3. Neo4j Direct Writer Interface

/** Replaces the old neo4j-sync.ts "read from Postgres" pattern */
interface Neo4jDirectWriter {
  /** Write entities directly to Neo4j (idempotent MERGE) */
  writeEntities(entities: Neo4jEntityInput[], companyId: string): Promise<number>;
  /** Write relationships with dynamic Cypher types (idempotent MERGE) */
  writeRelationships(relationships: Neo4jRelationshipInput[], companyId: string): Promise<string[]>;
  /** Write section nodes and MENTIONED_IN edges */
  writeMentions(mentions: Neo4jMentionInput[], companyId: string): Promise<number>;
  /** Write document content graph nodes (Document, Topic, cross-doc links) */
  writeDocumentGraph(doc: Neo4jDocumentGraphInput, companyId: string): Promise<void>;
  /** Ensure vector indexes exist (idempotent) */
  ensureIndexes(): Promise<void>;
}

interface Neo4jEntityInput {
  name: string;           // normalized lowercase
  displayName: string;    // original casing
  label: string;          // PER, ORG, LOC, PRODUCT, EVENT, MISC, OTHER
  confidence: number;
  mentionCount: number;
  companyId: string;
  embedding?: number[];   // 768-dim BERT CLS vector (nullable)
}

interface Neo4jRelationshipInput {
  sourceName: string;
  sourceLabel: string;
  targetName: string;
  targetLabel: string;
  relationType: string;   // SCREAMING_SNAKE_CASE dynamic type
  weight: number;
  evidenceCount: number;
  detail?: string;
  documentId: number;
  companyId: string;
}

interface Neo4jMentionInput {
  entityName: string;
  entityLabel: string;
  sectionId: number;
  documentId: number;
  confidence: number;
  companyId: string;
}

interface Neo4jWriteResult {
  entities: number;
  mentions: number;
  relationships: number;
  dynamicRelTypes: string[];
  durationMs: number;
}

---

Data Models

Neo4j Graph Schema

-- ═══════════════════════════════════════════════════════════════
-- NODE TYPES
-- ═══════════════════════════════════════════════════════════════

-- Entity node (core knowledge graph node)
(:Entity {
  name: String,            -- normalized lowercase (MERGE key)
  displayName: String,     -- original casing
  label: String,           -- PER, ORG, LOC, PRODUCT, EVENT, MISC, OTHER (MERGE key)
  confidence: Float,       -- average extraction confidence
  mentionCount: Integer,   -- total mentions across all documents
  companyId: String,       -- company scope (MERGE key)
  embedding: List<Float>,  -- 768-dim BERT CLS vector (nullable)
  communityId: Integer     -- Leiden community assignment (nullable, Req 11)
})

-- Section node (lightweight — content stays in Postgres)
(:Section {
  id: Integer,             -- matches documentSections.id in Postgres
  documentId: Integer      -- matches document.id in Postgres
})

-- Document node (Req 9: content graph)
(:Document {
  id: Integer,             -- matches document.id in Postgres
  name: String,            -- document name
  companyId: String,       -- company scope
  uploadedAt: String       -- ISO timestamp
})

-- Topic node (Req 9: content graph)
(:Topic {
  name: String,            -- topic name
  companyId: String,       -- company scope
  embedding: List<Float>   -- 768-dim embedding for similarity
})

-- Community node (Req 11: community detection)
(:Community {
  id: Integer,             -- community identifier
  summary: String,         -- LLM-generated 2-3 sentence summary
  companyId: String,       -- company scope
  embedding: List<Float>   -- embedding of the summary text
})

-- ═══════════════════════════════════════════════════════════════
-- RELATIONSHIP TYPES
-- ═══════════════════════════════════════════════════════════════

-- Entity ↔ Section (mentions)
(:Entity)-[:MENTIONED_IN {confidence: Float}]->(:Section)

-- Entity ↔ Entity (dynamic types — NOT generic RELATES_TO)
-- Examples: CEO_OF, ACQUIRED, HEADQUARTERED_IN, COMPETES_WITH, CO_OCCURS
(:Entity)-[:<DYNAMIC_TYPE> {
  weight: Float,
  evidenceCount: Integer,
  detail: String,          -- evidence text (nullable)
  documentId: Integer      -- source document
}]->(:Entity)

-- Entity resolution (Req 6)
(:Entity)-[:ALIAS_OF]->(:Entity)  -- alias → canonical

-- Document content graph (Req 9)
(:Document)-[:CONTAINS]->(:Section)
(:Section)-[:DISCUSSES]->(:Topic)
(:Document)-[:REFERENCES {sharedEntityCount: Integer}]->(:Document)
(:Topic)-[:RELATED_TO]->(:Topic)

-- Community membership (Req 11)
(:Entity)-[:BELONGS_TO]->(:Community)

-- ═══════════════════════════════════════════════════════════════
-- VECTOR INDEXES
-- ═══════════════════════════════════════════════════════════════

-- Entity embeddings (768-dim, cosine similarity)
CREATE VECTOR INDEX `entity-embeddings` IF NOT EXISTS
FOR (e:Entity) ON (e.embedding)
OPTIONS {indexConfig: {`vector.dimensions`: 768, `vector.similarity_function`: 'cosine'}};

-- Topic embeddings (768-dim, cosine similarity) — Req 9
CREATE VECTOR INDEX `topic-embeddings` IF NOT EXISTS
FOR (t:Topic) ON (t.embedding)
OPTIONS {indexConfig: {`vector.dimensions`: 768, `vector.similarity_function`: 'cosine'}};

-- Community summary embeddings — Req 11
CREATE VECTOR INDEX `community-embeddings` IF NOT EXISTS
FOR (c:Community) ON (c.embedding)
OPTIONS {indexConfig: {`vector.dimensions`: 768, `vector.similarity_function`: 'cosine'}};

Shared Zod Schemas (Parent-Level Contracts)

These are the runtime-validated contracts that ALL sub-feature specs must conform to. Updated from the old gemma-schemas.ts with model-agnostic naming and new node types.

import { z } from "zod";

// ═══════════════════════════════════════════════════════════════
// SIDECAR API SCHEMAS
// ═══════════════════════════════════════════════════════════════

// ── BERT Entity (base, without embedding) ────────────────────

export const EntityBaseSchema = z.object({
  text: z.string().min(1),
  label: z.string().min(1),
  score: z.number().min(0).max(1),
});

// ── BERT Entity (with CLS embedding) ────────────────────────

export const EntityWithEmbeddingSchema = EntityBaseSchema.extend({
  embedding: z.array(z.number()).length(768),
});

// ── Extract Entities Response (base, backward compatible) ────

export const ExtractEntitiesResponseSchema = z.object({
  results: z.array(z.object({
    text: z.string(),
    entities: z.array(EntityBaseSchema),
  })),
  total_entities: z.number().int().nonnegative(),
});

// ── Extract Entities Enhanced Response (with embeddings) ─────

export const ExtractEntitiesEnhancedResponseSchema = z.object({
  results: z.array(z.object({
    text: z.string(),
    entities: z.array(EntityWithEmbeddingSchema),
  })),
  total_entities: z.number().int().nonnegative(),
});

// ── Extraction Entity (LLM-extracted, model-agnostic) ────────

export const ExtractionEntitySchema = z.object({
  name: z.string().min(1),
  type: z.enum(["PERSON", "ORGANIZATION", "LOCATION", "PRODUCT", "EVENT", "OTHER"]),
});

// ── Extraction Relationship ──────────────────────────────────

export const ExtractionRelationshipSchema = z.object({
  source: z.string().min(1),
  target: z.string().min(1),
  type: z.string().min(1).regex(/^[A-Z][A-Z0-9_]*$/), // SCREAMING_SNAKE_CASE
  detail: z.string(),
});

// ── Extraction Chunk Result ──────────────────────────────────

export const ExtractionChunkResultSchema = z.object({
  text: z.string(),
  entities: z.array(ExtractionEntitySchema),
  relationships: z.array(ExtractionRelationshipSchema),
  dropped_relationships: z.array(ExtractionRelationshipSchema),
});

// ── Extract Relationships Response ───────────────────────────

export const ExtractRelationshipsResponseSchema = z.object({
  results: z.array(ExtractionChunkResultSchema),
  total_entities: z.number().int().nonnegative(),
  total_relationships: z.number().int().nonnegative(),
  total_dropped: z.number().int().nonnegative(),
});

// ═══════════════════════════════════════════════════════════════
// NEO4J DATA SHAPE SCHEMAS
// ═══════════════════════════════════════════════════════════════

// ── Neo4j Entity Node ────────────────────────────────────────

export const Neo4jEntityNodeSchema = z.object({
  name: z.string().min(1),
  displayName: z.string().min(1),
  label: z.string().min(1),
  confidence: z.number().min(0).max(1),
  mentionCount: z.number().int().positive(),
  companyId: z.string().min(1),
  embedding: z.array(z.number()).length(768).nullable(),
});

// ── Neo4j Section Node ───────────────────────────────────────

export const Neo4jSectionNodeSchema = z.object({
  id: z.number().int().positive(),
  documentId: z.number().int().positive(),
});

// ── Neo4j Document Node (Req 9) ──────────────────────────────

export const Neo4jDocumentNodeSchema = z.object({
  id: z.number().int().positive(),
  name: z.string().min(1),
  companyId: z.string().min(1),
  uploadedAt: z.string().min(1),
});

// ── Neo4j Topic Node (Req 9) ─────────────────────────────────

export const Neo4jTopicNodeSchema = z.object({
  name: z.string().min(1),
  companyId: z.string().min(1),
  embedding: z.array(z.number()).length(768),
});

// ── Neo4j Community Node (Req 11) ────────────────────────────

export const Neo4jCommunityNodeSchema = z.object({
  id: z.number().int().nonnegative(),
  summary: z.string().min(1),
  companyId: z.string().min(1),
  embedding: z.array(z.number()).length(768),
});

// ── Neo4j Relationship Properties ────────────────────────────

export const Neo4jDynamicRelPropertiesSchema = z.object({
  weight: z.number().min(0).max(1),
  evidenceCount: z.number().int().positive(),
  detail: z.string().nullable(),
  documentId: z.number().int().positive(),
});

// ── Neo4j Write Result ───────────────────────────────────────

export const Neo4jWriteResultSchema = z.object({
  entities: z.number().int().nonnegative(),
  mentions: z.number().int().nonnegative(),
  relationships: z.number().int().nonnegative(),
  dynamicRelTypes: z.array(z.string()),
  durationMs: z.number().nonnegative(),
});

// ═══════════════════════════════════════════════════════════════
// EMBEDDING PROVIDER SCHEMA
// ═══════════════════════════════════════════════════════════════

export const EmbeddingResultSchema = z.object({
  embedding: z.array(z.number()).min(1),
  dimensions: z.number().int().positive(),
  providerName: z.string().min(1),
});

export const EmbeddingBatchResultSchema = z.object({
  embeddings: z.array(z.array(z.number()).min(1)),
  dimensions: z.number().int().positive(),
  providerName: z.string().min(1),
});

// ═══════════════════════════════════════════════════════════════
// ENSEMBLE SEARCH SCHEMAS
// ═══════════════════════════════════════════════════════════════

export const GraphRetrieverResultSchema = z.object({
  sectionIds: z.array(z.number().int().positive()),
  entityMatchCount: z.number().int().nonnegative(),
  traversalHops: z.number().int().nonnegative(),
  durationMs: z.number().nonnegative(),
});

// ═══════════════════════════════════════════════════════════════
// TYPE EXPORTS
// ═══════════════════════════════════════════════════════════════

export type EntityBase = z.infer<typeof EntityBaseSchema>;
export type EntityWithEmbedding = z.infer<typeof EntityWithEmbeddingSchema>;
export type ExtractionEntity = z.infer<typeof ExtractionEntitySchema>;
export type ExtractionRelationship = z.infer<typeof ExtractionRelationshipSchema>;
export type ExtractionChunkResult = z.infer<typeof ExtractionChunkResultSchema>;
export type ExtractRelationshipsResponse = z.infer<typeof ExtractRelationshipsResponseSchema>;
export type Neo4jEntityNode = z.infer<typeof Neo4jEntityNodeSchema>;
export type Neo4jDocumentNode = z.infer<typeof Neo4jDocumentNodeSchema>;
export type Neo4jTopicNode = z.infer<typeof Neo4jTopicNodeSchema>;
export type Neo4jCommunityNode = z.infer<typeof Neo4jCommunityNodeSchema>;
export type Neo4jWriteResult = z.infer<typeof Neo4jWriteResultSchema>;
export type EmbeddingResult = z.infer<typeof EmbeddingResultSchema>;

Environment Variable Contract

All Neo4j-related environment variables across all requirements, with validation rules:

// ═══════════════════════════════════════════════════════════════
// ENVIRONMENT VARIABLE CONTRACT
// ═══════════════════════════════════════════════════════════════
//
// All variables are OPTIONAL — the system works without any of them.
// Each variable gates a specific capability.
//
// ┌─────────────────────────────────┬──────────┬─────────────────────────────────────────┐
// │ Variable                        │ Required │ Gates                                   │
// ├─────────────────────────────────┼──────────┼─────────────────────────────────────────┤
// │ NEO4J_URI                       │ No       │ All Neo4j operations (Req 1,5,6,7,8,9)  │
// │ NEO4J_USERNAME                  │ No       │ Neo4j auth (default: "neo4j")            │
// │ NEO4J_PASSWORD                  │ No       │ Neo4j auth (default: "password")         │
// │ EXTRACTION_LLM_BASE_URL         │ No       │ Relationship extraction (Req 4,5)        │
// │ EXTRACTION_LLM_MODEL            │ No       │ LLM model selection (Req 4)              │
// │ EMBEDDING_PROVIDER              │ No       │ Embedding backend: "bert" | "openai-compatible" │
// │ EMBEDDING_API_URL               │ No       │ External embedding API (Req 2)           │
// │ EMBEDDING_MODEL                 │ No       │ External embedding model name (Req 2)    │
// │ EMBEDDING_DIMENSIONS            │ No       │ External embedding dimensions (Req 2)    │
// │ ENABLE_GRAPH_RETRIEVER          │ No       │ Graph retriever in ensemble (Req 7,8)    │
// │ ENTITY_RESOLUTION_THRESHOLD     │ No       │ Cosine similarity threshold (default: 0.85) │
// │ SIDECAR_URL                     │ No       │ All sidecar operations (Req 3,4)         │
// └─────────────────────────────────┴──────────┴─────────────────────────────────────────┘
//
// GATING LOGIC:
//   - NEO4J_URI unset        → skip ENTIRE graph pipeline (Steps F, F2, G, G2, G3, G4)
//                               No entity extraction, no relationship extraction,
//                               no graph data stored anywhere. Pipeline ends at Step E.
//   - NEO4J_URI set + EXTRACTION_LLM_BASE_URL unset → skip relationship extraction only,
//                               graph has BERT entities with CO_OCCURS relationships
//   - NEO4J_URI set + SIDECAR_URL unset → skip entity extraction, no graph data
//   - ENABLE_GRAPH_RETRIEVER=false → ensemble uses only BM25+Vector (2-retriever)
//   - EMBEDDING_PROVIDER unset → default to BERT via Sidecar (768-dim, free)
//
// MIGRATION FROM OLD SPEC:
//   - GEMMA_BASE_URL → EXTRACTION_LLM_BASE_URL (model-agnostic naming)
//   - GEMMA_MODEL    → EXTRACTION_LLM_MODEL    (model-agnostic naming)
//   - Old vars remain in src/env.ts for backward compatibility during transition

// Zod validation schema additions for src/env.ts:
const neo4jGraphRAGEnvSchema = z.object({
  // Neo4j connection (optional — enables graph storage)
  NEO4J_URI: optionalString(),
  NEO4J_USERNAME: optionalString(),
  NEO4J_PASSWORD: optionalString(),

  // Extraction LLM (optional — enables relationship extraction)
  // Model-agnostic: works with Ollama, LM Studio, OpenAI, Azure, any OpenAI-compatible
  EXTRACTION_LLM_BASE_URL: optionalString(),
  EXTRACTION_LLM_MODEL: optionalString(),

  // Embedding provider (optional — defaults to BERT via Sidecar)
  EMBEDDING_PROVIDER: z.enum(["bert", "openai-compatible"]).optional(),
  EMBEDDING_API_URL: optionalString(),
  EMBEDDING_MODEL: optionalString(),
  EMBEDDING_DIMENSIONS: z.coerce.number().int().positive().optional(),

  // Graph retriever toggle
  ENABLE_GRAPH_RETRIEVER: z.preprocess(
    (val) => val === "true" || val === "1",
    z.boolean().optional()
  ),

  // Entity resolution
  ENTITY_RESOLUTION_THRESHOLD: z.coerce.number().min(0).max(1).optional(),

  // Backward compatibility (old spec names — deprecated)
  GEMMA_BASE_URL: optionalString(),
  GEMMA_MODEL: optionalString(),
});

Ingestion Pipeline Contract

Step ordering with inputs/outputs and gating logic:

═══════════════════════════════════════════════════════════════
EXISTING STEPS (A-E) — UNCHANGED, NEVER MODIFIED
═══════════════════════════════════════════════════════════════

Step A: Upload
  Input:  File (PDF, DOCX, etc.)
  Output: Raw file stored, ocrJob created
  Gate:   Always runs

Step B: OCR / Normalize
  Input:  Raw file
  Output: PageContent[] (text per page)
  Gate:   Always runs

Step C: Chunk
  Input:  PageContent[]
  Output: DocumentChunk[] (parent + child chunks)
  Gate:   Always runs

Step D: Embed
  Input:  DocumentChunk[]
  Output: VectorizedChunk[] (with 1536-dim OpenAI embeddings)
  Gate:   Always runs (uses OpenAI or Sidecar)

Step E: Store Sections
  Input:  VectorizedChunk[]
  Output: StoredSection[] {sectionId, content} in Postgres
  Gate:   Always runs

═══════════════════════════════════════════════════════════════
NEO4J GRAPH PIPELINE — ALL GATED BY NEO4J_URI
═══════════════════════════════════════════════════════════════

When NEO4J_URI is NOT set, NONE of these steps run. No entity
extraction, no relationship extraction, no graph data stored
anywhere. The pipeline ends at Step E.

Step F: BERT NER Entity Extraction
  Input:  StoredSection[] (from Step E)
  Output: Entities + CO_OCCURS relationships + CLS embeddings
  Gate:   NEO4J_URI is set AND SIDECAR_URL is set AND sidecar is healthy
  Target: Passed directly to Step G (Neo4j) — NOT stored in Postgres

Step F2: LLM Relationship Extraction (NEW)
  Input:  StoredSection[] (same chunks as Step F)
  Output: Typed relationships (CEO_OF, ACQUIRED, etc.)
          + additional entities from LLM
  Gate:   NEO4J_URI is set AND EXTRACTION_LLM_BASE_URL is set AND sidecar is healthy
  Target: Passed directly to Step G (Neo4j) — NOT stored in Postgres
  Note:   Independent of Step F — if BERT fails, LLM still runs

Step G: Neo4j Direct Write
  Input:  All entities + relationships from Steps F and F2
          (passed directly in memory, NOT read from Postgres)
  Output: Neo4jWriteResult
  Gate:   NEO4J_URI is set AND Neo4j is healthy
  Target: Neo4j graph store (direct MERGE queries)
  Note:   Neo4j is the ONLY place graph data lives.
          Postgres kg_* tables are NOT used by this pipeline.

Step G2: Document Content Graph (NEW, Req 9)
  Input:  Document metadata + Section IDs + extracted topics
  Output: Document, Topic nodes + CONTAINS, DISCUSSES, REFERENCES edges
  Gate:   NEO4J_URI is set AND Neo4j is healthy
  Target: Neo4j graph store

Step G3: Entity Resolution (NEW, Req 6)
  Input:  Newly written entities with embeddings
  Output: Merged entities + ALIAS_OF relationships
  Gate:   NEO4J_URI is set AND entity-embeddings vector index exists
  Target: Neo4j graph store

Step G4: Community Detection (NEW, Req 11)
  Input:  Entity graph in Neo4j
  Output: Community assignments + Community summary nodes
  Gate:   NEO4J_URI is set AND GDS plugin available
  Target: Neo4j graph store

═══════════════════════════════════════════════════════════════
FAILURE ISOLATION
═══════════════════════════════════════════════════════════════

- Step F failure  → Step F2 still attempts to run, Step G writes whatever data is available
- Step F2 failure → Step G still writes BERT-only data from Step F
- Step G failure  → Graph data is lost for this document (nowhere else to store it)
                    Document is still marked as successfully ingested (Postgres Steps A-E are source of truth)
- Step G2 failure → Entity graph is still valid, just no content graph
- Step G3 failure → Entities exist but may have duplicates
- Step G4 failure → Graph works, just no community summaries

Ensemble Search Contract

How the graph retriever plugs into the existing ensemble:

// ═══════════════════════════════════════════════════════════════
// ENSEMBLE SEARCH INTEGRATION
// ═══════════════════════════════════════════════════════════════

// Weight configurations
const WEIGHTS_2_RETRIEVER = [0.4, 0.6];           // BM25, Vector (current default)
const WEIGHTS_3_RETRIEVER = [0.3, 0.5, 0.2];      // BM25, Vector, Graph (static default)

// Dynamic weight adjustment (Req 8.4)
// When graph retriever returns results, adjust weights based on match quality:
//   entityMatchCount >= 3 → graph weight increases to 0.3-0.5
//   entityMatchCount 1-2  → graph weight stays at 0.1-0.2
//   entityMatchCount 0    → graph weight drops to 0, effectively 2-retriever mode

interface GraphRetrieverMetrics {
  entityMatchCount: number;
  avgConfidence: number;
  traversalHops: number;
}

function computeDynamicWeights(metrics: GraphRetrieverMetrics): [number, number, number] {
  if (metrics.entityMatchCount >= 3 && metrics.avgConfidence > 0.7) {
    return [0.15, 0.35, 0.50]; // Strong graph signal
  }
  if (metrics.entityMatchCount >= 1) {
    return [0.30, 0.50, 0.20]; // Moderate graph signal
  }
  return [0.40, 0.60, 0.00];   // No graph signal → 2-retriever fallback
}

// Graceful degradation interface
// The ensemble MUST handle these failure modes without user-visible errors:
//   1. Neo4j unreachable → use 2-retriever weights, log warning
//   2. Graph retriever timeout (>2s) → use 2-retriever weights, log warning
//   3. Graph retriever returns empty → use 2-retriever weights (normal behavior)
//   4. Sidecar unreachable → skip reranking, return RRF-fused results

---

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: Embedding output normalization

For any valid text input and any configured embedding backend (BERT or OpenAI-compatible), the EmbeddingProvider.embed() method SHALL return an array of floats with length equal to provider.dimensions, and EmbeddingProvider.embedBatch() SHALL return arrays all of the same dimensionality.

Validates: Requirements 2.1, 2.4

Property 2: BERT CLS embedding dimensionality

For any text chunk that produces at least one entity, when /extract-entities?include_embeddings=true is called, every entity in the response SHALL have an embedding field that is an array of exactly 768 floating-point numbers.

Validates: Requirements 3.1, 3.3

Property 3: Relationship extraction structural validity

For any extraction response from POST /extract-relationships, every relationship in the relationships array SHALL have source and target values that each match the name of an entity in the same chunk's entities array, AND every relationship type SHALL match the regex ^[A-Z][A-Z0-9_]*$.

Validates: Requirements 4.2, 4.3

Property 4: Think-tag stripping preserves JSON content

For any string containing <think>...</think> tags wrapping arbitrary text, followed by valid JSON content, the think-tag stripping function SHALL produce a string that is valid JSON-parseable.

Validates: Requirement 4.7

Property 5: Neo4j entity write completeness

For any entity written to Neo4j via the direct writer, the resulting Entity node SHALL contain all required properties (name, displayName, label, confidence, mentionCount, companyId), and when the input entity has a non-null embedding, the node SHALL have an embedding property of length 768. For any entity mention, there SHALL exist a MENTIONED_IN edge from the Entity node to the corresponding Section node.

Validates: Requirements 5.3, 5.5, 6.1

Property 6: Dynamic Cypher relationship types

For any relationship written to Neo4j, the Cypher relationship type SHALL be the actual relationship type string (e.g., CEO_OF, ACQUIRED, CO_OCCURS) and SHALL NOT be a generic RELATES_TO type with a type property.

Validates: Requirement 5.4

Property 7: Neo4j write idempotence

For any set of entities and relationships, writing them to Neo4j twice via the direct writer SHALL produce the same graph state (same node count, same edge count, same property values) as writing them once.

Validates: Requirement 5.7

Property 8: Entity resolution merges duplicates

For any two Entity nodes within the same companyId whose embedding cosine similarity exceeds the configured threshold (default 0.85), the entity resolution module SHALL merge them into a single canonical Entity node and create an ALIAS_OF relationship from the alias to the canonical entity.

Validates: Requirements 6.3, 6.4

Property 9: Graph retriever traverses all relationship types

For any Neo4j graph containing entities connected by multiple relationship types (e.g., CEO_OF, ACQUIRED, CO_OCCURS), the graph retriever SHALL traverse all relationship types when finding connected entities — not just CO_OCCURS.

Validates: Requirement 7.1

Property 10: Dynamic ensemble weight adjustment

For any graph retriever result with entityMatchCount >= 3 and avgConfidence > 0.7, the ensemble SHALL assign a graph weight of at least 0.3. For any result with entityMatchCount == 0, the graph weight SHALL be 0 (effectively 2-retriever mode).

Validates: Requirement 8.4

Property 11: Document content graph structure

For any document ingested with Neo4j enabled, there SHALL exist a Document node with the correct id, name, companyId, and uploadedAt properties, and for each section belonging to that document, there SHALL exist a CONTAINS relationship from the Document node to the Section node.

Validates: Requirements 9.1, 9.2

Property 12: Cross-document and topic linking

For any two documents within the same companyId that share 3 or more entities, there SHALL exist a REFERENCES relationship between their Document nodes with a sharedEntityCount property. For any two Topic nodes within the same companyId with embedding cosine similarity above 0.8, there SHALL exist a RELATED_TO relationship.

Validates: Requirements 9.4, 9.5

Property 13: Combined graph+vector scoring

For any retrieval result from the graph-guided hybrid retriever, the result score SHALL incorporate both graph proximity (inversely proportional to hop distance) and vector similarity (cosine similarity to query embedding).

Validates: Requirement 10.3

Property 14: Community node data shape

For any detected community, there SHALL exist a Community node in Neo4j with id, summary (non-empty string), companyId, and embedding (768-dim float array) properties.

Validates: Requirement 11.3

Property 15: Text2Cypher read-only validation

For any Cypher query generated by the Text2Cypher module, the read-only validator SHALL reject queries containing CREATE, DELETE, SET, MERGE, or DETACH keywords (case-insensitive), preventing accidental graph mutations.

Validates: Requirement 12.4

Property 16: Postgres search independence

For any search query executed against the same dataset, the BM25+Vector retrieval results (excluding the graph retriever contribution) SHALL be identical whether Neo4j is enabled or disabled — Neo4j is purely additive and never modifies Postgres search behavior.

Validates: Requirement 13.1

---

Error Handling

Graceful Degradation Cascade

Every Neo4j feature follows a strict degradation hierarchy. No optional component failure should ever cause a user-visible error or block document ingestion.

Component Health Check Order (at pipeline start):
  1. Sidecar (/health)     → gates Steps F, F2
  2. Neo4j (RETURN 1)      → gates Steps G, G2, G3, G4
  3. Extraction LLM (/health or first call) → gates Step F2
  4. GDS plugin (CALL gds.graph.list()) → gates Step G4

Failure Handling:
┌──────────────────────────┬────────────────────────────────────────────┐
│ Component Down           │ Behavior                                  │
├──────────────────────────┼────────────────────────────────────────────┤
│ NEO4J_URI not set        │ Entire graph pipeline disabled            │
│                          │ No entity extraction, no relationships    │
│                          │ No graph data stored anywhere             │
│                          │ Pipeline ends at Step E (Postgres only)   │
├──────────────────────────┼────────────────────────────────────────────┤
│ Sidecar unreachable      │ Skip entity extraction (Step F)           │
│ (NEO4J_URI is set)       │ No graph data for this document           │
│                          │ Document still ingested (Steps A-E ok)    │
├──────────────────────────┼────────────────────────────────────────────┤
│ Neo4j unreachable        │ Skip Neo4j writes (Step G)                │
│ (NEO4J_URI is set)       │ Entity extraction still runs but data     │
│                          │ is discarded (nowhere to write it)        │
│                          │ Ensemble search uses 2-retriever mode     │
├──────────────────────────┼────────────────────────────────────────────┤
│ Extraction LLM down      │ Skip relationship extraction (Step F2)    │
│                          │ Graph has BERT entities + CO_OCCURS only  │
├──────────────────────────┼────────────────────────────────────────────┤
│ GDS plugin unavailable   │ Skip community detection (Step G4)        │
│                          │ All other graph operations work normally  │
├──────────────────────────┼────────────────────────────────────────────┤
│ Vector index missing     │ Skip entity resolution (Step G3)          │
│                          │ Entities may have duplicates              │
├──────────────────────────┼────────────────────────────────────────────┤
│ Neo4j down at query time │ Graph retriever returns empty []          │
│                          │ Ensemble falls back to BM25+Vector        │
│                          │ No user-visible error                     │
├──────────────────────────┼────────────────────────────────────────────┤
│ External embedding API   │ Fall back to BERT via Sidecar             │
│ unreachable              │ Log warning, continue with 768-dim        │
└──────────────────────────┴────────────────────────────────────────────┘

Error Logging Convention

All Neo4j-related errors use structured log prefixes for easy filtering:

[Neo4jWriter]     — Direct write operations
[Neo4jRetriever]  — Query-time graph retrieval
[EntityResolution] — Duplicate entity merging
[ExtractionLLM]   — Relationship extraction via LLM
[EmbeddingProvider] — Embedding generation
[CommunityDetect] — Leiden clustering and summaries
[Text2Cypher]     — Natural language to Cypher translation

---

Testing Strategy

Dual Testing Approach

• Unit tests: Specific examples, edge cases, error conditions, graceful degradation paths
• Property tests: Universal properties across all inputs (fast-check, minimum 100 iterations)
• Contract tests: Zod schema validation — runnable today with no infrastructure
• Integration tests: Cross-component data flow with mocked boundaries

Parent Contract Test Strategy

These test files validate parent-level contracts. They are runnable immediately with no Neo4j, no Sidecar, no LLM — just Zod schema validation.

__tests__/api/graphrag/contracts/
├── graphrag-schemas.ts                    # Shared Zod schemas (source of truth)
├── contract.extract-entities.test.ts      # BERT entity response shape validation
├── contract.extract-relationships.test.ts # LLM extraction response shape validation
├── contract.neo4j-write-result.test.ts    # Neo4j write result shape validation
├── contract.neo4j-node-shapes.test.ts     # Entity, Section, Document, Topic, Community node shapes
├── contract.embedding-provider.test.ts    # Embedding result shape validation
├── contract.env-vars.test.ts              # Environment variable schema validation
└── contract.ensemble-search.test.ts       # Graph retriever result shape validation

Test File	What It Validates	Runs Against
contract.extract-entities.test.ts	BERT entity response (base + enhanced with embeddings)	ExtractEntitiesResponseSchema, ExtractEntitiesEnhancedResponseSchema
contract.extract-relationships.test.ts	LLM extraction response, entity/relationship shapes, SCREAMING_SNAKE_CASE, dropped_relationships	ExtractRelationshipsResponseSchema, ExtractionEntitySchema, ExtractionRelationshipSchema
contract.neo4j-write-result.test.ts	Neo4j write result shape, dynamicRelTypes array	Neo4jWriteResultSchema
contract.neo4j-node-shapes.test.ts	All Neo4j node types: Entity, Section, Document, Topic, Community	Neo4jEntityNodeSchema, Neo4jSectionNodeSchema, Neo4jDocumentNodeSchema, Neo4jTopicNodeSchema, Neo4jCommunityNodeSchema
contract.embedding-provider.test.ts	Embedding result shape, batch result shape, dimensionality	EmbeddingResultSchema, EmbeddingBatchResultSchema
contract.env-vars.test.ts	All new env vars parse correctly, optional vars accept undefined, backward compat with GEMMA_* vars	neo4jGraphRAGEnvSchema
contract.ensemble-search.test.ts	Graph retriever result shape, dynamic weight computation	GraphRetrieverResultSchema

Agent hook: On sub-feature spec completion, the agent runs pnpm test -- __tests__/api/graphrag/contracts/ to validate all parent contracts still pass.

Property-Based Test Configuration

• Library: fast-check (already in devDependencies)
• Minimum 100 iterations per property
• Tag format: Feature: neo4j-graphrag, Property {N}: {title}
• Each correctness property maps to exactly one *.pbt.test.ts file
• Property tests are created in sub-feature specs, not in this parent spec

Cross-Cutting Concern Tests

Postgres Independence (Req 13):

• Each sub-feature spec MUST include a test proving that Postgres search results are identical with Neo4j on vs off
• Test pattern: run the same query twice (once with NEO4J_URI set, once without), compare BM25+Vector results

Graceful Degradation (Req 14):

• Each sub-feature spec MUST include tests for every failure mode in the degradation cascade
• Test pattern: mock the failing component, verify the pipeline continues without errors

Model-Agnostic Behavior (Reqs 2, 4):

• Sub-feature specs for Reqs 2 and 4 MUST include tests proving no hardcoded model names or provider URLs
• Test pattern: configure different EXTRACTION_LLM_BASE_URL values, verify the same code path is used

[kien-ship-it]

Implementation Plan: Neo4j GraphRAG (Parent Vision Spec)

Overview

This is a parent vision spec — each task below represents a feature packet that will become its own full sub-spec (requirements → design → tasks → implementation). Tasks are only marked complete when their entire sub-spec is completed. R13 and R14 are cross-cutting concerns enforced in every feature packet, not standalone tasks.

Dependency Summary

PARALLELIZATION MAP
═══════════════════════════════════════════════════════════════

Layer 0 — Foundations (fully parallel, no dependencies):
  R1  Neo4j Docker Infrastructure        ✅ DONE (verified: APOC+GDS plugins, health check, profile gating, env vars)
  R2  Embedding Provider Abstraction      ✅ DONE (graphrag-foundations sub-spec)
  R3  BERT Entity CLS Embeddings          ⬛ contracts validated
  R4  Relationship Extraction Endpoint    ⬛ contracts validated

Layer 1 — Core Pipeline (depends on Layer 0):
  R5  Direct Neo4j Write Pipeline         ← R1, R3, R4

Layer 2 — Indexing & Resolution (depends on Layer 1):
  R6  Vector Index + Entity Resolution    ← R2, R5

Layer 3 — Retrieval (depends on Layer 2):
  R7  Enhanced Graph Retriever            ← R5, R6
  R9  Document Content Graph              ← R5, R6
      (R7 and R9 are parallelizable with each other)

Layer 4 — Integration & Advanced (depends on Layer 3):
  R8  Modular Ensemble Search             ← R7
  R10 Graph-Guided Hybrid Retrieval       ← R6, R7, R9
  R12 Text2Cypher                         ← R5, R9
      (R8, R10, R12 are parallelizable with each other)

Layer 5 — SOTA Features (depends on Layer 4):
  R11 Community Detection                 ← R5, R6, R10

Cross-Cutting (applied to ALL tasks, not standalone):
  R13 Postgres Search Independence
  R14 Graceful Degradation & Opt-In

Tasks

• 1. R1 — Neo4j Docker Infrastructure

  • Scope: Add Neo4j CE to docker-compose.yml with dev/neo4j profiles, APOC+GDS plugins, health check, env var wiring, and .env.example updates. Validate neo4j-client.ts health check works.
  • Status: DONE — All verified: NEO4J_URI fix, env vars, APOC (436 procedures), GDS plugin, health check (healthy), profile gating (neo4j excluded from default), Browser UI (7474 → HTTP 200), Bolt (7687 → ok).
  • Dependencies: None (foundational)
  • Parallelizable with: R2, R3, R4
  • Effort: S (small — mostly verification of existing work)
  • Done when:
    • docker compose --profile dev up starts Neo4j accessible at localhost:7474 and bolt://localhost:7687
    • docker compose up (no profile) does NOT start Neo4j
    • APOC and GDS plugins are loaded (verified via CALL apoc.help('apoc') and CALL gds.graph.list())
    • Health check passes within 60 seconds
    • NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD, EXTRACTION_LLM_BASE_URL, EXTRACTION_LLM_MODEL present in .env.example
    • When NEO4J_URI is unset, no Neo4j operations run (debug log only)
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6

• 2. R2 — Model-Agnostic Embedding Provider Abstraction

  • Scope: Create EmbeddingProvider interface with BertEmbeddingProvider (Sidecar /embed) and OpenAICompatibleEmbeddingProvider implementations. Factory function reads env vars, returns appropriate provider. Fallback to BERT when external API is unreachable.
  • Status: DONE — Delivered by graphrag-foundations sub-spec. Interface, both implementations, factory, env vars, and contract tests all passing.
  • Dependencies: None (foundational)
  • Parallelizable with: R1, R3, R4
  • Effort: M (medium — new abstraction layer with two implementations + fallback logic)
  • Done when:
    • EmbeddingProvider interface exists at src/lib/embeddings/provider.ts
    • Default (no config) returns BertEmbeddingProvider producing 768-dim vectors
    • EMBEDDING_PROVIDER=openai-compatible with EMBEDDING_API_URL returns OpenAICompatibleEmbeddingProvider
    • External API failure falls back to BERT with warning log
    • All output embeddings are normalized to number[] regardless of backend
    • Env vars (EMBEDDING_PROVIDER, EMBEDDING_API_URL, EMBEDDING_MODEL, EMBEDDING_DIMENSIONS) validated in src/env.ts
    • Contract tests pass: EmbeddingResultSchema, EmbeddingBatchResultSchema
  • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6

• 3. R3 — BERT Entity Extraction with CLS Embeddings

  • Scope: Enhance Sidecar /extract-entities endpoint to accept ?include_embeddings=true query param and return 768-dim BERT CLS embedding per entity. Fully backward compatible when param is absent.
  • Status: Contracts validated from old spec experiments. Implementation is new but the API shape is proven.
  • Dependencies: None (enhances existing Sidecar endpoint)
  • Parallelizable with: R1, R2, R4
  • Effort: M (medium — requires extracting CLS/pooled output from BERT hidden states in Python)
  • Done when:
    • /extract-entities without include_embeddings returns existing response shape (no embedding field)
    • /extract-entities?include_embeddings=true returns embedding: number[768] per entity
    • Zero entities from a chunk returns empty array (no error)
    • /health still returns HTTP 200 regardless
    • Contract tests pass: ExtractEntitiesResponseSchema, ExtractEntitiesEnhancedResponseSchema
    • Python unit tests in sidecar/tests/ cover both modes
  • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5

• 4. R4 — Relationship Extraction Endpoint (Model-Agnostic)

  • Scope: Add POST /extract-relationships to the FastAPI sidecar. Uses any OpenAI-compatible LLM via EXTRACTION_LLM_BASE_URL. Structured JSON extraction with think-tag stripping, source/target validation, dropped_relationships reporting. No hardcoded model names.
  • Status: Contracts validated from old spec experiments. Implementation is new but the API shape is proven.
  • Dependencies: None (independent Sidecar endpoint)
  • Parallelizable with: R1, R2, R3
  • Effort: M (medium — LLM prompt engineering, JSON parsing, validation logic)
  • Done when:
    • POST /extract-relationships returns entities and typed relationships per chunk
    • Relationship types are SCREAMING_SNAKE_CASE matching ^[A-Z][A-Z0-9_]*$
    • Invalid source/target relationships moved to dropped_relationships
    • EXTRACTION_LLM_BASE_URL unset or unreachable → HTTP 200 with empty results (no 500)
    • Think-tag <think>...</think> stripping works before JSON parse
    • Temperature 0.1, min max_tokens 4096
    • No hardcoded model names or provider-specific logic
    • /health returns HTTP 200 even when LLM is unavailable
    • Contract tests pass: ExtractRelationshipsResponseSchema
    • Python unit tests in sidecar/tests/ cover happy path, degradation, think-tag stripping
  • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9

• 5. R5 — Direct Neo4j Write Pipeline

  • Scope: Create neo4j-direct-writer.ts that writes entities, relationships, sections, and mentions directly to Neo4j during ingestion. Idempotent MERGE queries, dynamic Cypher relationship types (not generic RELATES_TO), entity merging (BERT + LLM data). New ingestion steps F, F2, G gated by NEO4J_URI.
  • Dependencies: R1 (Neo4j infrastructure), R3 (BERT CLS embeddings), R4 (relationship extraction)
  • Parallelizable with: Nothing — this is the critical path bottleneck after Layer 0
  • Effort: L (large — core pipeline integration, Cypher query design, multi-source entity merging, failure isolation)
  • Done when:
    • NEO4J_URI set + Neo4j healthy → full graph pipeline activates (Steps F, F2, G)
    • NEO4J_URI unset → no entity extraction, no relationship extraction, no graph data anywhere, pipeline ends at Step E
    • Entity nodes written with all required properties (name, displayName, label, confidence, mentionCount, companyId, embedding)
    • Relationships use dynamic Cypher types (MERGE (a)-[r:CEO_OF]->(b)), NOT generic RELATES_TO
    • Section nodes and MENTIONED_IN edges created
    • MERGE queries are idempotent (re-processing doesn't create duplicates)
    • BERT entities + LLM relationships merged into unified graph
    • Neo4j failure during write → warning log, pipeline continues, document marked as ingested
    • Postgres Steps A-E completely unchanged (R13 cross-cutting)
    • All degradation paths work (R14 cross-cutting)
    • Contract tests pass: Neo4jWriteResultSchema, Neo4jEntityNodeSchema, Neo4jSectionNodeSchema
  • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8

• 6. R6 — Neo4j Vector Index and Semantic Entity Resolution

  • Scope: Create entity-embeddings vector index on Entity nodes (768-dim, cosine). Implement entity resolution: on write, query vector index for near-duplicates above configurable threshold, merge into canonical entity with ALIAS_OF provenance.
  • Dependencies: R2 (embedding provider), R5 (direct write pipeline)
  • Parallelizable with: Nothing at this layer — R7 and R9 depend on this
  • Effort: M (medium — vector index creation, cosine similarity queries, merge logic)
  • Done when:
    • entity-embeddings vector index exists on Entity nodes (768-dim, cosine)
    • New entity write triggers vector similarity check within same companyId
    • Entities above threshold (default 0.85, configurable via ENTITY_RESOLUTION_THRESHOLD) are merged
    • Merged entities get ALIAS_OF relationship (alias → canonical)
    • db.index.vector.queryNodes('entity-embeddings', $embedding, $topK) works for nearest-neighbor lookup
    • Vector index creation failure → warning log, continues without indexing
    • Contract tests pass: Neo4jEntityNodeSchema (with embedding)
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6

• 7. R7 — Enhanced Neo4j Graph Retriever

  • Scope: Update neo4j-graph-retriever.ts Cypher queries to traverse ALL relationship types (not just CO_OCCURS). Add optional vector similarity scoring on entity embeddings. Configurable hop depth and topK.
  • Dependencies: R5 (direct write pipeline), R6 (vector index)
  • Parallelizable with: R9 (document content graph)
  • Effort: M (medium — Cypher query redesign, vector similarity integration, fallback paths)
  • Done when:
    • Graph retriever traverses all relationship types (CEO_OF, ACQUIRED, CO_OCCURS, etc.)
    • Optional vector similarity ranking on query embedding when entity embeddings available
    • Section content returned from PostgreSQL (Neo4j stores only IDs/metadata)
    • Neo4j unreachable → empty result set, no error, ensemble continues with BM25+Vector
    • Configurable hop depth (default: 1) and topK (default: 10)
    • Contract tests pass: GraphRetrieverResultSchema
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5

• 8. R8 — Modular Ensemble Search Integration

  • Scope: Harden the graph retriever integration in ensemble-search.ts. Add dynamic weight adjustment based on graph match quality. Ensure graceful degradation is robust.
  • Dependencies: R7 (enhanced graph retriever)
  • Parallelizable with: R10, R12
  • Effort: S (small — mostly hardening existing integration + dynamic weight logic)
  • Done when:
    • ENABLE_GRAPH_RETRIEVER=true + Neo4j healthy → 3-retriever ensemble with graph weight 0.2
    • ENABLE_GRAPH_RETRIEVER unset or Neo4j unconfigured → 2-retriever (0.4 BM25, 0.6 Vector), identical to current behavior
    • Graph retriever failure → 2-retriever fallback with warning log
    • Dynamic weights: ≥3 strong entity matches → graph weight increases; 0 matches → graph weight 0
    • Reranking via Sidecar cross-encoder preserved regardless of graph retriever state
    • Postgres search results identical with Neo4j on vs off (R13 cross-cutting)
  • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5

• 9. R9 — Document Content Graph

  • Scope: Model document hierarchy in Neo4j: Document → Section → Topic nodes. Cross-document REFERENCES edges (shared entities), cross-topic RELATED_TO edges (embedding similarity). Topic extraction via Extraction LLM.
  • Dependencies: R5 (direct write pipeline), R6 (vector index)
  • Parallelizable with: R7 (enhanced graph retriever)
  • Effort: M (medium — new node types, topic extraction via LLM, cross-document linking logic)
  • Done when:
    • Document nodes created with id, name, companyId, uploadedAt
    • CONTAINS relationships from Document → Section
    • Topic nodes with name, companyId, embedding; DISCUSSES from Section → Topic
    • Two documents sharing ≥3 entities → REFERENCES edge with sharedEntityCount
    • Two Topics with embedding cosine similarity >0.8 → RELATED_TO edge
    • Queryable via Cypher: "what documents discuss topic X", "related documents"
    • topic-embeddings vector index created
    • Contract tests pass: Neo4jDocumentNodeSchema, Neo4jTopicNodeSchema
  • Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6

• 10. R10 — Graph-Guided Hybrid Retrieval

  • Scope: Single Cypher query combining graph traversal with vector similarity in Neo4j. Query-time BERT NER for entity extraction. Combined scoring (graph proximity + vector similarity). Fallback to vector-only when no graph matches.
  • Dependencies: R6 (vector index), R7 (enhanced graph retriever), R9 (document content graph)
  • Parallelizable with: R8, R12
  • Effort: L (large — complex Cypher query design, dual scoring algorithm, performance tuning for 100K nodes)
  • Done when:
    • Query entities extracted via BERT NER at query time
    • Single Cypher query: entity match → traverse 1-3 hops → rank by vector similarity
    • Combined score = graph proximity + vector similarity
    • Zero entity matches → fallback to vector-only search within Neo4j
    • Combined query completes within 500ms for graphs up to 100K entity nodes
  • Requirements: 10.1, 10.2, 10.3, 10.4, 10.5

• 11. R11 — Community Detection and Global Queries

  • Scope: Leiden clustering via Neo4j GDS on entity graph. LLM-generated community summaries stored as Community nodes with embeddings. Query classification (global vs local) routing to community summaries or graph-guided retrieval.
  • Dependencies: R5 (direct write pipeline), R6 (vector index), R10 (graph-guided retrieval)
  • Parallelizable with: R12 (if R10 is complete)
  • Effort: L (large — GDS integration, LLM summary generation, query classification, community embedding)
  • Done when:
    • Leiden clustering assigns communityId to Entity nodes within companyId scope
    • Community summaries (2-3 sentences) generated via Extraction LLM
    • Community nodes with id, summary, companyId, embedding stored in Neo4j
    • community-embeddings vector index created
    • Global queries → community summary vector search → aggregated answer
    • Local queries → graph-guided hybrid retrieval (R10)
    • GDS plugin unavailable → skip community detection, warning log, no other impact
    • Contract tests pass: Neo4jCommunityNodeSchema
  • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6

• 12. R12 — Text2Cypher Natural Language Queries

  • Scope: LLM-based natural language to Cypher translation. Schema-aware system prompt. Read-only Cypher validation. Fallback to standard retrieval on Cypher failure. Works with any OpenAI-compatible LLM.
  • Dependencies: R5 (direct write pipeline), R9 (document content graph)
  • Parallelizable with: R8, R10
  • Effort: M (medium — LLM prompt engineering, Cypher validation, schema introspection)
  • Done when:
    • Natural language queries translated to valid Cypher via configured LLM
    • Graph schema (node labels, relationship types, property keys) included in LLM system prompt
    • Generated Cypher validated for read-only (rejects CREATE, DELETE, SET, MERGE, DETACH)
    • Failed Cypher execution → fallback to graph-guided hybrid retrieval + debug log
    • Works with any OpenAI-compatible endpoint via EXTRACTION_LLM_BASE_URL
  • Requirements: 12.1, 12.2, 12.3, 12.4, 12.5

Cross-Cutting Concerns

R13 and R14 are NOT standalone tasks. They are constraints enforced in EVERY feature packet above.

R13 — Postgres Search Independence

Every sub-spec MUST include:

• Acceptance criteria proving Postgres Steps A-E are unchanged
• A regression test comparing BM25+Vector results with Neo4j on vs off (must be identical)
• Proof that Neo4j writes occur AFTER Postgres storage (no added latency to Steps A-E)
• Proof that Neo4j ingestion failure still marks document as successfully ingested
  Requirements: 13.1, 13.2, 13.3, 13.4, 13.5

R14 — Graceful Degradation & Opt-In Behavior

Every sub-spec MUST include:

• Tests for every failure mode in the degradation cascade relevant to that packet
• NEO4J_URI unset → entire graph pipeline disabled (no entity extraction, no graph data anywhere)
• EXTRACTION_LLM_BASE_URL unset → skip relationship extraction, BERT entities + CO_OCCURS only
• Neo4j unreachable at query time → BM25+Vector only, no user-visible error
• Sidecar unreachable → skip entity extraction, Postgres-only ingestion
• GDS unavailable → skip community detection, warning log only
• Startup logging of active/inactive optional components
  Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6

Proposed Grouping (3-Dev Team)

This is a suggested plan. Dev A lays foundations and contracts first, then two devs build against those contracts in parallel. Each phase is a sub-spec that gets its own feature packet following the Team Workflow.

---

Phase 1: Foundations + Core Pipeline (Weeks 1) — R1, R2, R3, R4, R5, R6

Goal: Upload a document → entities + typed relationships extracted → written directly to Neo4j → vector indexed → entity resolution deduplicating. Graph is populated and queryable in Neo4j Browser.

Week 1 — Dev A lays the foundation (unblocks Dev B and Dev C)

Role	Work	Deliverables
Dev A	R1: Verify Neo4j Docker setup (1-2 days)	Neo4j running with --profile dev, health check passing
Dev A	R2: Build EmbeddingProvider interface + factory + both implementations	src/lib/embeddings/provider.ts committed
Dev A	Write shared Zod schemas (graphrag-schemas.ts) and contract compliance tests	__tests__/api/graphrag/contracts/ committed
Dev A	Write Neo4jDirectWriter TypeScript interface (not implementation)	Interface committed — Dev C builds against this
Dev A	Push to main	Dev B and Dev C unblocked

Weeks 1 — All three devs in parallel

Role	Packets	What they build against	Notes
Dev A	Integration + glue	Wire Steps F/F2/G into doc-ingestion/index.ts, review PRs, run contract tests	You're the integration point
Dev B	R3 (BERT CLS Embeddings) + R4 (Relationship Extraction)	Zod schemas from week 1, sidecar codebase	Pure Python sidecar work, no TypeScript or Neo4j dependency. Validates against contract tests.
Dev C	R5 (Direct Neo4j Write Pipeline) + R6 (Vector Index + Entity Resolution)	Neo4jDirectWriter interface from week 1, Zod schemas	Implements the Cypher queries and ingestion steps. Mocks sidecar responses using Zod schemas until Dev B delivers.

Parallelization guarantee: Dev B and Dev C never touch the same files. Dev B works in sidecar/app/. Dev C works in src/lib/graph/ and src/lib/tools/doc-ingestion/. They meet at the contract boundary (Zod schemas), not in code.

Phase 1 verification: Upload a document with Neo4j enabled → Neo4j Browser shows entities with typed relationships (CEO_OF, ACQUIRED, etc.) → entity resolution has merged duplicates → contract tests all pass.

---

Phase 2: Retrieval Layer (Weeks 2) — R7, R8, R9

Goal: Graph retriever traverses all relationship types, plugged into ensemble search with dynamic weights. Document content graph enables cross-document discovery. End users get measurably better search results.

Role	Packets	Notes
Dev A	R8 (Modular Ensemble Search Integration)	Hardening existing integration, dynamic weight logic. Smallest packet — leaves time for integration testing and PR review.
Dev B	R7 (Enhanced Graph Retriever)	Cypher query redesign to traverse all relationship types + vector similarity scoring. Builds on R5/R6 from Phase 1.
Dev C	R9 (Document Content Graph)	New node types (Document, Topic), cross-doc linking, topic extraction via LLM. Builds on R5/R6 from Phase 1.

Parallelization: R7 and R9 are fully independent — different Neo4j node types, different Cypher queries, different files. R8 depends on R7's retriever interface but you can start the dynamic weight logic and degradation paths while Dev B finalizes R7.

Phase 2 verification: Search query returns graph-enriched results → ensemble uses 3-retriever mode → Neo4j Browser shows Document → Section → Topic hierarchy → cross-document REFERENCES edges exist.

This is the core functionality milestone. After Phase 2, the system is production-usable.

---

Phase 3: Advanced Features (Weeks 3) — R10, R11, R12

Goal: Graph-guided hybrid retrieval (single Cypher query combining graph + vector), community detection for global queries, Text2Cypher for natural language graph queries.

Role	Packets	Notes
Dev A	R10 (Graph-Guided Hybrid Retrieval)	Complex Cypher query design, dual scoring algorithm, performance tuning for 100K nodes. Largest advanced packet.
Dev B	R12 (Text2Cypher)	LLM prompt engineering with graph schema, Cypher validation, read-only enforcement. Depends on R9 (document content graph from Phase 2).
Dev C	R11 (Community Detection)	GDS Leiden clustering, LLM community summaries, query classification. Depends on R10 — can start GDS integration while you finalize R10, then wires in the query routing.

Parallelization: R10 and R12 are fully independent. R11 depends on R10 but Dev C can build the Leiden clustering and summary generation while R10 is in progress — the dependency is only on the query routing interface.

Phase 3 verification: Single Cypher query returns graph+vector scored results within 500ms → "what are the main themes?" returns community summaries → natural language query generates valid read-only Cypher.

---

Timeline Summary

Phase	Weeks	Packets	Milestone
Phase 1	1	R1, R2, R3, R4, R5, R6	Graph populated, entities + relationships in Neo4j, vector indexed, entity resolution active
Phase 2	2	R7, R8, R9	Core functionality — graph-enriched search results, document content graph, ensemble integration
Phase 3	3	R10, R11, R12	Full SOTA — hybrid retrieval, community detection, Text2Cypher

Core functionality delivered: Week 5. Full SOTA: Week 7.

Notes

• Tasks marked as feature packets — each becomes its own full sub-spec before implementation
• R13 and R14 are cross-cutting constraints verified in every sub-spec, not standalone work items
• R1 is partially done from the old spec — verify before marking complete
• R3 and R4 have validated contracts from old spec experiments — implementation is new but API shapes are proven
• Critical path: R1 → R5 → R6 → R7 → R10 → R11
• The __tests__/api/graphrag/contracts/ test suite validates parent-level Zod schemas across all packets
• The phase-completion-check agent hook runs contract tests and syncs the parent spec when a sub-phase completes