🧬 Clinical NLP Pipeline

Extract diseases, drugs, symptoms, lab values, dosages, and clinical relations from unstructured medical text — with negation detection and UMLS concept mapping.

A production-grade BioNLP pipeline built from scratch, combining rule-based NLP, dictionary NER, NegEx negation detection, and dependency-based relation extraction. Designed for clinical notes, discharge summaries, and EHR text.

---

🔬 What It Does

Given raw clinical text like:

"Patient has hypertension. Aspirin 325mg and lisinopril 10mg prescribed. Denies fever or chest pain. BP 158/94 mmHg. HbA1c 8.2%."

The pipeline extracts:

Entity	Type	Code
Hypertension	DISEASE	ICD-10:I10
Aspirin	DRUG	RxNorm:1191
Lisinopril	DRUG	RxNorm:29046
325mg	DOSAGE	—
BP 158/94 mmHg	LAB_VALUE	Blood Pressure
HbA1c 8.2%	LAB_VALUE	HbA1c
Fever	SYMPTOM (negated)	—
Chest pain	SYMPTOM (negated)	—

And relations:

[DRUG] Aspirin    ─HAS_DOSAGE→  [DOSAGE] 325mg
[DRUG] Lisinopril ─TREATS→      [DISEASE] Hypertension

---

🏗️ Pipeline Architecture

Raw Clinical Text
      ↓
[Stage 1] Text Normalization
          Abbreviation expansion (htn→hypertension, sob→shortness of breath, ...)
      ↓
[Stage 2] Dictionary NER
          Diseases  → UMLS-inspired vocabulary → ICD-10 codes
          Drugs     → RxNorm vocabulary        → RxNorm IDs
          Symptoms  → Clinical symptom lexicon
          Procedures → Medical procedure list
      ↓
[Stage 3] Regex NER
          Lab values (glucose, HbA1c, BP, SpO2, ...)
          Dosages   (mg, mcg, ml, tablets, ...)
      ↓
[Stage 4] Negation Detection (NegEx Algorithm)
          Detects: "no fever", "denies chest pain", "absent edema"
          Window-based trigger search + pattern matching
      ↓
[Stage 5] Span Deduplication
          Removes overlapping entities, keeps longest match
      ↓
[Stage 6] Relation Extraction
          DRUG ─TREATS→      DISEASE   (via dependency patterns)
          DRUG ─HAS_DOSAGE→  DOSAGE    (proximity-based)
          SYMPTOM ─INDICATES→ DISEASE  (co-occurrence in sentence)
      ↓
ClinicalReport (entities, relations, negations, summary)

---

📁 Project Structure

clinical-nlp-pipeline/
├── src/
│   ├── __init__.py
│   └── pipeline.py          # Full NLP pipeline (700+ lines)
├── data/
│   └── samples/
│       └── sample_notes.txt # 2 realistic clinical notes for testing
├── tests/
│   ├── __init__.py
│   └── test_pipeline.py     # 25 unit tests (pytest)
├── app.py                   # Streamlit web UI (dark clinical theme)
├── cli.py                   # CLI with colored terminal output
├── requirements.txt
└── README.md

---

⚙️ Installation

# 1. Clone the repo
git clone https://github.com/YOUR_USERNAME/clinical-nlp-pipeline.git
cd clinical-nlp-pipeline

# 2. Create virtual environment
python -m venv venv
source venv/bin/activate     # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Download spaCy model
python -m spacy download en_core_web_md

---

🚀 Usage

🌐 Web App

streamlit run app.py

Open http://localhost:8501 — paste any clinical text and get instant extraction results with a clean clinical dashboard.

---

💻 CLI

# From a text file
python cli.py --file data/samples/sample_notes.txt

# From a text string
python cli.py --text "Patient has hypertension. Started lisinopril 10 mg daily."

# JSON output
python cli.py --file data/samples/sample_notes.txt --json

Sample output:

════════════════════════════════════════════════════
       CLINICAL NLP PIPELINE — EXTRACTION REPORT
════════════════════════════════════════════════════

  📊 Total Entities  : 21
  🔗 Relations Found : 7
  ❌ Negated Findings: 4

  🔴 Active Diagnoses:
     • Myocardial Infarction
     • Hypertension
     • Type 2 Diabetes Mellitus

  💊 Active Medications:
     • Aspirin  • Clopidogrel  • Metformin  • Lisinopril

  🔗 CLINICAL RELATIONS
  [DRUG] Aspirin     ─HAS_DOSAGE→  [DOSAGE] 325 mg
  [DRUG] Metformin   ─TREATS→      [DISEASE] Diabetes Mellitus
  [DRUG] Lisinopril  ─TREATS→      [DISEASE] Hypertension

---

🐍 Python API

from src.pipeline import run_pipeline

report = run_pipeline("""
Patient presented with chest pain and shortness of breath.
History of hypertension. Denies fever. BP 158/94 mmHg.
Started lisinopril 10 mg and aspirin 325 mg.
""")

# Access results
print(report.summary["active_diagnoses"])   # ['Hypertension']
print(report.summary["active_medications"]) # ['Lisinopril', 'Aspirin']
print(report.negated_findings)              # ['Fever']

# All entities
for ent in report.entities:
    print(f"{ent.label}: {ent.normalized} | Code: {ent.code} | Negated: {ent.negated}")

# Relations
for rel in report.relations:
    print(f"{rel.subject} ─{rel.relation}→ {rel.object}")

---

🧪 Tests

pytest tests/ -v

25 unit tests covering normalization, negation detection, all entity types, and the full pipeline.

---

🛠️ Tech Stack

Component	Technology
NLP Engine	spaCy (en_core_web_md)
Dictionary NER	Custom UMLS-inspired vocabularies
Regex NER	Python re patterns
Negation Detection	NegEx algorithm (window + patterns)
Relation Extraction	Dependency parsing + proximity
Concept Mapping	ICD-10 & RxNorm approximation
Web UI	Streamlit
Testing	pytest

---

📚 Medical Vocabularies Included

• 500+ disease terms with ICD-10 codes (cardiovascular, metabolic, neurological, pulmonary, oncological, psychiatric)
• 50+ drugs with RxNorm identifiers (antibiotics, antihypertensives, antidiabetics, statins, analgesics, psychotropics)
• 60+ clinical symptoms (UMLS-aligned)
• 30+ medical procedures (diagnostic & therapeutic)
• 50+ medical abbreviations expanded automatically

---

🔮 Roadmap

• Fine-tuned BioBERT / ClinicalBERT for NER
• MIMIC-III dataset evaluation
• ICD-10 full code lookup API
• Temporal information extraction (onset, duration)
• Patient timeline visualization
• FHIR-compatible JSON output
• Named entity linking to full UMLS

---

⚠️ Disclaimer

This tool is for research and educational purposes only. It is not a certified medical device and should not be used for clinical decision-making.

---