Faculty Finder

1. Core Problem & Scope

Academic research outreach is bottlenecked by fragmented institutional profiles and manual email scraping routines. Faculty directories are inconsistently formatted across universities — PDF listings, scanned pages, unstructured HTML — and manually finding faculty relevant to a specific research interest doesn't scale past a handful of institutions. The scope was a sub-second semantic search and outreach pipeline over ingested institutional directories.

2. Comparative System Research

• Pure keyword search (grep-style) — trivial to build, but fails on any query where the relevant terminology doesn't literally appear in the source text (e.g., "vision-language models" vs. a professor's page describing "multimodal deep learning").
• Pure dense vector search — captures semantic similarity well, but underperforms on queries with specific proper nouns, acronyms, or exact technical terms where keyword precision matters more than semantic breadth.
• BM25 + dense hybrid ranking (chosen) — combining a sparse keyword index with dense embedding similarity captures both exact-term precision and semantic recall, which matched the actual query patterns seen (a mix of specific technical terms and broader topic descriptions).

3. Architectural Pipeline Layouts

Institutional Directories → Tesseract OCR + LLM Parsing → SentenceTransformers Embeddings → Vector Store
                                                                                                    │
                                                                          BM25 Hybrid Ranking ←─────┘
                                                                                    │
                                                                          n8n Outreach Automation

Unformatted directory pages are ingested through async pipelines combining Tesseract OCR (for scanned/image-based listings) with LLM-assisted parsing to normalize inconsistent layouts into structured faculty records. Each record is embedded via SentenceTransformers (all-mpnet-base-v2) into a vector store, and queries are resolved through a custom BM25 hybrid ranker that blends keyword and semantic scores before handing matched records to n8n workflows for outreach automation and Google Sheets logging.

4. Trade-off Engineering Logs

Why LLM-assisted parsing instead of a rules-based scraper per institution: A rules-based scraper needs a custom extraction template per institution's page layout, which doesn't scale past a small number of universities and breaks silently on any layout change. LLM-assisted parsing trades a small per-page inference cost for extraction logic that generalizes across layouts without per-site maintenance — the right trade for a pipeline meant to onboard new institutions without engineering time per source.

Hybrid ranking weight tuning: The blend weight between BM25 and dense similarity scores was tuned empirically rather than fixed, since the ideal weighting shifted depending on whether queries were acronym-heavy (favoring BM25) or topic-descriptive (favoring dense similarity).

5. Scaling & Production Failures

OCR throughput was the first bottleneck — synchronous Tesseract calls inside the ingestion path serialized what should have been parallel document processing. Moving OCR into its own async worker pool, decoupled from the parsing and embedding steps, restored expected throughput. A second issue surfaced in ranking quality: early hybrid scoring occasionally surfaced faculty with a single keyword match but low topical relevance; adding a minimum dense-similarity floor before a result could surface, regardless of BM25 score, corrected the false-positive rate.