Document Ingestion: Clean Text, Metadata and Incremental Updates from PDF, HTML and Markdown
RAG quality starts at ingestion: a "fi" ligature from a PDF made Contains("fiyat") return false, a page number landed in the middle of a section, and an HTML menu leaked into a chunk. Clean text, metadata on every chunk, a pipeline with partial success using Microsoft.Extensions.DataIngestion (preview), incremental updates with content hashes, and removing deleted documents from the index.
8 Nov 2026
·
14 min
·
4
