Build an Open-Source Document Pipeline for AI Agents
Learn to extract PDFs, websites, and documents using the open-source Docling library, chunk them intelligently, embed them with OpenAI, store in LanceDB, and retrieve relevant context in a chat application—all without proprietary APIs.
Why Open-Source Document Extraction Matters
The Problem with Closed-Source Solutions
Most document extraction tools require API keys and proprietary platforms, forcing you to send data externally for parsing. Open-source alternatives like Docling offer the same capability without vendor lock-in or external dependencies.
Docling: The Open-Source Choice
Docling, an IBM project, is the leading open-source document extraction library. It handles PDFs, PowerPoint, DOCX, and websites, converting them into a unified data model called the Docling document that works across all input types.
Five-Step Knowledge Pipeline
The complete process involves extraction (convert documents to structured format), chunking (split into logical pieces), embedding (convert text to vectors), storage (save in vector database), and retrieval (search and answer questions).
Document Extraction with Docling
PDF Extraction: From Raw File to Structured Data
Docling's DocumentConverter takes a PDF, analyzes all blocks and components using OCR, and returns a Docling document object. This object can be exported to Markdown, JSON, or DJSON for downstream processing.
Table Extraction Excellence
Unlike many open-source PDF libraries, Docling excels at extracting tables with perfect formatting, clean markdown structure, and correct heading formats—no weird characters or parsing errors.
Single Web Page Extraction
Pass a URL to Docling's convert method instead of a PDF path. It parses the HTML and returns a Markdown replica of the page, making it easy to extract web content without manual HTML parsing.
Bulk Website Extraction via Sitemap
Use the sitemap.xml file (available at any website's root) to discover all URLs. Docling's convert_all method loops through the sitemap and extracts every page, returning a unified list of Docling documents.
Intelligent Chunking Strategies
Why Chunking Matters
Instead of storing entire documents in a vector database, chunking splits them into logical pieces. This ensures that when you query the system, you retrieve only relevant sections rather than whole books or documents.
Hierarchical Chunking
Docling's hierarchical chunker automatically splits documents based on logical structure—lists, paragraphs, headers—creating parent-child relationships that preserve document hierarchy and meaning.
Hybrid Chunking with Token Limits
The hybrid chunker splits chunks that exceed your embedding model's max input tokens and stitches together chunks that are too small. It uses your model's tokenizer to ensure chunks fit perfectly within token limits.
Chunking in Action: PDF Example
A single PDF document was chunked into 36 logical pieces using the hybrid chunker with OpenAI's text-embedding-3-large model, ensuring each chunk fits within the model's 8,191 token limit.
Embedding and Vector Storage
LanceDB: Lightweight Vector Database
LanceDB stores the vector database as a persistent file (like SQLite), making it easy to work with locally without external infrastructure. It has a clean Python API and handles embeddings automatically at the table level.
Schema Design for Chunks
Define a Pydantic model with three fields: text (the chunk content), vector (the embedding), and metadata (filename, page numbers, section title). This structure enables both semantic search and source attribution.
Automatic Embedding on Insert
When you add chunks to a LanceDB table with an embedding function specified, the database automatically generates embeddings using your chosen model (e.g., OpenAI's text-embedding-3-large) without manual API calls.
Storing 36 Chunks with Metadata
All 36 chunks from the PDF were successfully inserted into LanceDB with their embeddings and metadata (filename, page numbers, section titles), ready for semantic search.
Search and Retrieval
Vector Similarity Search
Query the vector database using LanceDB's search method with query_type='vector'. The system embeds your question, finds the most similar chunks by vector distance, and returns the top N results (e.g., top 5).
Search Results Include Metadata
Each retrieved chunk returns the text, vector, metadata (filename, page numbers), and distance score. This allows you to cite sources and show users exactly where information came from.
Query Examples
Searching for 'PDF' retrieves chunks about PDF handling; searching for 'what's Docling' retrieves chunks explaining Docling. The vector search understands semantic meaning, not just keyword matching.
Adjustable Result Limits
Set the limit parameter to control how many chunks are retrieved (e.g., limit=5 returns top 5 results). More results provide more context but may include less relevant information.
Building the Chat Application
Streamlit for Rapid Prototyping
Streamlit is a Python framework that turns scripts into interactive web apps without frontend code. Its chat components make it trivial to build a conversational interface for querying your knowledge base.
Chat Application Architecture
The app connects to the LanceDB database, searches for relevant chunks based on user questions, passes the retrieved context to an LLM for answering, and displays both the answer and source citations.
Live Configuration Changes
Modify parameters like the number of retrieved results (e.g., change from 3 to 5) and save the file. Streamlit hot-reloads the app, allowing you to test different configurations instantly.
Scalability Through Data Addition
The chat app is dynamic and scalable. Add more PDFs, websites, or documents by running the extraction and chunking pipeline again, then insert the new chunks into the same LanceDB table.
Key Takeaways and Next Steps
Complete Open-Source Pipeline
You now have a fully functional, open-source document extraction and retrieval system using Docling, LanceDB, and Streamlit. No proprietary APIs required for extraction; optional OpenAI for embeddings and chat.
Universally Applicable Concepts
The extraction, chunking, embedding, and retrieval concepts work with any vector database (PostgreSQL + pgvector, Pinecone, Weaviate), any embedding model (open-source or commercial), and any LLM.
Foundation for AI Agents
This knowledge extraction pipeline is the foundation for building AI agents with access to your company's proprietary data—documents, PDFs, websites—enabling them to answer questions with accurate, sourced information.
Notable quotes
Most tools come at the cost that they are closed source, but there are also open source alternatives available that work just as well. — Dave Ebbelaar
Docling is by far right now the open-source document extraction library of choice. — Dave Ebbelaar
The cool thing about LanceDB is that in the background it performed the embeddings as well, and you don't have to worry about the embeddings. — Dave Ebbelaar
Action items
- Clone the GitHub repository linked in the video description and set up a Python environment with the requirements.txt file.
- Obtain an OpenAI API key (optional if using open-source embedding models) and set it as an environment variable.
- Run the extraction script on a sample PDF to verify Docling installation and see the parsed output in Markdown format.
- Test the chunking pipeline on your own documents to understand how the hybrid chunker splits content based on your embedding model's token limits.
- Create a LanceDB database and insert your chunks with embeddings using the provided schema.
- Test the search functionality by querying the vector database with different questions and examining retrieved chunks and metadata.
- Launch the Streamlit chat application locally and interact with your knowledge base through the conversational interface.
- Expand the pipeline by adding more documents (PDFs, websites) and re-running the extraction and insertion steps to scale your knowledge base.