| First, apologies because I used ChatGPT to draft this text. First time poster here. I’ve developed a lightweight Python-based solution that lets you turn your personal PDF library into a searchable knowledge base—no complex setup, no bloated dependencies. What It Does:
1. Extracts text from PDFs and saves it as JSON files, chunked by pages.
2. Optionally generates text embeddings (using `fastembed`) for semantic search.
3. Provides a user-friendly GUI for searching, navigating results, and previewing matching content. The search interface supports:
- BM25 Algorithm: Great for keyword-based search.
- Simple Text Search: For exact phrase matching.
- Embedding-Based Search (optional): If you want semantic search capabilities, embeddings are supported but not required. Why This Project?
I wanted something simple, portable, and fast for managing my own PDF archives—something that doesn’t require setting up a database or maintaining a server. The focus is on the search interface simplicity and usefulness. With this tool:
- Your knowledge base is stored as portable JSON and EMB files. You can easily move it across devices.
- The GUI allows for intuitive searches and even opens matching pages in your default PDF viewer (Okular by default, but configurable). Key Features:
- Minimal Dependencies: Works out of the box with BM25 and a small set of Python libraries.
- Optional Semantic Search: Add `fastembed` for enhanced search capabilities.
- Customizable Workflow: JSON-only mode for simplicity or JSON+EMB for full functionality.
- Cross-Platform Support: As long as Python and your chosen PDF viewer are installed. Use Cases:
- Researchers managing collections of academic papers.
- Anyone needing quick access to information buried in PDFs. How to Get Started:
1. Install Python and the required libraries.
2. Process your PDFs into JSON files with a single script.
3. (Optional) Generate embeddings for richer search results.
4. Use the search interface to start finding what you need! Check out the code and detailed instructions here: https://github.com/Topping1/BM25-PDF-Search I hope this is useful for the community. |