pathwaycom/llm-app
Ready-to-run cloud templates for building real-time RAG, AI pipelines, and enterprise search applications that synchronize with various live data sources.
Awesome Infra for AI › Vector Databases & Retrieval Infrastructure
paulpierre/markdown-crawler
Markdown Crawler is a specialized web crawling tool that recursively navigates websites, converting each page into a structured markdown file. Its primary purpose is to prepare data for Large Language Model (LLM) applications, particularly for Retrieval Augmented Generation (RAG) use cases. By transforming diverse web content into a consistent markdown format, it simplifies the subsequent steps of LLM processing, such as chunking, indexing, and context retrieval. The tool offers features like multithreading for faster crawling, the ability to resume scraping, configurable maximum crawl depth, and support for various HTML elements including tables and images. It also includes validation for URLs, HTML, and filepaths, along with options to restrict crawling to specific base paths or domains. Its CLI interface makes it easy to use for data ingestion pipelines, allowing users to specify parameters like the number of threads, depth, and output directory. The use of markdown preserves document structure while maintaining a small footprint, which is beneficial for creating large corpuses for fine-tuning LLMs or for building knowledge bases for autonomous agents.
https://github.com/paulpierre/markdown-crawler
Also in Vector Databases & Retrieval Infrastructure
run-llama/llama_index
LlamaIndex is an open-source data framework for building LLM applications by connecting custom data sources to large language models, focusing on data ingestion, indexing, and retrieval augmented g...
milvus-io/milvus
Milvus is a high-performance, cloud-native vector database designed for scalable vector Approximate Nearest Neighbor (ANN) search, efficiently organizing and searching vast amounts of unstructured ...
VectifyAI/PageIndex
PageIndex is a vectorless, reasoning-based RAG system that builds hierarchical tree indexes from documents and uses LLMs to reason over them for context-aware retrieval.
qdrant/qdrant
Qdrant is an open-source, high-performance vector similarity search engine and vector database designed specifically for AI applications, enabling fast storage, search, and management of vectors wi...
Tencent/WeKnora
WeKnora is an open-source, LLM-powered knowledge framework for enterprise document understanding, semantic retrieval, and autonomous reasoning, featuring RAG, ReAct agents, and an auto-maintaining ...
topoteretes/cognee
Cognee is an open-source AI memory platform that provides AI agents with persistent long-term memory through a self-hosted knowledge graph, combining vector embeddings and graph reasoning.
RyanCodrai/turbovec
TurboVec is a Rust-based approximate nearest neighbor (ANN) vector index with Python bindings, built on Google Research's TurboQuant algorithm for efficient, memory-optimized vector similarity search.