Welcome to the Chunking subnet, Subnet 40 on the Bittensor Protocol! This subnet is designed to advance the field of Retrieval-Augmented Generation (RAG) by incentivizing the development and service of sophisticated chunking solutions. Specifically, the subnet aims to create, host, and serve an intelligent chunking solution that maximizes intrachunk similarity and interchunk dissimilarity.
Explore our subnet pitch deck.
Our article on why this is a valuable problem to solve: The Case for Intelligent Chunking
Learn more about our project at vectorchat.ai
See visualizations of subnet data at subnet.chunking.com
See how organic queries are handled here.
- ⚖️ Validator
- ⛏️ Default Miner
- 💰 Incentive Mechanism
- 📝 Evaluation
- 📊 Ranking
- 🧪 Synthetic Queries
- 🌱 Organic Queries & the Task API
- 📚 W&B Guide
- 🛣️ Roadmap
As mentioned in our pitch deck, chunking is an infinitely complex problem that can be approached from countless different avenues. Given sufficiently long, semantically meaningful text, there is no single correct answer, only "more" correct ones. Bittensor is an excellent way to tackle such a problem, as it incentivizes both innovation and fine-tuned optimization to find the most effective solution.
We do not open-source the models created, nor do we ever receive them. We believe this greatly increases the incentive for developing and/or providing the best solution, as miners retain full ownership of their work.
At the same time, we believe this increases the value brought to the Bittensor protocol, as access to the best chunking model will require a constant sufficient stake. Since validators never receive the model, but only the right to serve queries, losing stake in the network also results in losing access to any model produced by the subnet.
For those new to chunking or Retrieval Augmented Generation (RAG), we strongly recommend you check out our articles here:
We also recommend these resources by Pinecone.io: