When working with generative AI or RAG (Retrieval-Augmented Generation) systems, one of the biggest challenges is context. Models can’t process infinite text, so we need to split it. But doing it right isn’t trivial: cut too much and you lose coherence; cut too little and you overload the model and make retrieval worse.
That’s where Chonkie comes in, an open-source library that automates chunking: splitting text into fragments that make sense on their own. In this article I explain how it works, which strategies it offers, how to use its Pipeline with code tested against the current version, and what has changed since I wrote the first version of this post.
What Chonkie is
Chonkie is a Python library for chunking text before indexing it in a RAG system. It offers around ten chunking strategies, tools to add overlap or embeddings to each chunk, and connectors to export the result or store it in a vector database. MIT license. And it’s light.
The project started in November 2024 as a personal project by Bhavnick Minhas, who was tired of rewriting the same chunking code for every RAG application. Shreyash Nigam joined, they founded a company, went through Y Combinator in 2025, and today they maintain the library alongside an optional cloud service. The code is on GitHub and the documentation at docs.chonkie.ai.
Its clearest design choice is installing only what you need. According to its README, the base wheel is about 505 KB and about 49 MB installed, compared with 80 to 171 MB for alternatives that pull in many dependencies:
pip install chonkie # base: token, fast, sentence, recursive...
pip install "chonkie[semantic]" # semantic chunking with embeddings
pip install "chonkie[chroma]" # ChromaDB connector
pip install "chonkie[all]" # everything, not recommended in production
How it works
Chonkie organises the process into stages you can combine: fetch the documents, prepare them, chunk them, refine the chunks and store them. Each stage has its own named components, and in current versions you can chain all of them with a Pipeline object.
The stages, with the names the library uses:
| Stage | Component | What it does |
|---|---|---|
| Fetch | Fetchers | Read files or folders |
| Prepare | Chefs | Clean text, Markdown or tables before chunking |
| Chunk | Chunkers | Split the text with a given strategy |
| Refine | Refineries | Add overlap between chunks or compute embeddings |
| Export | Porters | Save as JSON or as a Hugging Face dataset |
| Store | Handshakes | Send to Chroma, Qdrant, pgvector, Pinecone, Weaviate, Milvus and others |
Chunking strategies
Chonkie offers strategies ranging from the simplest, cutting by size, to the most expensive, asking an LLM where to cut. The first three or four are usually enough; the rest only pay off if you have measured that they improve retrieval on your own data.
The chunkers available in version 1.7:
- TokenChunker: splits by token count. Fast and predictable.
- FastChunker: byte-based splitting for large volumes. Added in 1.5.2.
- SentenceChunker: groups whole sentences without breaking them.
- RecursiveChunker: follows the text structure (sections, paragraphs, sentences) with configurable rules.
- SemanticChunker: compares sentence embeddings and cuts where the topic changes.
- LateChunker: embeds the whole document first and then splits it, so each chunk keeps global context.
- CodeChunker: chunks code following its syntax.
- TableChunker: splits Markdown or HTML tables by rows without breaking columns.
- NeuralChunker: a trained model picks the split points.
- SlumberChunker: an LLM picks the split points. The most expensive option.
One detail worth knowing: if you don’t set a tokenizer, Chonkie counts characters. If you want chunk_size=512 to mean 512 tokens of your model, pass tokenizer="gpt2" or whichever you use (it accepts tiktoken, tokenizers and transformers).
A simple example
The most direct usage is still to create a chunker and call it with the text. Each returned chunk carries its text, its token count and its exact position in the original, which lets you link the model’s answer back to the source paragraph later.
from chonkie import RecursiveChunker
text = "Artificial intelligence is transforming entire industries. But understanding it requires precision and context."
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker(text)
for c in chunks:
print(c.text, c.token_count, c.start_index, c.end_index)
With just a few lines the text is split into coherent chunks, ready to embed or send to a model.
The Pipeline: the whole flow in one expression
Pipeline arrived in version 1.4.0, in October 2025, and it is the recommended way to build a complete flow. It chains the stages with methods such as chunk_with, refine_with, export_with or store_in, and runs with run, which returns a Document with its chunks.
from chonkie import Pipeline
pipe = (
Pipeline()
.fetch_from("file", dir="./docs", ext=[".md"])
.process_with("markdown")
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=64)
.export_with("json", file="chunks.json")
)
docs = pipe.run() # a folder returns a list of Document
for doc in docs:
print(len(doc.chunks))
If you are not reading files, pass the text directly with pipe.run(texts="..."), which returns a single Document. To store in a vector database add, for example, .store_in("qdrant", collection_name="docs", url="http://localhost:6333"), and to compute embeddings .refine_with("embeddings", model="all-MiniLM-L6-v2") (note that the name is embeddings, plural). There is also an async version, await pipe.arun(...), for high volumes.
I have checked this code against version 1.7.0. If you are coming from versions before 1.4, note that the old tokenizer_or_token_counter argument is gone and is now called tokenizer everywhere.
Chonkie compared with other options
Chonkie competes with the text splitters in LangChain and LlamaIndex, which you already get if you use those frameworks. The difference is that Chonkie is a specialised, standalone library: more strategies, lighter, and it doesn’t force you to adopt a whole framework.
| Chonkie | LangChain text splitters | LlamaIndex node parsers | |
|---|---|---|---|
| Focus | Chunking and ingestion only | Part of a large framework | Part of a large framework |
| Strategies | Token, sentence, recursive, semantic, late, code, tables, neural, LLM | Recursive, character, token, Markdown, code | Sentence, token, semantic, hierarchical |
| Vector connectors | Included as extras | Through the framework | Through the framework |
| License | MIT | MIT | MIT |
If you already have everything built on LangChain or LlamaIndex, their splitters may be enough. If you want a piece you can swap without touching the rest, Chonkie fits better.
Limitations
Chonkie handles chunking well, but it doesn’t read scanned PDFs, it doesn’t understand the layout of a complex document and it won’t tell you which strategy is best for your data. That part is on you: measure retrieval quality with real questions and tune the size and strategy.
- Input. It works on text. For PDF, Word or images you first need a converter such as Docling, or the newer Mistral OCR and LiteParse chefs.
- Cost of advanced strategies. Semantic, neural and LLM chunking need extra models and dependencies, and they are much slower.
- Self-published benchmarks. The speed figures (for example, up to 33 times faster token chunking) come from the project itself.
- Moving API. The library changes fast: there have been several breaking changes between minor versions, so pin the version in production.
What has happened since
Since October 2025 Chonkie has shipped three minor versions at a pace of almost one update a month. It has gained Pipeline, HTML table chunking, a FastChunker, native async support, a chunking API you can host yourself and new document converters. Its repository has more than 4,800 stars.
According to the release notes on GitHub and the PyPI history:
- 1.4.0 (11 October 2025).
chonkie.Pipelinearrives, together withTableChefandTableChunker. - 1.5.0 (16 December 2025). Drops Python 3.9 and adds 3.13; new byte and row tokenizers.
- 1.5.2 (January 2026). New
FastChunker. - 1.6.0 (11 March 2026). HTML table support, a chunking API you can deploy on your own server and native async execution.
- 1.7.0 (7 July 2026). LiteParse integration to read more formats and a public
ComponentRegistryfor building your own components.
The repository now redirects to the feyninc organisation, although the old links still work. The library is also available in TypeScript for those working in Node.
If your flow needs to keep the index up to date when documents change, it is worth reading how RAG fits into an event-driven architecture with LLMs.
Conclusion
Chonkie helps models read better, because it gives them chunks that make sense instead of arbitrary cuts. With Pipeline, the whole ingestion flow fits in a few lines and each piece can be swapped without touching the rest.
If you want to dive deeper into this type of tool, I recommend the article where I compare Chonkie with Docling, two complementary approaches to document processing in RAG architectures, and the full article on Docling.
If you found this useful, keep reading Chunking in RAG: strategies, overlap, chunk size and how to evaluate it and Chonkie and Docling: two complementary approaches to document processing in RAG pipelines.