marc mayol
← Back to blog

Chonkie: how to chunk text for RAG with chunkers and Pipeline

Chonkie: how to chunk text for RAG with chunkers and Pipeline

When working with generative AI or RAG (Retrieval-Augmented Generation) systems, one of the biggest challenges is context. Models can’t process infinite text, so we need to split it. But doing it right isn’t trivial: cut too much and you lose coherence; cut too little and you overload the model and make retrieval worse.

That’s where Chonkie comes in, an open-source library that automates chunking: splitting text into fragments that make sense on their own. In this article I explain how it works, which strategies it offers, how to use its Pipeline with code tested against the current version, and what has changed since I wrote the first version of this post.

What Chonkie is

Chonkie is a Python library for chunking text before indexing it in a RAG system. It offers around ten chunking strategies, tools to add overlap or embeddings to each chunk, and connectors to export the result or store it in a vector database. MIT license. And it’s light.

The project started in November 2024 as a personal project by Bhavnick Minhas, who was tired of rewriting the same chunking code for every RAG application. Shreyash Nigam joined, they founded a company, went through Y Combinator in 2025, and today they maintain the library alongside an optional cloud service. The code is on GitHub and the documentation at docs.chonkie.ai.

Its clearest design choice is installing only what you need. According to its README, the base wheel is about 505 KB and about 49 MB installed, compared with 80 to 171 MB for alternatives that pull in many dependencies:

pip install chonkie                 # base: token, fast, sentence, recursive...
pip install "chonkie[semantic]"     # semantic chunking with embeddings
pip install "chonkie[chroma]"       # ChromaDB connector
pip install "chonkie[all]"          # everything, not recommended in production

How it works

Chonkie organises the process into stages you can combine: fetch the documents, prepare them, chunk them, refine the chunks and store them. Each stage has its own named components, and in current versions you can chain all of them with a Pipeline object.

The stages, with the names the library uses:

StageComponentWhat it does
FetchFetchersRead files or folders
PrepareChefsClean text, Markdown or tables before chunking
ChunkChunkersSplit the text with a given strategy
RefineRefineriesAdd overlap between chunks or compute embeddings
ExportPortersSave as JSON or as a Hugging Face dataset
StoreHandshakesSend to Chroma, Qdrant, pgvector, Pinecone, Weaviate, Milvus and others

Chunking strategies

Chonkie offers strategies ranging from the simplest, cutting by size, to the most expensive, asking an LLM where to cut. The first three or four are usually enough; the rest only pay off if you have measured that they improve retrieval on your own data.

The chunkers available in version 1.7:

One detail worth knowing: if you don’t set a tokenizer, Chonkie counts characters. If you want chunk_size=512 to mean 512 tokens of your model, pass tokenizer="gpt2" or whichever you use (it accepts tiktoken, tokenizers and transformers).

A simple example

The most direct usage is still to create a chunker and call it with the text. Each returned chunk carries its text, its token count and its exact position in the original, which lets you link the model’s answer back to the source paragraph later.

from chonkie import RecursiveChunker

text = "Artificial intelligence is transforming entire industries. But understanding it requires precision and context."

chunker = RecursiveChunker(chunk_size=512)
chunks = chunker(text)

for c in chunks:
    print(c.text, c.token_count, c.start_index, c.end_index)

With just a few lines the text is split into coherent chunks, ready to embed or send to a model.

The Pipeline: the whole flow in one expression

Pipeline arrived in version 1.4.0, in October 2025, and it is the recommended way to build a complete flow. It chains the stages with methods such as chunk_with, refine_with, export_with or store_in, and runs with run, which returns a Document with its chunks.

from chonkie import Pipeline

pipe = (
    Pipeline()
    .fetch_from("file", dir="./docs", ext=[".md"])
    .process_with("markdown")
    .chunk_with("recursive", chunk_size=512)
    .refine_with("overlap", context_size=64)
    .export_with("json", file="chunks.json")
)

docs = pipe.run()  # a folder returns a list of Document
for doc in docs:
    print(len(doc.chunks))

If you are not reading files, pass the text directly with pipe.run(texts="..."), which returns a single Document. To store in a vector database add, for example, .store_in("qdrant", collection_name="docs", url="http://localhost:6333"), and to compute embeddings .refine_with("embeddings", model="all-MiniLM-L6-v2") (note that the name is embeddings, plural). There is also an async version, await pipe.arun(...), for high volumes.

I have checked this code against version 1.7.0. If you are coming from versions before 1.4, note that the old tokenizer_or_token_counter argument is gone and is now called tokenizer everywhere.

Chonkie compared with other options

Chonkie competes with the text splitters in LangChain and LlamaIndex, which you already get if you use those frameworks. The difference is that Chonkie is a specialised, standalone library: more strategies, lighter, and it doesn’t force you to adopt a whole framework.

ChonkieLangChain text splittersLlamaIndex node parsers
FocusChunking and ingestion onlyPart of a large frameworkPart of a large framework
StrategiesToken, sentence, recursive, semantic, late, code, tables, neural, LLMRecursive, character, token, Markdown, codeSentence, token, semantic, hierarchical
Vector connectorsIncluded as extrasThrough the frameworkThrough the framework
LicenseMITMITMIT

If you already have everything built on LangChain or LlamaIndex, their splitters may be enough. If you want a piece you can swap without touching the rest, Chonkie fits better.

Limitations

Chonkie handles chunking well, but it doesn’t read scanned PDFs, it doesn’t understand the layout of a complex document and it won’t tell you which strategy is best for your data. That part is on you: measure retrieval quality with real questions and tune the size and strategy.

What has happened since

Since October 2025 Chonkie has shipped three minor versions at a pace of almost one update a month. It has gained Pipeline, HTML table chunking, a FastChunker, native async support, a chunking API you can host yourself and new document converters. Its repository has more than 4,800 stars.

According to the release notes on GitHub and the PyPI history:

The repository now redirects to the feyninc organisation, although the old links still work. The library is also available in TypeScript for those working in Node.

If your flow needs to keep the index up to date when documents change, it is worth reading how RAG fits into an event-driven architecture with LLMs.

Conclusion

Chonkie helps models read better, because it gives them chunks that make sense instead of arbitrary cuts. With Pipeline, the whole ingestion flow fits in a few lines and each piece can be swapped without touching the rest.

If you want to dive deeper into this type of tool, I recommend the article where I compare Chonkie with Docling, two complementary approaches to document processing in RAG architectures, and the full article on Docling.

If you found this useful, keep reading Chunking in RAG: strategies, overlap, chunk size and how to evaluate it and Chonkie and Docling: two complementary approaches to document processing in RAG pipelines.