Chonkie Documentation

Not sure which chunker to use? This guide helps you pick the right one based on your use case, performance needs, and content type.

Quick Recommendations

Building a RAG chatbot?

Use RecursiveChunker for general-purpose document chunking, or SemanticChunker if you need high topical coherence.

Processing source code?

Use CodeChunker — it understands AST structure and keeps functions/classes intact.

Need maximum throughput?

Use FastChunker — SIMD-accelerated, processes 100+ GB/s at the byte level.

Working with tabular data?

Use TableChunker — preserves headers and splits by rows.

Decision Guide

Ask yourself these questions in order:

Are you chunking source code? → Use CodeChunker
Are you chunking tables? → Use TableChunker
Is raw throughput your top priority? → Use FastChunker
Do you need chunks grouped by topic?
- Have an embedding model? → Use SemanticChunker
- Don’t have one? → Use NeuralChunker
Are simple sentence-boundary splits enough? → Use SentenceChunker
None of the above? → Use RecursiveChunker (best default)

Comparison Table

Chunker	Speed	Quality	Dependencies	Best For
FastChunker	Fastest (100+ GB/s)	Basic	None (SIMD)	High-throughput pipelines, byte-size limits
TokenChunker	Very fast	Good	Tokenizer	Fixed-size chunks, token-based models
SentenceChunker	Fast	Good	Tokenizer	Clean sentence boundaries, simple docs
RecursiveChunker	Fast	Very good	Tokenizer	General-purpose, structured docs
TableChunker	Fast	Excellent (tables)	None	Markdown/HTML tables
CodeChunker	Moderate	Excellent (code)	Tree-sitter	Source code files
SemanticChunker	Slower	Excellent	Embedding model	Topic-coherent chunks, RAG quality
NeuralChunker	Slower	Excellent	BERT model	Topic segmentation without embeddings
LateChunker	Slower	Excellent	Embedding model	Higher recall in RAG
SlumberChunker	Slowest	S-tier	LLM (Genie)	Maximum quality, cost not a concern

When to Use Each

You just need it to work

Start with RecursiveChunker. It’s the best general-purpose option — fast, no heavy dependencies beyond a tokenizer, handles most document types well.

from chonkie import RecursiveChunker

chunker = RecursiveChunker(chunk_size=512)
chunks = chunker.chunk(text)

You need high-quality semantic chunks

Use SemanticChunker if you have an embedding model available. It groups sentences by meaning, so each chunk stays on-topic.

from chonkie import SemanticChunker

chunker = SemanticChunker(
    embedding_model="all-MiniLM-L6-v2",
    chunk_size=512,
    similarity_threshold=0.5
)
chunks = chunker.chunk(text)

You're processing code

Use CodeChunker — it parses the AST so your chunks respect function/class boundaries instead of cutting mid-statement.

from chonkie import CodeChunker

chunker = CodeChunker(language="python", chunk_size=512)
chunks = chunker.chunk(source_code)

You need raw speed above all else

Use FastChunker for pipelines where throughput matters more than chunk boundary quality. It uses SIMD instructions for 100+ GB/s processing.

from chonkie import FastChunker

chunker = FastChunker(chunk_size=4096)  # byte size
chunks = chunker.chunk(text)

You want the absolute best quality

Use SlumberChunker with a generative model. It’s the slowest and most expensive, but produces the highest-quality chunks by using an LLM to decide boundaries.

from chonkie import SlumberChunker

chunker = SlumberChunker(
    genie="openai",
    chunk_size=512
)
chunks = chunker.chunk(text)

F.A.Q.

Can I switch chunkers without changing my pipeline?

Yes. All chunkers share the same interface (chunk(), chunk_batch(), async variants). Swap one for another and everything downstream stays the same.

Should I use SemanticChunker or NeuralChunker?

SemanticChunker if you already have an embedding model in your pipeline (reuse it). NeuralChunker if you don’t — it uses a small BERT model specifically trained for topic segmentation, so it doesn’t require a separate embedding setup.

Is FastChunker good enough for RAG?

It depends on your tolerance for imperfect boundaries. FastChunker splits on byte counts, which can cut mid-word or mid-sentence. For RAG where retrieval quality matters, prefer RecursiveChunker or SemanticChunker. Use FastChunker when you need to process terabytes quickly and can tolerate rough boundaries.

What about LateChunker?

LateChunker implements the “Late Chunking” algorithm which produces embeddings with better recall for retrieval tasks. Use it when retrieval accuracy is your top priority and you can afford the extra compute.

Getting Started

API Server

Chefs

Fetchers

Chunkers

Embeddings

Refinery

Handshakes

Porters

Utils

Experimental

Changelog

Choosing a Chunker

Quick Recommendations

Building a RAG chatbot?

Processing source code?

Need maximum throughput?

Working with tabular data?

Decision Guide

Comparison Table

When to Use Each

F.A.Q.

​Quick Recommendations

Building a RAG chatbot?

Processing source code?

Need maximum throughput?

Working with tabular data?

​Decision Guide

​Comparison Table

​When to Use Each

​F.A.Q.

Quick Recommendations

Decision Guide

Comparison Table

When to Use Each

F.A.Q.