Skip to main content
Chonkie provides multiple chunking strategies to handle different text processing needs. Each chunker in Chonkie is designed to follow the same core principles outlined in the concepts page.

TokenChunker

Splits text into fixed-size token chunks. Best for maintaining consistent chunk sizes and working with token-based models.

FastChunker

SIMD-accelerated byte-based chunking at 100+ GB/s. Best for high-throughput pipelines where byte size limits are acceptable.

SentenceChunker

Splits text at sentence boundaries. Perfect for maintaining semantic completeness at the sentence level.

RecursiveChunker

Recursively chunks documents into smaller chunks. Best for long documents with well-defined structure.

SemanticChunker

Groups content based on semantic similarity. Best for preserving context and topical coherence.

LateChunker

Chunks using Late Chunking algorithm, best for higher recall in your RAG applications.

CodeChunker

Splits code based on its structure using ASTs. Ideal for chunking source code files.

TeraflopAIChunker

Segments text using the TeraflopAI Segmentation API. Ideal for domain-specific segmentation such as legal documents.

NeuralChunker

Uses a fine-tuned BERT model to split text based on semantic shifts. Great for topic-coherent chunks.

SlumberChunker

Agentic chunking using generative models (LLMs) via the Genie interface for S-tier chunk quality. 🦛🧞

TableChunker

Splits large markdown tables into smaller, manageable chunks by row, preserving headers. Great for tabular data in RAG and LLM pipelines.

​
Availability

Different chunkers are available depending on your installation:
ChunkerDefaultembeddings"chonkie[all]"Chonkie JSAPI Chunking
TokenChunker
FastChunker
SentenceChunker
RecursiveChunker
TableChunker
CodeChunker
SemanticChunker
LateChunker
NeuralChunker
SlumberChunker

​
Common Interface

All chunkers share a consistent interface: 
# Single text chunking
chunks = chunker.chunk(text)

# Batch processing
chunks = chunker.chunk_batch(texts)

# Direct calling
chunks = chunker(text)  # or chunker([text1, text2])

# Async variants (all chunkers support these)
chunks = await chunker.achunk(text)
chunks = await chunker.achunk_batch(texts)

​
Async Support

Every chunker supports async out of the box — no extra setup required.
MethodAsync EquivalentDescription
chunk(text)achunk(text)Chunk a single text
chunk_batch(texts)achunk_batch(texts)Chunk a list of texts
chunk_document(doc)achunk_document(doc)Chunk a Document object

​
Basic Usage

import asyncio
from chonkie import RecursiveChunker

async def main():
    chunker = RecursiveChunker(chunk_size=512)

    chunks = await chunker.achunk("Your document text here...")
    all_chunks = await chunker.achunk_batch([
        "First document...",
        "Second document...",
        "Third document...",
    ])

asyncio.run(main())

​
Concurrent Chunking

Use asyncio.gather to chunk multiple texts concurrently: 
import asyncio
from chonkie import SemanticChunker

async def process_documents(texts: list[str]):
    chunker = SemanticChunker(chunk_size=512)
    results = await asyncio.gather(
        *[chunker.achunk(text) for text in texts]
    )
    return results

​
How It Works

  • achunk and achunk_batch run the synchronous methods in a thread pool via asyncio.to_thread, so CPU-bound chunking does not block your event loop.
  • achunk_document goes further: if the document has pre-existing chunks, it dispatches a concurrent asyncio.gather over all of them.
Because achunk and achunk_batch use asyncio.to_thread, they are safe to use in async web frameworks (FastAPI, Starlette, aiohttp, etc.) without blocking the event loop.

​
F.A.Q.

Yes, all the chunkers are thread-safe. Though, the performance might vary since some chunkers use threading under the hood. So, monitor your performance accordingly.
No. Async support is built into every chunker via BaseChunker. Any chunker you import from chonkie already has achunk, achunk_batch, and achunk_document available.
Yes, especially when chunking many texts concurrently. achunk offloads work to a thread pool, so multiple coroutines can chunk in parallel without blocking the event loop. For single-text chunking the overhead is minimal.
Yes. All chunkers are thread-safe, so sharing a single instance across concurrent asyncio.gather calls is fine and avoids redundant initialization costs.
Any framework that uses asyncio — FastAPI, Starlette, aiohttp, Sanic, Litestar, and others. The async methods use standard asyncio primitives with no framework-specific dependencies.