Chonkie Documentation

Chonkie provides multiple chunking strategies to handle different text processing needs. Each chunker in Chonkie is designed to follow the same core principles outlined in the concepts page.

Not sure which chunker to use? Check out the Choosing a Chunker guide for recommendations based on your use case.

CodeChunker

Splits code based on its structure using ASTs. Ideal for chunking source code files.

FastChunker

SIMD-accelerated byte-based chunking at 100+ GB/s. Best for high-throughput pipelines where byte size limits are acceptable.

LateChunker

Chunks using Late Chunking algorithm, best for higher recall in your RAG applications.

NeuralChunker

Uses a fine-tuned BERT model to split text based on semantic shifts. Great for topic-coherent chunks.

RecursiveChunker

Recursively chunks documents into smaller chunks. Best for long documents with well-defined structure.

SemanticChunker

Groups content based on semantic similarity. Best for preserving context and topical coherence.

SentenceChunker

Splits text at sentence boundaries. Perfect for maintaining semantic completeness at the sentence level.

SlumberChunker

Agentic chunking using generative models (LLMs) via the Genie interface for S-tier chunk quality. 🦛🧞

TableChunker

Splits large markdown tables into smaller, manageable chunks by row, preserving headers. Great for tabular data in RAG and LLM pipelines.

TeraflopAIChunker

Segments text using the TeraflopAI Segmentation API. Ideal for domain-specific segmentation such as legal documents.

TokenChunker

Splits text into fixed-size token chunks. Best for maintaining consistent chunk sizes and working with token-based models.

Availability

Different chunkers are available depending on your installation:

Chunker	Default	embeddings	`"chonkie[all]"`	Chonkie JS	API Chunking
TokenChunker
FastChunker
SentenceChunker
RecursiveChunker
TableChunker
CodeChunker
SemanticChunker
LateChunker
NeuralChunker
SlumberChunker

Common Interface

All chunkers share a consistent interface:

# Single text chunking
chunks = chunker.chunk(text)

# Batch processing
chunks = chunker.chunk_batch(texts)

# Direct calling
chunks = chunker(text)  # or chunker([text1, text2])

# Async variants (all chunkers support these)
chunks = await chunker.achunk(text)
chunks = await chunker.achunk_batch(texts)

// Single text chunking
const chunks = await chunker.chunk(text);

Async Support

Every chunker supports async out of the box — no extra setup required.

Method	Async Equivalent	Description
`chunk(text)`	`achunk(text)`	Chunk a single text
`chunk_batch(texts)`	`achunk_batch(texts)`	Chunk a list of texts
`chunk_document(doc)`	`achunk_document(doc)`	Chunk a `Document` object

Basic Usage

import asyncio
from chonkie import RecursiveChunker

async def main():
    chunker = RecursiveChunker(chunk_size=512)

    chunks = await chunker.achunk("Your document text here...")
    all_chunks = await chunker.achunk_batch([
        "First document...",
        "Second document...",
        "Third document...",
    ])

asyncio.run(main())

Concurrent Chunking

Use asyncio.gather to chunk multiple texts concurrently:

import asyncio
from chonkie import SemanticChunker

async def process_documents(texts: list[str]):
    chunker = SemanticChunker(chunk_size=512)
    results = await asyncio.gather(
        *[chunker.achunk(text) for text in texts]
    )
    return results

How It Works

achunk and achunk_batch run the synchronous methods in a thread pool via asyncio.to_thread, so CPU-bound chunking does not block your event loop.
achunk_document goes further: if the document has pre-existing chunks, it dispatches a concurrent asyncio.gather over all of them.

Because achunk and achunk_batch use asyncio.to_thread, they are safe to use in async web frameworks (FastAPI, Starlette, aiohttp, etc.) without blocking the event loop.

F.A.Q.

Are all the chunkers thread-safe?

Yes, all the chunkers are thread-safe. Though, the performance might vary since some chunkers use threading under the hood. So, monitor your performance accordingly.

Do I need to install anything extra for async support?

No. Async support is built into every chunker via BaseChunker. Any chunker you import from chonkie already has achunk, achunk_batch, and achunk_document available.

Does async chunking improve throughput?

Yes, especially when chunking many texts concurrently. achunk offloads work to a thread pool, so multiple coroutines can chunk in parallel without blocking the event loop. For single-text chunking the overhead is minimal.

Which async frameworks are supported?

Any framework that uses asyncio — FastAPI, Starlette, aiohttp, Sanic, Litestar, and others. The async methods use standard asyncio primitives with no framework-specific dependencies.

Getting Started

API Server

Chefs

Fetchers

Chunkers

Embeddings

Refinery

Handshakes

Porters

Utils

Experimental

Changelog

Chunkers Overview

CodeChunker

FastChunker

LateChunker

NeuralChunker

RecursiveChunker

SemanticChunker

SentenceChunker

SlumberChunker

TableChunker

TeraflopAIChunker

TokenChunker

Availability

Common Interface

Async Support

Basic Usage

Concurrent Chunking

How It Works

F.A.Q.

CodeChunker

FastChunker

LateChunker

NeuralChunker

RecursiveChunker

SemanticChunker

SentenceChunker

SlumberChunker

TableChunker

TeraflopAIChunker

TokenChunker

​Availability

​Common Interface

​Async Support

​Basic Usage

​Concurrent Chunking

​How It Works

​F.A.Q.

Availability

Common Interface

Async Support

Basic Usage

Concurrent Chunking

How It Works

F.A.Q.