> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chonkie.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Chunkers Overview
> Overview of the different chunkers available in Chonkie
Chonkie provides multiple chunking strategies to handle different text processing needs. Each chunker in Chonkie is designed to follow the same core principles outlined in the [concepts](/common/concepts) page.
Splits code based on its structure using ASTs. Ideal for chunking source
code files.
SIMD-accelerated byte-based chunking at 100+ GB/s. Best for high-throughput
pipelines where byte size limits are acceptable.
Chunks using Late Chunking algorithm, best for higher recall in your RAG
applications.
Uses a fine-tuned BERT model to split text based on semantic shifts. Great
for topic-coherent chunks.
Recursively chunks documents into smaller chunks. Best for long documents
with well-defined structure.
Groups content based on semantic similarity. Best for preserving context and
topical coherence.
Splits text at sentence boundaries. Perfect for maintaining semantic
completeness at the sentence level.
Agentic chunking using generative models (LLMs) via the Genie interface for
S-tier chunk quality. 🦛🧞
Splits large markdown tables into smaller, manageable chunks by row,
preserving headers. Great for tabular data in RAG and LLM pipelines.
Segments text using the TeraflopAI Segmentation API. Ideal for
domain-specific segmentation such as legal documents.
Splits text into fixed-size token chunks. Best for maintaining consistent
chunk sizes and working with token-based models.
## Availability
Different chunkers are available depending on your installation:
| Chunker | Default | embeddings | `"chonkie[all]"` | Chonkie JS | API Chunking |
| ---------------- | :----------------------------------------------------: | :----------------------------------------------------: | :----------------------------------------------------: | :----------------------------------------------------: | :----------------------------------------------------: |
| TokenChunker | | | | | |
| FastChunker | | | | | |
| SentenceChunker | | | | | |
| RecursiveChunker | | | | | |
| TableChunker | | | | | |
| CodeChunker | | | | | |
| SemanticChunker | | | | | |
| LateChunker | | | | | |
| NeuralChunker | | | | | |
| SlumberChunker | | | | | |
## Common Interface
All chunkers share a consistent interface:
```python Python theme={"system"}
# Single text chunking
chunks = chunker.chunk(text)
# Batch processing
chunks = chunker.chunk_batch(texts)
# Direct calling
chunks = chunker(text) # or chunker([text1, text2])
# Async variants (all chunkers support these)
chunks = await chunker.achunk(text)
chunks = await chunker.achunk_batch(texts)
```
```javascript JavaScript theme={"system"}
// Single text chunking
const chunks = await chunker.chunk(text);
```
## Async Support
Every chunker supports async out of the box — no extra setup required.
| Method | Async Equivalent | Description |
| --------------------- | ---------------------- | ------------------------- |
| `chunk(text)` | `achunk(text)` | Chunk a single text |
| `chunk_batch(texts)` | `achunk_batch(texts)` | Chunk a list of texts |
| `chunk_document(doc)` | `achunk_document(doc)` | Chunk a `Document` object |
### Basic Usage
```python theme={"system"}
import asyncio
from chonkie import RecursiveChunker
async def main():
chunker = RecursiveChunker(chunk_size=512)
chunks = await chunker.achunk("Your document text here...")
all_chunks = await chunker.achunk_batch([
"First document...",
"Second document...",
"Third document...",
])
asyncio.run(main())
```
### Concurrent Chunking
Use `asyncio.gather` to chunk multiple texts concurrently:
```python theme={"system"}
import asyncio
from chonkie import SemanticChunker
async def process_documents(texts: list[str]):
chunker = SemanticChunker(chunk_size=512)
results = await asyncio.gather(
*[chunker.achunk(text) for text in texts]
)
return results
```
### How It Works
* **`achunk` and `achunk_batch`** run the synchronous methods in a thread pool via `asyncio.to_thread`, so CPU-bound chunking does not block your event loop.
* **`achunk_document`** goes further: if the document has pre-existing chunks, it dispatches a concurrent `asyncio.gather` over all of them.
Because `achunk` and `achunk_batch` use `asyncio.to_thread`, they are safe to use in async web frameworks (FastAPI, Starlette, aiohttp, etc.) without blocking the event loop.
## F.A.Q.
Yes, all the chunkers are thread-safe. Though, the performance might vary
since some chunkers use threading under the hood. So, monitor your
performance accordingly.
No. Async support is built into every chunker via `BaseChunker`. Any chunker you import from `chonkie` already has `achunk`, `achunk_batch`, and `achunk_document` available.
Yes, especially when chunking many texts concurrently. `achunk` offloads work to a thread pool, so multiple coroutines can chunk in parallel without blocking the event loop. For single-text chunking the overhead is minimal.
Yes. All chunkers are thread-safe, so sharing a single instance across concurrent `asyncio.gather` calls is fine and avoids redundant initialization costs.
Any framework that uses `asyncio` — FastAPI, Starlette, aiohttp, Sanic, Litestar, and others. The async methods use standard `asyncio` primitives with no framework-specific dependencies.