# Concepts
Source: https://docs.chonkie.ai/common/concepts
Common concepts of Chonkie
This page outlines some common concepts of Chonkie, that will help you understand how to use Chonkie effectively.
## What are Chonkie's core values?
Chonkie is a very opinionated library, and it all stems from innate human mortality. We are all going to die one day, and we have no reason to waste time figuring out how to chunk documents. Just use Chonkie.
Chonkie needs to be and always adheres to be:
* **Simple**: We care about how simple it is to use Chonkie. No brainer.
* **Fast**: We care about your latency. No time to waste.
* **Lightweight**: We care about your memory. No space to waste.
* **Flexible**: We care about your customization needs. Hassle free.
Chonkie just works. It's that simple.
## What is chunking? What is an ideal chunk and chunker?
Chunking is the process of breaking down a text into smaller, more manageable pieces, that can be used for RAG applications.
An ideal chunk is one that is:
* **Reconstructable**: A chunk should be part of the whole text, such that combining chunks should give you the original text back.
* **Independent**: It should be a standalone unit tackling only one idea, i.e., removing it from the chunk should not remove important information from the original text.
* **Sufficient**: It should be long enough to be meaningful, i.e., it should contain enough information to be useful.
As a consequence, an (ideal) chunker is one that:
* Breaks down the text into chunks that are reconstructable, independent and sufficient.
* Is deterministic, i.e., given the same text, it should always return the same chunks.
* Is efficient, i.e., it should be fast and lightweight.
This is how Chonkie's chunkers are designed to be. Understanding this will help you understand why Chonkie divides the chunking process into multiple stages: Pre-processing, chunking and post-processing.
## Is chunking necessary? Can't I just use the entire document as a chunk?
Nope. Here’s why chunking is absolutely essential:
#### 1. **Limited Context Windows**
All models have a limit on how much text they can process at once.
This is referred to as their "context window".
Chunking breaks down large documents into manageable pieces that fit within these limits.
#### 2. **Computational Efficiency**
Processing a 100GB document every time you make a query? Bad idea.
Attention mechanisms, even when optimized, are computationally expensive (`O(n)`).
Chunking keeps things efficient and memory-friendly.
#### 3. **Better Representation**
As mentioned earlier, chunks represent each idea as an independent entity.
Not chunking your document will likely cause your model to conflate concepts and get confused.
Representation models use lossy compression, so keeping chunks concise ensures the model understands the context better.
#### 4. **Reduced Hallucination**
Feeding too much context at once makes models hallucinate.
They start using irrelevant information to answer queries, and that’s a **big** no-no.
Smaller, focused chunks reduce this risk.
All of this makes chunking a **must-have** for RAG applications. Don't get caught using your whole document as a single chunk!
## How is Chonkie so fast? What is the secret sauce?
Chonkie is fast because it cares about your latency. Chonkie is raised with love, care, and strong beliefs that the speed of light should be the only limit to your RAG applications.
Of course, we do a lot of optimizations under the hood to make sure that Chonkie is as fast as it gets. Here are some of the things we do:
* **Pipelining**: We use a pipelining approach to process the document, so as to make stronger heuristics for chunking. This allows to have a faster chunking process, without compromising on the quality of chunks.
* **Caching and Pre-computation**: We cache the results of the chunking process, so as to avoid re-computation. This allows to have a faster chunking process, without compromising on the quality of chunks.
* **Smart Token Estimate-Validate feedback Loops**: We use a token estimate-validate feedback loops to make sure that we have near optimal chunk sizes, while bypassing some of the inefficiencies of the tokenizers.
* **Faster Tokenizers**: We use a faster tokenizer, [tiktoken](https://github.com/openai/tiktoken), which is faster and more efficient than the default tokenizer. Tiktoken by default does not support all model types, so we use a wrapper around it, AutoTikTokenizer which adds support for all HF models.
* **Ultra-fast embedding**: By default, Chonkie uses Static Embeddings from Model2Vec, which are ultra-fast and lightweight. Static Embeddings are pre-computed and stored in a lookup table, so as to avoid the overhead of running an embedding model at query time.
* **Parallel Processing**: We use parallel processing to process the document in parallel, so as to make better use of the available resources. This allows to have a faster chunking process, without compromising on the quality of chunks.
All these optimizations allow Chonkie to process documents at the speed of light, without compromising on the quality of chunks. So, the next time you want to process a large document, remember to use Chonkie!
# Open Source
Source: https://docs.chonkie.ai/common/open-source
The Open Source Library For RAG
*✨Look Inside! We're Open Source!✨*
**Chonkie's Open Source library** provides lightweight, and high-performance features for building modern RAG applications.
Install it locally, run anywhere, and keep full control over your chunking pipeline.
## Why Chonkie OSS?
Released under the MIT license. Use however you like.
All processing happens locally. Your data never leaves your infrastructure.
Battle-tested algorithms used by thousands of developers. Optimized for
speed and reliability.
Optimized with caching, parallel processing, and fast tokenizers. Process
millions of chunks efficiently.
## Core Capabilities
### Advanced Chunkers
Chonkie OSS includes a comprehensive suite of chunking algorithms, each designed for specific document types and use cases:
**Best for**: General-purpose chunking, most use cases
Splits text into fixed-size token chunks with configurable overlap. The most straightforward and reliable chunking strategy.
Available in: Python, JavaScript
**Best for**: Q\&A systems, maintaining complete thoughts
Chunks at sentence boundaries while respecting token limits. Ensures sentences are never split mid-thought.
Available in: Python, JavaScript
**Best for**: Markdown, structured documents, hierarchical content
Hierarchically chunks using multiple delimiters—paragraphs, then sentences, then words. Preserves document structure naturally.
Available in: Python, JavaScript
**Best for**: High-throughput pipelines, large-scale document processing
SIMD-accelerated chunking with 100+ GB/s throughput. Uses byte-size limits for extreme performance without tokenization overhead.
Available in: Python, JavaScript
**Best for**: Markdown tables, tabular data
Splits large tables into manageable chunks by rows while preserving headers. Perfect for data-heavy documents.
Available in: Python, JavaScript
**Best for**: Multi-topic documents, maintaining topical coherence
Uses embeddings to identify natural topic boundaries. Creates chunks based on semantic similarity, not just structure. Includes Savitzky-Golay filtering and skip-window merging for advanced boundary detection.
Available in: Python, JavaScript
**Best for**: Retrieval optimization, higher recall RAG systems
Implements the Late Chunking algorithm from research. Generates document-level embeddings first, then derives chunk embeddings for richer contextual representation.
Available in: Python
**Best for**: Source code, API documentation, technical content
Language-aware chunking using Abstract Syntax Trees (AST). Preserves function and class boundaries for better code understanding.
Available in: Python, JavaScript
**Best for**: Maximum quality, complex documents with subtle topic shifts
Uses a fine-tuned BERT model to detect semantic shifts in text. ML-powered boundary detection for topic-coherent chunks.
Available in: Python
**Best for**: Books, research papers, when quality matters most
Agentic chunking powered by LLMs via the Genie interface. Uses generative models (Gemini, OpenAI, etc.) to intelligently determine optimal chunk boundaries.
Available in: Python
### Embedding Providers
Flexible embedding support for semantic chunking and refineries:
* **AutoEmbeddings** - Automatically select the best embeddings for your use case
* **Model2VecEmbeddings** - Ultra-fast static embeddings (default for semantic chunking)
* **SentenceTransformerEmbeddings** - Hugging Face Sentence Transformers models
* **OpenAIEmbeddings** - OpenAI's text-embedding models
* **AzureOpenAIEmbeddings** - Azure-hosted OpenAI embeddings
* **CohereEmbeddings** - Cohere's embedding models
* **JinaEmbeddings** - Jina AI embeddings
* **GeminiEmbeddings** - Google Gemini embeddings
* **VoyageAIEmbeddings** - Voyage AI embeddings
* **Custom Embeddings** - Bring your own embedding model
All embeddings follow a consistent interface and can be swapped seamlessly.
### Refineries
Enhance your chunks with additional context and embeddings:
Adds contextual overlap between chunks to prevent information loss at boundaries. Configurable overlap sizes for optimal retrieval.
Generates and attaches vector embeddings to your chunks. Supports all major embedding providers with automatic dimension detection.
### Database Handshakes
Seamlessly connect Chonkie to your favorite database:
Ephemeral or persistent ChromaDB instances
High-performance vector search with Qdrant
Knowledge graph + vector search with Weaviate
Serverless vector database by Turbopuffer
Managed vector database with Pinecone
PostgreSQL with pgvector extension
MongoDB Atlas Vector Search
Elasticsearch vector search
Each handshake provides a simple interface to embed chunks and write them directly to your database.
### Chefs
Chefs automatically prepare raw data for chunking:
* **TableChef** - Extracts tables from markdown text
* **TextChef** - Processes plain text files into structured Documents
* **MarkdownChef** - Parses markdown with tables, code blocks, and images
### Porters
Export chunks to common formats:
* **JSONPorter** - Export chunks to JSON for storage or processing
* **DatasetsPorter** - Export to Hugging Face Datasets format
### Utils
* **Visualizer** - Rich text visualization of chunks with color-coded boundaries
* **Hubbie** - Hugging Face Hub integration for sharing and loading chunkers
## Language Support
**Full Feature Set**
All chunkers, embedding providers, refineries, handshakes, chefs, and porters available. Choose from minimal to full installations based on your needs.
* Default install: Token, Sentence, Recursive, Table chunkers
* Semantic install: + SemanticChunker, LateChunker, NeuralChunker with Model2Vec
* All install: Every feature available
**Core Chunking**
JavaScript support includes the most commonly used chunkers:
* TokenChunker
* SentenceChunker
* RecursiveChunker
* FastChunker
* TableChunker
* SemanticChunker
* CodeChunker
Available via `@chonkiejs/core` package with full TypeScript support. Other chunkers available through the Chonkie Cloud API via `@chonkiejs/cloud`.
To use custom tokenizers with the chunkers, install `@chonkiejs/token`
## Performance Characteristics
Chonkie OSS is optimized for speed:
* **Pipelining** - Efficient multi-stage processing
* **Caching** - Smart caching to avoid recomputation
* **Fast Tokenizers** - TikToken and AutoTikTokenizer for speed
* **Parallel Processing** - Multi-threaded batch operations
* **Ultra-fast Embeddings** - Model2Vec static embeddings (default)
* **Token Estimate-Validate** - Efficient feedback loops for optimal chunk sizes
Process thousands of documents per second on commodity hardware.
## Next Steps
Ready to get started with Chonkie OSS?
Install and create your first chunk in under 2 minutes
Detailed installation options for all features
Explore all chunking algorithms in detail
Star the repo and contribute to the project
***
Need hosted chunking with zero setup? Check out our [Chunking API](/common/chunking-api) for a managed solution.
# 🦛 Chonkie ✨
Source: https://docs.chonkie.ai/common/welcome
The lightweight ingestion library for fast, efficient and robust RAG pipelines
Install via npx skills add chonkie-inc/skills or browse on skills.sh — works with 20+ AI coding agents.
Ever found yourself making a RAG pipeline yet again (your 2,342,148th one), only to realize you're stuck having to write your ingestion logic with bloated software library X or the painfully feature-less library Y?
*WHY CAN'T THIS JUST BE SIMPLE, UGH?*
Well, look no further than Chonkie! (chonkie boi is a gud boi 🦛)
Clean, CHONK, Embed, Refine and Store your data - all from one library!
Install, Import, CHONK - it's that simple!
CHONK at the speed of light! zooooooooom
Supports your favorite tokenizers, chunkers, embeddings and vector DBs
CHONK in Python, JavaScript or via our API. Chonkie is there wherever you
need it
psst it's a pygmy hippo btw! Moto Moto approved
Install via `npx skills add chonkie-inc/skills` or browse on skills.sh — works with Claude Code, Cursor, Copilot, and 20+ agents
***
# Docker
Source: https://docs.chonkie.ai/oss/api/docker
Deploy the Chonkie API server with Docker and docker-compose
## Quick Start
```bash theme={"system"}
docker compose up
```
The API is available at `http://localhost:8000`. Visit `/docs` for the interactive Swagger UI.
## docker-compose.yml
The repository ships with a ready-to-use `docker-compose.yml`:
```yaml theme={"system"}
services:
chonkie-api:
build:
context: .
dockerfile: Dockerfile
image: chonkie-oss-api:latest
container_name: chonkie-api
ports:
- "8000:8000"
volumes:
- ./data:/app/data
environment:
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
CORS_ORIGINS: "${CORS_ORIGINS:-*}"
DATABASE_URL: "sqlite+aiosqlite:////app/data/chonkie.db"
OPENAI_API_KEY: "${OPENAI_API_KEY:-}"
COHERE_API_KEY: "${COHERE_API_KEY:-}"
VOYAGE_API_KEY: "${VOYAGE_API_KEY:-}"
MISTRAL_API_KEY: "${MISTRAL_API_KEY:-}"
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c",
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
```
The `./data` volume mount persists the SQLite database (`chonkie.db`) across container restarts.
## Environment Variables
| Variable | Default | Description |
| ----------------- | --------------------------------------- | ------------------------------------------------------ |
| `LOG_LEVEL` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `CORS_ORIGINS` | `*` | Comma-separated allowed origins. Use `*` to allow all. |
| `DATABASE_URL` | `sqlite+aiosqlite:///./data/chonkie.db` | SQLite database path. Override for custom locations. |
| `OPENAI_API_KEY` | *(empty)* | For OpenAI embeddings (`text-embedding-3-small`, etc.) |
| `COHERE_API_KEY` | *(empty)* | For Cohere embeddings (`embed-english-v3.0`, etc.) |
| `VOYAGE_API_KEY` | *(empty)* | For Voyage AI embeddings (`voyage-large-2`, etc.) |
| `MISTRAL_API_KEY` | *(empty)* | For Mistral embeddings (`mistral-embed`) |
Pass them inline:
```bash theme={"system"}
LOG_LEVEL=DEBUG CORS_ORIGINS=https://myapp.com docker compose up
```
Or create a `.env` file in the project root:
```bash theme={"system"}
LOG_LEVEL=INFO
CORS_ORIGINS=https://myapp.com,https://api.myapp.com
# Set your preferred embedding provider key:
OPENAI_API_KEY=sk-...
# COHERE_API_KEY=...
# VOYAGE_API_KEY=...
```
## Build and Run Without Compose
```bash theme={"system"}
# Build the image
docker build -t chonkie-oss-api .
# Run the container
docker run -p 8000:8000 chonkie-oss-api
# With environment variables
docker run -p 8000:8000 \
-e LOG_LEVEL=DEBUG \
-e OPENAI_API_KEY=sk-... \
chonkie-oss-api
```
## Image Details
The Dockerfile uses a multi-stage build to keep the final image lean:
* **Builder stage** — installs `chonkie[api,semantic,code,openai]` into a virtual environment
* **Runtime stage** — copies only the venv; runs as a non-root `chonkie` user
* **Exposed port** — `8000`
* **Health check** — HTTP GET to `/health` every 30 seconds
## Production Tips
**Restrict CORS** — in production, replace `*` with your actual domains:
```bash theme={"system"}
CORS_ORIGINS=https://myapp.com,https://admin.myapp.com docker compose up
```
**Add a reverse proxy** — put Nginx or Caddy in front for TLS termination and rate limiting:
```nginx theme={"system"}
server {
listen 443 ssl;
server_name api.myapp.com;
location / {
proxy_pass http://chonkie-api:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```
**Scale horizontally** — run multiple replicas behind a load balancer:
```yaml theme={"system"}
services:
chonkie-api:
image: chonkie-oss-api:latest
deploy:
replicas: 3
ports:
- "8000:8000"
```
The `SemanticChunker` loads its embedding model on first use. Send a warm-up request after startup to avoid cold-start latency on the first real request in production.
# Endpoints
Source: https://docs.chonkie.ai/oss/api/endpoints
API reference for all Chonkie chunkers and refineries
Start the server and visit `http://localhost:8000/docs` for an interactive Swagger UI where you can try every endpoint directly in your browser.
## Response Format
All chunking endpoints return a list of chunk objects:
```json theme={"system"}
[
{
"text": "chunk content",
"start_index": 0,
"end_index": 42,
"token_count": 8
}
]
```
Submit a **list of strings** instead of a single string to get back a **list of lists** — one inner list per input document.
***
## Chunkers
### Token Chunker
`POST /v1/chunk/token`
Splits text into fixed-size token windows. The fastest and most predictable chunker.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/token \
-H "Content-Type: application/json" \
-d '{
"text": "Your text here...",
"chunk_size": 512,
"chunk_overlap": 50
}'
```
Text or list of texts to chunk.
Tokenizer to use. Options: `"character"`, `"gpt2"`, `"cl100k_base"`, or any HuggingFace tokenizer name.
Maximum tokens per chunk.
Token overlap between consecutive chunks.
***
### Sentence Chunker
`POST /v1/chunk/sentence`
Groups sentences into chunks while respecting a token-size limit. Preserves sentence boundaries — no mid-sentence splits.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/sentence \
-H "Content-Type: application/json" \
-d '{
"text": "First sentence. Second sentence. Third sentence.",
"chunk_size": 256,
"min_sentences_per_chunk": 2
}'
```
Text or list of texts to chunk.
Tokenizer to use.
Maximum tokens per chunk.
Token overlap between chunks.
Minimum sentences to include in each chunk.
Minimum characters required to count as a sentence.
Use approximate token counting for faster processing.
Sentence delimiter(s).
Attach the delimiter to the previous (`"prev"`) or next (`"next"`) sentence.
***
### Recursive Chunker
`POST /v1/chunk/recursive`
Splits text using a hierarchy of separators defined by a named recipe. Great for structured text like Markdown or code. Chunker instances are cached per `(recipe, lang, tokenizer)` for speed.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/recursive \
-H "Content-Type: application/json" \
-d '{
"text": "# Heading\n\nParagraph one.\n\nParagraph two.",
"chunk_size": 256,
"recipe": "markdown"
}'
```
Text or list of texts to chunk.
Tokenizer to use.
Maximum tokens per chunk.
Named splitting recipe. Options: `"default"` (paragraph → sentence → word), `"markdown"`, `"python"`, `"js"`.
Language hint for the recipe.
Minimum characters to include in a chunk.
***
### Semantic Chunker
`POST /v1/chunk/semantic`
Splits where semantic similarity between adjacent sentences drops below a threshold. Produces topically coherent chunks. Requires the `semantic` extra.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/semantic \
-H "Content-Type: application/json" \
-d '{
"text": "Dogs are loyal and friendly pets. Cats are independent animals. Quantum physics studies subatomic particles.",
"embedding_model": "minishlab/potion-base-8M",
"threshold": 0.5
}'
```
Text or list of texts to chunk.
Sentence-embedding model for computing similarity. Any model compatible with `sentence-transformers` works.
Cosine-similarity threshold for splitting (0.0–1.0). Lower values produce larger, fewer chunks.
Maximum tokens per chunk.
Number of surrounding sentences to consider when computing similarity.
Minimum sentences per chunk.
Minimum characters per sentence.
***
### Code Chunker
`POST /v1/chunk/code`
Splits source code at syntactic boundaries using AST parsing. Never breaks inside a function or class. Requires the `code` extra.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/code \
-H "Content-Type: application/json" \
-d '{
"text": "def hello():\n print(\"Hello\")\n\ndef world():\n print(\"World\")",
"language": "python",
"chunk_size": 100
}'
```
Source code or list of source code snippets to chunk.
Tokenizer to use.
Maximum tokens per chunk.
Programming language. Supported: `"python"`, `"javascript"`, `"typescript"`, `"java"`, `"go"`, `"rust"`, `"c"`, `"cpp"`, and more.
Include AST node metadata (node type, line numbers) in the chunk output.
***
## Refineries
Refineries enrich an existing list of chunks. Pass the output of any chunker endpoint directly into a refinery.
### Overlap Refinery
`POST /v1/refine/overlap`
Appends or prepends overlapping context from neighbouring chunks. Useful when downstream consumers need continuity across chunk boundaries.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/refine/overlap \
-H "Content-Type: application/json" \
-d '{
"chunks": [
{"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3},
{"text": "Second chunk.", "start_index": 13, "end_index": 26, "token_count": 3}
],
"context_size": 0.25,
"method": "suffix"
}'
```
List of chunk objects from any chunker endpoint. Each must contain `text`, `start_index`, `end_index`, and `token_count`.
Tokenizer to use.
Size of the overlap context. A float (0–1) is treated as a fraction of the chunk size; an integer is an absolute token count.
Strategy used to create the overlap window.
`"suffix"` appends context from the next chunk; `"prefix"` prepends context from the previous chunk; `"justified"` adds context from both sides.
Merge the overlap context into the chunk text field.
***
### Embeddings Refinery
`POST /v1/refine/embeddings`
Computes and attaches embeddings to each chunk via Chonkie's `AutoEmbeddings`. Each chunk in the response gains an `embedding` field containing a list of floats.
**Local models** (e.g. `minishlab/potion-base-8M`) run entirely on-device and require no API key. **API-based models** require the appropriate environment variable for your provider.
```bash theme={"system"}
# Local model (no API key required)
curl -X POST http://localhost:8000/v1/refine/embeddings \
-H "Content-Type: application/json" \
-d '{
"chunks": [
{"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3},
{"text": "Second chunk.", "start_index": 13, "end_index": 26, "token_count": 3}
],
"embedding_model": "minishlab/potion-base-8M"
}'
# OpenAI (requires OPENAI_API_KEY)
curl -X POST http://localhost:8000/v1/refine/embeddings \
-H "Content-Type: application/json" \
-d '{
"chunks": [
{"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3}
],
"embedding_model": "text-embedding-3-small"
}'
```
## Embeddings Providers
| Type | Example Model | Requirement |
| ----------------- | ------------------------------------------------------------ | ---------------- |
| Local (model2vec) | `minishlab/potion-base-8M`, `minishlab/potion-retrieval-32M` | None |
| OpenAI | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` |
| Cohere | `embed-english-v3.0`, `embed-multilingual-v3.0` | `COHERE_API_KEY` |
| Voyage AI | `voyage-large-2`, `voyage-code-2` | `VOYAGE_API_KEY` |
List of chunk objects to embed.
Embedding model name. Local model2vec models (e.g. `minishlab/potion-base-8M`) require no API key. For API-based models, set the appropriate environment variable for your provider.
***
## Batch Processing
Send a list of strings to process multiple documents in one request:
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/token \
-H "Content-Type: application/json" \
-d '{
"text": ["First document.", "Second document.", "Third document."],
"chunk_size": 512
}'
```
The response is a **list of lists** — one inner list of chunks per input document:
```json theme={"system"}
[
[{"text": "First document.", "start_index": 0, "end_index": 15, "token_count": 3}],
[{"text": "Second document.", "start_index": 0, "end_index": 16, "token_count": 3}],
[{"text": "Third document.", "start_index": 0, "end_index": 15, "token_count": 3}]
]
```
***
## Chaining Chunkers and Refineries
Pipeline example — chunk semantically, then add overlap context:
```python theme={"system"}
import requests
BASE = "http://localhost:8000"
# Step 1: chunk
chunks = requests.post(f"{BASE}/v1/chunk/semantic", json={
"text": "Your long document here...",
"threshold": 0.5,
}).json()
# Step 2: add overlap
enriched = requests.post(f"{BASE}/v1/refine/overlap", json={
"chunks": chunks,
"context_size": 0.2,
}).json()
# Step 3: embed (requires OPENAI_API_KEY)
embedded = requests.post(f"{BASE}/v1/refine/embeddings", json={
"chunks": enriched,
"embedding_model": "text-embedding-3-small",
}).json()
```
***
## Error Handling
| Status | Meaning |
| ------ | ------------------------------------------------------------ |
| `200` | Success |
| `400` | Invalid request parameters or chunk format |
| `500` | Internal error (missing extras, model loading failure, etc.) |
Error responses follow FastAPI's standard format:
```json theme={"system"}
{
"detail": "SemanticChunker requires the 'semantic' extra. Install it with: pip install 'chonkie[semantic]'"
}
```
***
## Health & Info
```bash theme={"system"}
# Health check (used by load balancers and container orchestrators)
curl http://localhost:8000/health
# {"status": "ok"}
# API info
curl http://localhost:8000/
# {"name": "Chonkie OSS API", "version": "...", "docs": "/docs", ...}
```
# API Server
Source: https://docs.chonkie.ai/oss/api/overview
Self-host Chonkie as a REST API for language-agnostic text chunking
Run Chonkie as a self-hosted REST API and call any chunker or refinery from any language, framework, or HTTP client — no auth, no billing, no data leaving your infra.
Up and running in under a minute
All chunkers and refineries
Save and execute reusable chunking workflows
Container and production deployment
## Why Use the API?
* **Language-agnostic** — call from JavaScript, Go, Ruby, or any HTTP client
* **Self-hosted** — your data never leaves your infrastructure
* **Full feature parity** — all Chonkie chunkers and refineries, over HTTP
* **Batch support** — chunk multiple documents in a single request
* **No auth required** — just run it and chunk away
## Available Endpoints
| Endpoint | Description |
| --------------------------------- | ---------------------------------- |
| `POST /v1/chunk/token` | Fixed-size token windows |
| `POST /v1/chunk/sentence` | Sentence-boundary splitting |
| `POST /v1/chunk/recursive` | Structural/hierarchical splitting |
| `POST /v1/chunk/semantic` | Embedding-based semantic splitting |
| `POST /v1/chunk/code` | AST-aware code splitting |
| `POST /v1/refine/overlap` | Add overlap context to chunks |
| `POST /v1/refine/embeddings` | Attach embeddings to chunks |
| `POST /v1/pipelines` | Create a reusable pipeline |
| `GET /v1/pipelines` | List all pipelines |
| `GET /v1/pipelines/{id}` | Get a pipeline by ID |
| `PUT /v1/pipelines/{id}` | Update a pipeline |
| `DELETE /v1/pipelines/{id}` | Delete a pipeline |
| `POST /v1/pipelines/{id}/execute` | Execute a pipeline on text |
| `GET /health` | Health check |
| `GET /` | API info and available endpoints |
# Pipelines
Source: https://docs.chonkie.ai/oss/api/pipelines
Store and manage reusable chunking pipeline configurations
## What Are Pipelines?
A pipeline is a named, reusable configuration that describes a sequence of chunking and refinement steps. Instead of passing the same configuration on every request, you define it once and reference it by ID.
A pipeline step is either:
* **`chunk`** — runs a chunker (e.g. `"semantic"`, `"token"`, `"recursive"`)
* **`refine`** — runs a refinery (e.g. `"embeddings"`, `"overlap"`)
## Create a Pipeline
`POST /v1/pipelines`
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/pipelines \
-H "Content-Type: application/json" \
-d '{
"name": "rag-chunker",
"description": "Semantic chunking with embeddings for RAG",
"steps": [
{
"type": "chunk",
"chunker": "semantic",
"config": {"chunk_size": 512, "threshold": 0.5}
},
{
"type": "refine",
"refinery": "embeddings",
"config": {"embedding_model": "text-embedding-3-small"}
}
]
}'
```
**Response (201 Created):**
```json theme={"system"}
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "rag-chunker",
"description": "Semantic chunking with embeddings for RAG",
"config": {
"steps": [
{"type": "chunk", "chunker": "semantic", "refinery": null, "config": {"chunk_size": 512, "threshold": 0.5}},
{"type": "refine", "chunker": null, "refinery": "embeddings", "config": {"embedding_model": "text-embedding-3-small"}}
]
},
"created_at": "2026-02-20T10:00:00.000000",
"updated_at": "2026-02-20T10:00:00.000000"
}
```
Unique pipeline name. Used as a human-readable identifier.
Optional description of what this pipeline does.
Ordered list of steps to execute. Each step has:
* `type`: `"chunk"` or `"refine"`
* `chunker`: chunker name (for `chunk` steps, e.g. `"semantic"`, `"token"`)
* `refinery`: refinery name (for `refine` steps, e.g. `"embeddings"`, `"overlap"`)
* `config`: step-specific parameters (same fields as the individual endpoints)
***
## List Pipelines
`GET /v1/pipelines`
```bash theme={"system"}
curl http://localhost:8000/v1/pipelines
```
Returns all pipelines ordered by creation date (newest first).
***
## Get a Pipeline
`GET /v1/pipelines/{pipeline_id}`
```bash theme={"system"}
curl http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000
```
***
## Update a Pipeline
`PUT /v1/pipelines/{pipeline_id}`
You can update `name`, `description`, or `steps` independently:
```bash theme={"system"}
curl -X PUT http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000 \
-H "Content-Type: application/json" \
-d '{
"description": "Updated description",
"steps": [
{
"type": "chunk",
"chunker": "recursive",
"config": {"chunk_size": 1024, "recipe": "markdown"}
}
]
}'
```
***
## Delete a Pipeline
`DELETE /v1/pipelines/{pipeline_id}`
```bash theme={"system"}
curl -X DELETE http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000
```
Returns `204 No Content` on success.
***
## Pipeline Examples
### Basic Token Chunking
```json theme={"system"}
{
"name": "token-basic",
"steps": [
{"type": "chunk", "chunker": "token", "config": {"chunk_size": 512}}
]
}
```
### Markdown Documents with Overlap
```json theme={"system"}
{
"name": "markdown-with-overlap",
"description": "Recursive markdown chunking with overlap context",
"steps": [
{
"type": "chunk",
"chunker": "recursive",
"config": {"chunk_size": 512, "recipe": "markdown"}
},
{
"type": "refine",
"refinery": "overlap",
"config": {"context_size": 0.2, "method": "suffix"}
}
]
}
```
### Full RAG Pipeline
```json theme={"system"}
{
"name": "full-rag",
"description": "Semantic chunking + overlap + embeddings",
"steps": [
{
"type": "chunk",
"chunker": "semantic",
"config": {"chunk_size": 512, "threshold": 0.5}
},
{
"type": "refine",
"refinery": "overlap",
"config": {"context_size": 0.1}
},
{
"type": "refine",
"refinery": "embeddings",
"config": {"embedding_model": "voyage-large-2"}
}
]
}
```
***
## Storage
Pipelines are stored in a local SQLite database (`data/chonkie.db`). The database is created automatically on first startup. When using Docker, mount `./data:/app/data` to persist the database across container restarts.
***
## Execute a Pipeline
`POST /v1/pipelines/{pipeline_id}/execute`
Runs the pipeline steps sequentially on the provided text. Each `chunk` step produces chunks; each `refine` step enriches them. Returns the final list of chunks.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000/execute \
-H "Content-Type: application/json" \
-d '{"text": "Your document text goes here. It will be chunked and refined."}'
```
**Response:**
```json theme={"system"}
[
{
"id": "chnk_abc123",
"text": "Your document text goes here.",
"start_index": 0,
"end_index": 29,
"token_count": 29,
"context": null,
"embedding": null
},
{
"id": "chnk_def456",
"text": "It will be chunked and refined.",
"start_index": 30,
"end_index": 61,
"token_count": 31,
"context": null,
"embedding": null
}
]
```
### Batch Execution
Submit a list of strings to process multiple documents in one request. The response is a list of lists — one inner list per input document.
```bash theme={"system"}
curl -X POST http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000/execute \
-H "Content-Type: application/json" \
-d '{"text": ["First document.", "Second document.", "Third document."]}'
```
Text or list of texts to process through the pipeline.
### Error Responses
| Status | Cause |
| ------ | ------------------------------------------------------------------------------------------------------------ |
| `404` | Pipeline ID not found |
| `400` | Pipeline has no steps, a `refine` step appears before any `chunk` step, or a step is missing required fields |
| `500` | A step failed at runtime (e.g. missing extra, model error) |
# Quick Start
Source: https://docs.chonkie.ai/oss/api/quickstart
Get the Chonkie API server running in under a minute
## 1. Install
```bash theme={"system"}
pip install "chonkie[api,semantic,code,openai]"
```
The `api` extra includes FastAPI and uvicorn. Add `semantic` for the semantic chunker and `code` for the code chunker. The embeddings refinery works out of the box with local models (e.g. `minishlab/potion-base-8M`); add the relevant extra (e.g. `openai`) only if you plan to use API-based embedding providers.
## 2. Start the Server
```bash Default theme={"system"}
chonkie serve
# 🦛 Starting Chonkie API server on http://0.0.0.0:8000
# 📚 API docs available at http://0.0.0.0:8000/docs
# 🔍 Log level: info
#
# Press CTRL+C to stop the server
```
```bash Custom Port theme={"system"}
chonkie serve --port 3000 --reload
```
```bash Debug Logging theme={"system"}
chonkie serve --log-level debug
```
```bash Direct Uvicorn theme={"system"}
uvicorn chonkie.api.main:app --host 0.0.0.0 --port 8000
```
Visit `http://localhost:8000/docs` for the interactive Swagger UI, or `http://localhost:8000/redoc` for ReDoc.
## 3. Make Your First Request
```bash curl theme={"system"}
curl -X POST http://localhost:8000/v1/chunk/token \
-H "Content-Type: application/json" \
-d '{
"text": "Chonkie makes chunking easy. It splits text into manageable pieces for RAG pipelines.",
"chunk_size": 20
}'
```
```python Python theme={"system"}
import requests
response = requests.post(
"http://localhost:8000/v1/chunk/token",
json={
"text": "Chonkie makes chunking easy. It splits text into manageable pieces for RAG pipelines.",
"chunk_size": 20,
},
)
chunks = response.json()
for chunk in chunks:
print(chunk["text"])
```
```javascript JavaScript theme={"system"}
const response = await fetch("http://localhost:8000/v1/chunk/token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
text: "Chonkie makes chunking easy. It splits text into manageable pieces for RAG pipelines.",
chunk_size: 20,
}),
});
const chunks = await response.json();
chunks.forEach(chunk => console.log(chunk.text));
```
**Response:**
```json theme={"system"}
[
{
"text": "Chonkie makes chunking easy.",
"start_index": 0,
"end_index": 28,
"token_count": 5
},
{
"text": "It splits text into manageable pieces for RAG pipelines.",
"start_index": 29,
"end_index": 85,
"token_count": 10
}
]
```
## Or Use Docker
```bash theme={"system"}
docker compose up
```
The server starts on port `8000`. See the [Docker guide](/oss/api/docker) for the full `docker-compose.yml` and production setup.
## Server Options
| Flag | Default | Description |
| ------------- | --------- | -------------------------------------------------- |
| `--host` | `0.0.0.0` | Bind address |
| `--port` | `8000` | Port number |
| `--reload` | `false` | Auto-reload on file changes (development only) |
| `--log-level` | `info` | Log verbosity: `debug`, `info`, `warning`, `error` |
## Next Steps
Token, Sentence, Recursive, Semantic, Code, and refineries
Save and execute reusable chunking workflows
Production-ready Docker setup with env vars
# Changelog
Source: https://docs.chonkie.ai/oss/changelog
Chonkie's Release Notes and Updates 🦛✨
# v1.5.4 Release Highlights ✨
* **New `GroqGenie`**: Fast inference on Groq hardware! Use Llama models with blazing speed via Groq's infrastructure.
```bash theme={"system"}
pip install "chonkie[groq]"
```
```python theme={"system"}
from chonkie import GroqGenie
genie = GroqGenie(model="llama-3.3-70b-versatile")
response = genie.generate("Hello!")
```
* **New `CerebrasGenie`**: Fastest inference on Cerebras hardware! Experience ultra-fast LLM inference.
```bash theme={"system"}
pip install "chonkie[cerebras]"
```
```python theme={"system"}
from chonkie import CerebrasGenie
genie = CerebrasGenie(model="llama-3.3-70b")
response = genie.generate("Hello!")
```
Both new Genies support `generate()` for text generation and `generate_json()` for structured JSON output, following the same interface as existing Genies.
**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.5.3...v1.5.4](https://github.com/chonkie-inc/chonkie/compare/v1.5.3...v1.5.4)
# v1.3.0 Release Highlights ✨
## Breaking Changes
* **Unified Chunk Type**: All chunkers now return the base `Chunk` type instead of specialized types. The specialized chunk types (`SentenceChunk`, `RecursiveChunk`, `SemanticChunk`, `CodeChunk`, and `LateChunk`) have been removed entirely. This simplifies the API and improves interoperability between different chunkers and refineries.
* **Unified Sentence Type**: The `SemanticSentence` type has been removed. The base `Sentence` type now includes an optional `embedding` attribute, providing the same functionality with a simpler API.
* **New `embedding` Attribute**: Both the base `Chunk` and `Sentence` types now include an optional `embedding` attribute that can store embedding vectors (as lists or numpy arrays). This is automatically populated by `EmbeddingsRefinery` and certain chunkers like `LateChunker`.
## Migration Guide
If you were relying on specialized chunk attributes:
* `SentenceChunk.sentences` → No longer available in base Chunk
* `SemanticChunk.sentences` → No longer available in base Chunk
* `CodeChunk.nodes` → No longer available in base Chunk
* `RecursiveChunk.level` → No longer available in base Chunk
* `LateChunk` → Use base `Chunk` (embedding is now part of base type)
* `SemanticSentence` → Use base `Sentence` (embedding is now part of base type)
All chunkers now consistently return `Chunk` objects with:
```python theme={"system"}
@dataclass
class Chunk:
text: str
start_index: int
end_index: int
token_count: int
context: Optional[Context] = None
embedding: Union[list[float], "np.ndarray", None] = None # NEW!
```
## Import Changes
When importing the Chunk type, use:
```python theme={"system"}
from chonkie.types import Chunk
```
The specialized types are deprecated but remain available for backward compatibility in the legacy module.
# v1.0.6 Release Highlights ✨
* **New `SlumberChunker`**: Welcome Chonkie's very own agentic chunker! Requires the `genie` optional install and a `GEMINI_API_KEY`. It leverages `Genie`, Chonkie's interface for generative models.
```bash theme={"system"}
pip install "chonkie[genie]"
```
```python theme={"system"}
# Import
from chonkie import SlumberChunker
# Initialize
chunker = SlumberChunker(verbose=True) # set verbose to True, since it takes a while~
# CHONK!
chunker(text)
```
* **New `NeuralChunker`**: Introducing a fully neural approach to chunking! Requires the `neural` optional install. This uses a fine-tuned BERT-like model for fast, high-quality chunking.
```bash theme={"system"}
pip install "chonkie[neural]"
```
```python theme={"system"}
# import
from chonkie import NeuralChunker
# initialize
chunker = NeuralChunker()
# CHONK!
chunks = chunker(text)
```
* **`auto` Language Detection for `CodeChunker`**: `CodeChunker` can now automatically detect the programming language. Specify the language manually if performance is critical.
```python theme={"system"}
# Import
from chonkie import CodeChunker
# Initialize the "auto" CodeChunker
chunker = CodeChunker() # No need to specify, "auto" by default
# CHONK!
chunks = chunker(code)
```
* **Introducing `Genie`s**: Added `Genie` to power `SlumberChunker` and future generative features. `Genie`s are Chonkie's way to handle multiple generative APIs and model interfaces. The first is `GeminiGenie`, requiring the `genie` optional install.
```bash theme={"system"}
pip install "chonkie[genie]"
```
```python theme={"system"}
# Import
from chonkie import GeminiGenie
# Init
genie = GeminiGenie(api_key=YOUR_API_KEY)
# generate
genie.generate("Hi!")
# generate JSON
genie.generate_json("Hi", JSON_SCHEMA)
```
**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6](https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6)
# v1.0.5 Release Highlights ✨
This is a quick patch release to include `CodeChunker` in the `__init__.py` for `chonkie` so it can be properly accessed via `from chonkie import CodeChunker`.
**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.4...v1.0.5](https://github.com/chonkie-inc/chonkie/compare/v1.0.4...v1.0.5)
# v1.0.4 Release Highlights ✨
* **New `CodeChunker`**: Introducing the `CodeChunker`, specialized for handling code files across 100+ programming languages. It understands code structure to provide more meaningful chunks.
```bash theme={"system"}
pip install "chonkie[code]"
```
```python theme={"system"}
# Initialize the code chunker
chunker = CodeChunker(language="python")
# Chunk the code
code = ... # Your code string
# CHONK!
chunks = chunker(code)
```
* **`JinaAI` Embeddings Support**: Added `JinaEmbeddings`, enabling their use with `SemanticChunker` and `SDPMChunker`. Just install the `jina` optional install to use it!
```bash theme={"system"}
pip install "chonkie[jina]"
```
```python theme={"system"}
# Initialize the Jina embeddings
from chonkie import JinaEmbeddings, SemanticChunker
# Initialize the Jina embeddings
embeddings = JinaEmbeddings()
# Initialize the semantic chunker
chunker = SemanticChunker(embeddings)
# Chunk the text
text = ... # Your text string
# CHONK!
chunks = chunker(text)
```
* **`OverlapRefinery`**: Enhance your chunks by adding overlapping context using the new `OverlapRefinery`. It's included in the default install and works seamlessly with any chunker.
```python theme={"system"}
# Initialize the recursive chunker
from chonkie import RecursiveChunker, OverlapRefinery
chunker = RecursiveChunker()
# Initialize the overlap refinery
refinery = OverlapRefinery() # Or OverlapRefinery("gpt2")
# Chunk the text
text = ... # Your text string
# CHONK!
chunks = chunker(text)
# Refine the chunks
chunks = refinery(chunks)
```
* **`EmbeddingsRefinery`**: Compute and attach embeddings directly to your chunks using the `EmbeddingsRefinery`. Streamline the process of loading chunks into vector databases.
```python theme={"system"}
from chonkie import RecursiveChunker, EmbeddingsRefinery, JinaEmbeddings
# Initialize the recursive chunker
chunker = RecursiveChunker()
# Initialize the embeddings model
# Here we use Jina embeddings for this example, but you can use any other embeddings model
embeddings = JinaEmbeddings()
# Initialize the embeddings refinery
refinery = EmbeddingsRefinery(embeddings)
# Chunk the text
text = ... # Your text string
chunks = chunker(text)
chunks = refinery(chunks) # Each chunk now has a .embedding attribute
```
**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4](https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4)
# v1.0.3 Release Highlights ✨
* **Chonkie `Visualizer`**: Visualize and debug chunks easily via terminal printouts or HTML saves. Understand chunk quality and debug your chunker with visual feedback\~ Use the `print` method to print rich text on your terminal or use the `save` method to save a highlighted `html` on your device! It's very simple to use, just pass in your chunks\~
```python theme={"system"}
from chonkie import Visualizer
viz = Visualizer()
# Print the chunks on the terminal with .print or directly call the Visualizer object too
viz.print(chunks)
# Save the HTML file
viz.save("chonkie.html", chunks)
```
* **Recipes**: Chonkie now adds support for `Recipes` which allow you to use multilingual chunking out-of-the-box, as well as document specific chunking methods. Initial support starts with: `en`, `hi`, `zh`, `jp` and `ko`, while document type `markdown` is supported too. Use it via the `from_recipe` class method with any chunker that takes delimiters or `RecursiveRules`.
```python theme={"system"}
from chonkie import RecursiveChunker
# Initialize the recursive chunker to chunk Markdown
chunker = RecursiveChunker.from_recipe("markdown", lang="en")
# Initialize the recursive chunker to chunk Hindi texts
chunker = RecursiveChunker.from_recipe(lang="hi")
```
* Performance enhancements in `RecursiveChunker`, `SentenceChunker`, and `WordTokenizer`.
**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3](https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3)
# LiteParse
Source: https://docs.chonkie.ai/oss/chefs/liteparse
Extract text from PDFs, office documents, and images locally using LiteParse.
`LiteParse` extracts text from PDFs, office documents, and images using [LiteParse](https://github.com/run-llama/liteparse) from LlamaIndex. It runs entirely locally with no cloud API dependencies.
## Installation
```bash theme={"system"}
pip install chonkie[liteparse]
```
LiteParse runs locally. OCR uses bundled Tesseract by default. Office document conversion requires LibreOffice, and image conversion requires ImageMagick.
## Initialization
```python theme={"system"}
from chonkie import LiteParse
# Default initialization
chef = LiteParse()
# Custom configuration
chef = LiteParse(
ocr_enabled=True,
ocr_language="eng",
dpi=300,
max_pages=100,
target_pages="1-10",
num_workers=8,
)
```
### Parameters
Whether to enable OCR for scanned/image text (defaults to LiteParse's behavior when `None`).
Language code for OCR (e.g., `"eng"`, `"fra"`, `"deu"`).
Optional HTTP OCR server URL (e.g., EasyOCR or PaddleOCR server).
Maximum number of pages to parse.
Specific pages to parse (e.g., `"1-5,10"`).
Rendering resolution for PDF pages.
Number of pages to OCR in parallel (defaults to CPU cores - 1).
Password for protected PDFs.
## Methods
### process()
Process a file and return a `Document`.
#### Parameters
Path to the file to process.
#### Returns
`Document` containing the extracted text content.
### process\_batch()
Process multiple files at once.
#### Parameters
List of file paths to process.
#### Returns
`list[Document]` where each document contains extracted text from a file.
### parse()
Parse raw text into a `Document` (wraps text as-is, since LiteParse operates on files).
#### Parameters
Raw text to wrap into a Document.
#### Returns
`Document` containing the provided text.
## Supported File Types
| Type | Extensions |
| ------------ | ------------------------------------------------------------------------- |
| PDF | `.pdf` |
| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` |
| PowerPoint | `.ppt`, `.pptx`, `.pptm`, `.odp` |
| Spreadsheets | `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv` |
| Images | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tiff`, `.tif`, `.webp`, `.svg` |
## Usage
### Standalone
```python theme={"system"}
from chonkie import LiteParse
chef = LiteParse()
# Single file
doc = chef.process("research_paper.pdf")
print(doc.content)
print(f"Source: {doc.metadata['filename']}")
# Multiple files
docs = chef.process_batch(["report.pdf", "slides.pptx", "data.xlsx"])
# Async
import asyncio
doc = asyncio.run(chef.aprocess("document.pdf"))
```
### Pipeline
Use `.process_with("liteparse")` to add local document parsing to a pipeline:
```python theme={"system"}
from chonkie import Pipeline
# Process a PDF locally and chunk it
doc = (Pipeline()
.fetch_from("file", path="document.pdf")
.process_with("liteparse")
.chunk_with("recursive", chunk_size=512)
.run())
print(f"Extracted {len(doc.chunks)} chunks from PDF")
```
### Local RAG Pipeline
Build a complete pipeline from documents to vector database without any cloud OCR:
```python theme={"system"}
from chonkie import Pipeline
docs = (Pipeline()
.fetch_from("file", dir="./documents", ext=[".pdf", ".docx", ".pptx"])
.process_with("liteparse")
.chunk_with("recursive", chunk_size=1024)
.refine_with("overlap", context_size=100)
.store_in("qdrant", collection_name="local_documents")
.run())
print(f"Ingested {len(docs)} documents")
```
### Targeted Page Extraction
Parse only specific pages from a large PDF:
```python theme={"system"}
from chonkie import LiteParse
chef = LiteParse(target_pages="1-5,10,15-20", dpi=300)
doc = chef.process("large_report.pdf")
print(f"Extracted {len(doc.content)} characters from selected pages")
```
## Integration with Chunkers
LiteParse returns a `Document`, making it compatible with any chunker:
```python theme={"system"}
from chonkie import LiteParse, RecursiveChunker
# Step 1: Extract text from PDF
chef = LiteParse()
doc = chef.process("report.pdf")
# Step 2: Chunk the extracted content
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker.chunk(doc.content)
# Step 3: Store chunks in the document
doc.chunks = chunks
print(f"Document: {doc.metadata['filename']}")
print(f" Content: {len(doc.content)} characters")
print(f" Chunks: {len(doc.chunks)}")
```
## Notes
* Runs entirely locally with no API keys or cloud dependencies
* OCR quality depends on image resolution and the Tesseract language pack
* Office documents (Word, PowerPoint, Excel) require LibreOffice to be installed
* Image files require ImageMagick to be installed
* Use `num_workers` to control parallelism for multi-page OCR
* Use `target_pages` for efficient extraction from large PDFs
* API calls are synchronous by default; use `aprocess()` for async execution
# MarkdownChef
Source: https://docs.chonkie.ai/oss/chefs/markdownchef
Process markdown files, extracting tables, code blocks, and images.
The `MarkdownChef` processes markdown files and strings, extracting tables, code blocks, and images into a structured `MarkdownDocument`.
It intelligently parses markdown content and separates it into distinct components while preserving their positions in the original text.
## Installation
MarkdownChef is included in the base installation of Chonkie. No additional dependencies are required.
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import MarkdownChef
# Basic initialization with default tokenizer
chef = MarkdownChef()
# Initialize with a specific tokenizer
chef = MarkdownChef(tokenizer="gpt2")
# Or use a custom tokenizer instance
from transformers import AutoTokenizer
custom_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
chef = MarkdownChef(tokenizer=custom_tokenizer)
```
## Parameters
Tokenizer to use for counting tokens in text chunks. Can be a string
identifier ("character", "gpt2", etc.) or a tokenizer instance that follows
the TokenizerProtocol.
## Methods
### process()
Process a markdown file.
#### Parameters
Path to the markdown file (string or Path object)
#### Returns
`MarkdownDocument` containing parsed content with extracted tables, code, images, and text chunks
### process\_batch()
Process multiple markdown files at once.
#### Parameters
List of file paths to process
#### Returns
List of `MarkdownDocument` objects
## Basic Usage
```python theme={"system"}
from chonkie import MarkdownChef
# Initialize the chef
chef = MarkdownChef()
# Process a markdown file
doc = chef.process("example.md")
# Access the extracted components
print(f"Found {len(doc.tables)} tables")
print(f"Found {len(doc.code)} code blocks")
print(f"Found {len(doc.images)} images")
print(f"Found {len(doc.chunks)} text chunks")
```
## Return Type
MarkdownChef returns a `MarkdownDocument` object, which extends the base `Document` class with additional fields:
```python theme={"system"}
@dataclass
class MarkdownTable:
content: str # The table content
start_index: int # Starting position in original text
end_index: int # Ending position in original text
@dataclass
class MarkdownCode:
content: str # The code content
language: Optional[str] # Programming language (if specified)
start_index: int # Starting position in original text
end_index: int # Ending position in original text
@dataclass
class MarkdownImage:
alias: str # Alt text or filename
content: str # Image path or data URL
start_index: int # Starting position in original text
end_index: int # Ending position in original text
link: Optional[str] # Link URL (if image is clickable)
@dataclass
class MarkdownDocument(Document):
id: str # Unique document ID
content: str # Full markdown content
tables: list[MarkdownTable] # Extracted tables
code: list[MarkdownCode] # Extracted code blocks
images: list[MarkdownImage] # Extracted images
chunks: list[Chunk] # Remaining text chunks
metadata: dict[str, Any] # Additional metadata
```
# MistralOCR
Source: https://docs.chonkie.ai/oss/chefs/mistral-ocr
Extract text from images and PDFs using Mistral's OCR API.
The `MistralOCR` chef extracts text from images and PDF files using Mistral's OCR API, returning structured `MarkdownDocument` objects for further processing.
## Installation
```bash theme={"system"}
pip install chonkie[mistral]
```
You need a Mistral API key. Set the `MISTRAL_API_KEY` environment variable or pass it directly.
## Initialization
```python theme={"system"}
from chonkie import MistralOCR
# Default initialization (uses MISTRAL_API_KEY env var)
ocr = MistralOCR()
# Custom model and explicit API key
ocr = MistralOCR(model="mistral-ocr-2505", api_key="sk-...")
```
### Parameters
The Mistral OCR model to use.
Mistral API key. Falls back to the `MISTRAL_API_KEY` environment variable.
## Methods
### process()
Process an image or PDF file and return a `MarkdownDocument`.
#### Parameters
Path to the image or PDF file.
#### Returns
`MarkdownDocument` containing the extracted text as markdown content.
### process\_batch()
Process multiple image or PDF files at once.
#### Parameters
List of file paths to process.
#### Returns
`list[MarkdownDocument]` where each document contains extracted text from a file.
### parse()
Parse raw text into a `Document` (wraps text as-is, since OCR operates on files).
#### Parameters
Raw text to wrap into a Document.
#### Returns
`Document` containing the provided text.
## Supported File Types
| Type | Extensions |
| --------- | ----------------------------------------------------------------- |
| Images | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.webp`, `.tiff`, `.tif` |
| Documents | `.pdf` |
## Usage
### Standalone
```python theme={"system"}
from chonkie import MistralOCR
ocr = MistralOCR()
# Single file
doc = ocr.process("research_paper.pdf")
print(doc.content)
print(f"Source: {doc.metadata['filename']}")
# Multiple files
docs = ocr.process_batch(["page1.png", "page2.png"])
# Async
import asyncio
doc = asyncio.run(ocr.aprocess("document.pdf"))
```
### Pipeline
Use `.process_with("mistral")` to add OCR to a pipeline:
```python theme={"system"}
from chonkie import Pipeline
# Process a PDF with OCR and chunk it
doc = (Pipeline()
.fetch_from("file", path="document.pdf")
.process_with("mistral")
.chunk_with("recursive", chunk_size=512)
.run())
print(f"Extracted {len(doc.chunks)} chunks from PDF")
```
### OCR + RAG Pipeline
Build a complete pipeline from scanned documents to vector database:
```python theme={"system"}
from chonkie import Pipeline
docs = (Pipeline()
.fetch_from("file", dir="./scanned_docs", ext=[".pdf", ".png"])
.process_with("mistral")
.chunk_with("recursive", chunk_size=1024)
.refine_with("overlap", context_size=100)
.store_in("qdrant", collection_name="scanned_documents")
.run())
print(f"Ingested {len(docs)} documents")
```
### OCR + Semantic Chunking
Use semantic chunking on OCR output for intelligent retrieval boundaries:
```python theme={"system"}
from chonkie import Pipeline
doc = (Pipeline()
.fetch_from("file", path="textbook_chapter.pdf")
.process_with("mistral")
.chunk_with("semantic", threshold=0.8, chunk_size=1024)
.refine_with("embedding", model="text-embedding-3-small")
.export_with("json", file="textbook_chunks.json")
.run())
```
## Integration with Chunkers
MistralOCR returns a `MarkdownDocument`, making it compatible with any chunker:
```python theme={"system"}
from chonkie import MistralOCR, RecursiveChunker
# Step 1: Extract text from PDF
ocr = MistralOCR()
doc = ocr.process("report.pdf")
# Step 2: Chunk the extracted content
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker.chunk(doc.content)
# Step 3: Store chunks in the document
doc.chunks = chunks
print(f"Document: {doc.metadata['filename']}")
print(f" Content: {len(doc.content)} characters")
print(f" Chunks: {len(doc.chunks)}")
```
## Notes
* OCR quality depends on image resolution and clarity
* Large PDFs are processed page-by-page and concatenated with double newlines
* The extracted text is returned as markdown, preserving structure from the source document
* API calls are synchronous by default; use `aprocess()` for async execution
# Chefs Overview
Source: https://docs.chonkie.ai/oss/chefs/overview
Overview of the different chefs available in Chonkie
Chefs are simple classes that automatically prepare data for future usage. They are designed to make preprocessing and data transformation easy and reusable.
Chefs are available only in Python
Extracts tables from markdown text and prepares them for future usage.
Processes plain text files and returns structured Document objects.
Processes markdown files, extracting tables, code blocks, and images into a
MarkdownDocument.
Extracts text from images and PDFs using Mistral's OCR API.
Extracts text from PDFs, office documents, and images locally using LiteParse.
# TableChef
Source: https://docs.chonkie.ai/oss/chefs/tablechef
Extract tables from markdown text (including HTML tables) and prepare them for future usage.
The `TableChef` is a versatile chef that extracts and processes tables from multiple sources. It can read CSV and Excel files, convert them to markdown format, or extract tables from markdown text. The parsed tables are returned in a structured format ready for downstream processing.
## Installation
TableChef requires the `pandas` library for processing CSV and Excel files.
```bash theme={"system"}
pip install "chonkie[table]"
```
For more installation options, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import TableChef
chef = TableChef()
```
## Methods
### process()
Process a file or markdown string to extract tables.
#### Parameters
Can be a file path (CSV/Excel) or a markdown string containing tables
#### Returns
A list of `MarkdownTable` objects. `None` if no tables are found
### process\_batch()
Process multiple files or markdown strings at once.
#### Parameters
List of file paths or markdown strings to process
#### Returns
A list of `MarkdownTable` objects. `None` if no tables are found
## Usage
```python Files (CSV/Excel) theme={"system"}
from chonkie import TableChef
# Initialize the chef
chef = TableChef()
# Process a CSV file
doc = chef.process("data.csv")
print(f"Extracted {len(doc.tables)} table from CSV")
# Process an Excel file (all sheets)
doc = chef.process("spreadsheet.xlsx")
print(f"Extracted {len(doc.tables)} tables from Excel")
```
```python Markdown Tables theme={"system"}
from chonkie import TableChef
chef = TableChef()
markdown = """
# Data Analysis
| Name | Age | City |
|------|-----|------|
| Alice | 25 | New York |
| Bob | 30 | Paris |
Some text between tables...
| Product | Price |
|---------|-------|
| Apple | $1.50 |
| Banana | $0.75 |
"""
# Extract all markdown tables
doc = chef.process(markdown)
for i, table in enumerate(doc.tables):
print(f"Table {i+1} content:\n{table.content}\n")
```
```python HTML Tables theme={"system"}
from chonkie import TableChef
chef = TableChef()
html_content = """
Employee List
ID
Status
1
Active
2
Pending
3
Inactive
4
Active
"""
# Extract all html tables
doc = chef.process(html_content)
for i, table in enumerate(doc.tables):
print(f"Table {i+1} content:\n{table.content}\n")
```
## Supported File Formats
* **CSV files** (`.csv`) - Comma-separated values
* **Excel files** (`.xls`, `.xlsx`) - Microsoft Excel spreadsheets
* **Markdown strings** - Text containing pipe-separated tables or HTML tables (`
`)
# TextChef
Source: https://docs.chonkie.ai/oss/chefs/textchef
Process plain text files into Document objects.
The `TextChef` processes plain text files and returns structured `Document` objects for further processing.
## Installation
TextChef is included in the base installation of Chonkie. No additional dependencies are required.
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import TextChef
# Simple initialization - no parameters required
chef = TextChef()
```
## Methods
### process()
Process a text file and return a `Document` object.
#### Parameters
Path to the text file (string or Path object)
#### Returns
`Document` object containing the file content
### process\_batch()
Process multiple text files at once.
#### Parameters
List of file paths to process
#### Returns
`list[Document]` where each `Document` contains a file's contents.
## Usage
```python theme={"system"}
from chonkie import TextChef
# Initialize the chef
chef = TextChef()
# Process a text file
doc = chef.process("example.txt")
# Access the content
print(doc.content)
print(f"Document ID: {doc.id}")
```
## Integration with Chunkers
TextChef is designed to work seamlessly with Chonkie's chunkers:
```python theme={"system"}
from chonkie import TextChef, TokenChunker
# Step 1: Load text file
chef = TextChef()
doc = chef.process("article.txt")
# Step 2: Chunk the content
chunker = TokenChunker(chunk_size=512, chunk_overlap=50)
chunks = chunker.chunk(doc.content)
# Step 3: Store chunks back in the document
doc.chunks = chunks
# Now your document has both content and chunks
print(f"Document {doc.id}:")
print(f" Content: {len(doc.content)} characters")
print(f" Chunks: {len(doc.chunks)}")
```
## Encoding
TextChef reads files with UTF-8 encoding by default, ensuring proper handling of:
* Unicode characters
* International text
* Special symbols
* Emoji and other non-ASCII characters
All text is read as strings and preserved exactly as it appears in the source file.
# Code Chunker
Source: https://docs.chonkie.ai/oss/chunkers/code-chunker
Split code into chunks based on code structure
The `CodeChunker` splits code into chunks based on its structure, leveraging Abstract Syntax Trees (ASTs) to create contextually relevant segments.
## Overview
* Supports 165+ languages
* Powered by [tree-sitter-language-pack](https://github.com/Goldziher/tree-sitter-language-pack)
* Auto language detection via [Magika](https://github.com/google/magika), a language detection library made by Google
## Supported Languages
Each language is identified by the **key** used with `get_language(key)` and `get_parser(key)`.
### General-Purpose Programming Languages
| Language | Key | License |
| ------------ | ----------------------- | ------------ |
| ActionScript | `actionscript` | MIT |
| Ada | `ada` | MIT |
| Agda | `agda` | MIT |
| C | `c` | MIT |
| C++ | `cpp` | MIT |
| C# | `csharp` | MIT |
| Dart | `dart` | MIT |
| Go | `go` | MIT |
| Java | `java` | MIT |
| JavaScript | `javascript` | MIT |
| Julia | `julia` | MIT |
| Kotlin | `kotlin` | MIT |
| Nim | `nim` | MPL-2.0 |
| OCaml | `ocaml/ocaml_interface` | MIT |
| Perl | `perl` | Artistic-2.0 |
| Python | `python` | MIT |
| R | `r` | MIT |
| Ruby | `ruby` | MIT |
| Rust | `rust` | MIT |
| Scala | `scala` | MIT |
| Swift | `swift` | MIT |
| TypeScript | `typescript` | MIT |
| Zig | `zig` | MIT |
### Web, UI & Markup
| Language | Key | License |
| --------------- | ----------------- | ------- |
| HTML | `html` | MIT |
| CSS | `css` | MIT |
| SCSS | `scss` | MIT |
| Astro | `astro` | MIT |
| Vue | `vue` | MIT |
| Svelte | `svelte` | MIT |
| TSX | `tsx` | MIT |
| Markdown | `markdown` | MIT |
| Markdown Inline | `markdown_inline` | MIT |
| Mermaid | `mermaid` | MIT |
| XML | `xml` | MIT |
| YAML | `yaml` | MIT |
### Config, Build & DevOps
| Language | Key | License |
| ------------ | -------------- | ------- |
| Bash | `bash` | MIT |
| Dockerfile | `dockerfile` | MIT |
| Git Ignore | `gitignore` | MIT |
| Git Commit | `gitcommit` | WTFPL |
| Make | `make` | MIT |
| Ninja | `ninja` | MIT |
| Meson | `meson` | MIT |
| Prisma | `prisma` | MIT |
| Requirements | `requirements` | MIT |
### Systems, GPU & Low-level
| Language | Key | License |
| ---------- | --------- | ----------------- |
| ASM | `asm` | MIT |
| CUDA | `cuda` | MIT |
| GLSL | `glsl` | MIT |
| HLSL | `hlsl` | MIT |
| LLVM | `llvm` | MIT |
| Verilog | `verilog` | MIT |
| VHDL | `vhdl` | MIT |
| WGSL | `wgsl` | MIT |
| WAST / WAT | `wasm` | Apache-2.0 + LLVM |
## API Reference
To use the `CodeChunker` via the API, check out the [API reference documentation](../../api/chunkers/code-chunker).
## Installation
CodeChunker requires additional dependencies for code parsing. You can install it with:
```bash theme={"system"}
pip install "chonkie[code]"
```
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python Basic initialization theme={"system"}
from chonkie import CodeChunker
chunker = CodeChunker(
language="python", # Specify the programming language
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
chunk_size=2048, # Maximum tokens per chunk
include_nodes=False # Optionally include AST nodes in output
)
```
```python Auto theme={"system"}
from chonkie import CodeChunker
chunker = CodeChunker(
language="auto", # Auto detects programming language via Magika
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
chunk_size=2048, # Maximum tokens per chunk
include_nodes=False # Optionally include AST nodes in output
)
```
```python Custom tokenizer theme={"system"}
from chonkie import CodeChunker
from tokenizers import Tokenizer
custom_tokenizer = Tokenizer.from_pretrained("your-tokenizer")
chunker = CodeChunker(
language="javascript",
tokenizer=custom_tokenizer,
chunk_size=2048
)
```
Automatic language detection using Magika can impact performance. For best results, specify the language directly via the `language` parameter.
## Parameters
The programming language of the code. Accepts languages supported by
`tree-sitter-language-pack`.
Tokenizer or token counting function to use for measuring chunk size.
Maximum number of tokens per chunk.
Whether to include AST node information (Note: with the base Chunk type, node
information is not stored).
## Usage
### Single Code Chunking
```python theme={"system"}
code = """
def hello_world():
print("Hello, Chonkie!")
class MyClass:
def __init__(self):
self.value = 42
"""
chunks = chunker.chunk(code)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
```
```javascript theme={"system"}
const code = `
def hello_world():
print("Hello, Chonkie!")
class MyClass:
def __init__(self):
self.value = 42
`;
const chunks = await chunker.chunk(code);
for (const chunk of chunks) {
console.log(`Chunk text: ${chunk.text}`);
console.log(`Token count: ${chunk.tokenCount}`);
}
```
### Batch Chunking
```python theme={"system"}
codes = [
"def func1():\n pass",
"const x = 10;\nfunction add(a, b) { return a + b; }"
]
batch_chunks = chunker.chunk_batch(codes)
for doc_chunks in batch_chunks:
for chunk in doc_chunks:
print(f"Chunk: {chunk.text}")
```
```javascript theme={"system"}
const codes = [
"def func1():\n pass",
"const x = 10;\nfunction add(a, b) { return a + b; }"
];
const batchChunks = await chunker.chunkBatch(codes);
for (const docChunks of batchChunks) {
for (const chunk of docChunks) {
console.log(`Chunk: ${chunk.text}`);
}
}
```
### Using as a Callable
```python theme={"system"}
# Single code string
chunks = chunker("def greet(name):\n print(f'Hello, {name}')")
# Multiple code strings
batch_chunks = chunker(["int main() { return 0; }", "package main\nimport \"fmt\"\nfunc main() { fmt.Println(\"Hi\") }"])
```
## Return Type
CodeChunker returns chunks as `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text (code snippet)
start_index: int # Starting position in original code
end_index: int # Ending position in original code
token_count: int # Number of tokens in chunk
context: Optional[Context] = None # Optional context metadata
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
As of version 1.3.0, CodeChunker returns the base `Chunk` type instead of the
specialized `CodeChunk` type. This simplifies integration with other chunkers
and refineries.
# Fast Chunker
Source: https://docs.chonkie.ai/oss/chunkers/fast-chunker
SIMD-accelerated text chunking at 100+ GB/s throughput
The `FastChunker` uses [chonkie-core](https://github.com/chonkie-inc/chunk) for SIMD-accelerated boundary detection, enabling chunking speeds of 100+ GB/s.
Unlike other chunkers, FastChunker uses **byte size** limits instead of token counts.
This tradeoff enables extreme performance for high-throughput pipelines.
## Initialization
```python Basic initialization with default parameters theme={"system"}
from chonkie import FastChunker
chunker = FastChunker(
chunk_size=4096, # Target size in BYTES (not tokens)
delimiters="\n.?", # Split at newlines, periods, question marks
)
```
```python Split at paragraph boundaries theme={"system"}
chunker = FastChunker(
chunk_size=8192,
delimiters="\n\n",
)
```
```python Pattern-based splitting (e.g., for SentencePiece tokenizers) theme={"system"}
chunker = FastChunker(
chunk_size=4096,
pattern="▁", # Metaspace character
prefix=True, # Keep pattern at start of next chunk
)
```
```javascript Basic initialization with default parameters theme={"system"}
import { FastChunker } from "@chonkiejs/core";
let chunker = await FastChunker.create({
chunkSize: 4096, // Target size in BYTES (not tokens)
delimiters: "\n.?", // Split at newlines, periods, question marks
});
```
```javascript Split at paragraph boundaries theme={"system"}
chunker = await FastChunker.create({
chunkSize: 8192,
delimiters: "\n\n",
});
```
```javascript Pattern-based splitting (e.g., for SentencePiece tokenizers) theme={"system"}
chunker = await FastChunker.create({
chunkSize: 4096,
pattern: "▁", // Metaspace character
prefix: true, // Keep pattern at start of next chunk
});
```
## Parameters
Target chunk size in **bytes** (not tokens)
Single-byte delimiter characters to split on
Multi-byte pattern to split on (overrides delimiters if set)
If True, keep the delimiter/pattern at the start of the next chunk instead of the end of the current chunk
If True, split at the START of consecutive delimiter runs instead of the middle
If True, search forward for a delimiter when none is found in the backward search window
## Basic Usage
```python theme={"system"}
from chonkie import FastChunker
# Initialize the chunker
chunker = FastChunker(
chunk_size=1024,
delimiters=". \n",
)
# Chunk your text
text = "Your long document text here..."
chunks = chunker.chunk(text)
# Access chunk information
for chunk in chunks:
print(f"Chunk: {chunk.text[:50]}...")
print(f"Bytes: {len(chunk.text)}")
print(f"Position: {chunk.start_index}-{chunk.end_index}")
```
```javascript theme={"system"}
import { FastChunker } from "@chonkiejs/core";
// Initialize the chunker
const chunker = await FastChunker.create({
chunkSize: 1024,
delimiters: ". \n",
});
// Chunk your text
const text = "Your long document text here...";
const chunks = await chunker.chunk(text);
// Access chunk information
for (const chunk of chunks) {
console.log(`Chunk: ${chunk.text.slice(0, 50)}...`);
console.log(`Bytes: ${chunk.text.length}`);
console.log(`Position: ${chunk.startIndex}-${chunk.endIndex}`);
}
```
## Examples
```python theme={"system"}
from chonkie import FastChunker
# Split at sentence boundaries
chunker = FastChunker(
chunk_size=70,
delimiters=".!?\n",
)
text = """Machine learning has transformed technology.
It enables computers to learn from data.
Neural networks power many modern applications.
The field continues to evolve rapidly."""
chunks = chunker.chunk(text)
for i, chunk in enumerate(chunks):
print(f"\n--- Chunk {i+1} ---")
print(f"Text: {chunk.text}")
print(f"Bytes: {len(chunk.text)}")
```
```python theme={"system"}
from chonkie import FastChunker
# Split at metaspace boundaries (common in SentencePiece tokenizers)
chunker = FastChunker(
chunk_size=10,
pattern="▁", # Metaspace character
prefix=True, # Keep ▁ at start of next chunk
)
text = "Hello▁World▁this▁is▁a▁test▁sentence"
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk: {chunk.text}")
```
```python theme={"system"}
from chonkie import FastChunker
# Split at START of consecutive whitespace runs
chunker = FastChunker(
chunk_size=10,
pattern=" ",
consecutive=True,
)
text = """First paragraph with multiple sentences.
This is still the first paragraph.
Second paragraph starts here.
More content in the second paragraph.""" # Multiple spaces between words
chunks = chunker.chunk(text)
# Without consecutive=True: might split in middle of " "
# With consecutive=True: splits at START of " "
for chunk in chunks:
print(f"Chunk: '{chunk.text}'")
```
```python theme={"system"}
from chonkie import FastChunker
# Search forward if no delimiter found in backward window
chunker = FastChunker(
chunk_size=10,
pattern=" ",
forward_fallback=True,
)
text = "verylongword short"
chunks = chunker.chunk(text)
# Without forward_fallback: hard split at byte 10
# With forward_fallback: finds space after "verylongword"
for chunk in chunks:
print(f"Chunk: '{chunk.text}'")
```
```python theme={"system"}
from chonkie import FastChunker
chunker = FastChunker(chunk_size=2048)
documents = [
"First document content here...",
"Second document with different content...",
"Third document for processing...",
]
# Process all documents
batch_results = chunker.chunk_batch(documents)
for doc_idx, doc_chunks in enumerate(batch_results):
print(f"\nDocument {doc_idx + 1}: {len(doc_chunks)} chunks")
for chunk in doc_chunks:
print(f" - {chunk.text[:30]}... ({len(chunk.text)} bytes)")
```
```python theme={"system"}
from chonkie import FastChunker
import time
# Configure for maximum throughput
chunker = FastChunker(
chunk_size=8192,
delimiters="\n",
)
# Read a large file
with open("large_file.txt", "r") as f:
large_text = f.read()
# Benchmark chunking speed
start = time.perf_counter()
chunks = chunker.chunk(large_text)
elapsed = time.perf_counter() - start
mb_size = len(large_text) / (1024 * 1024)
throughput = mb_size / elapsed
print(f"Processed {mb_size:.1f} MB in {elapsed*1000:.1f}ms")
print(f"Throughput: {throughput:.1f} MB/s")
print(f"Chunks: {len(chunks)}")
```
## Comparison with Other Chunkers
| Feature | FastChunker | TokenChunker | SentenceChunker |
| ------------------ | ------------------------- | ---------------------- | ------------------- |
| Size unit | Bytes | Tokens | Tokens |
| Tokenizer required | No | Yes | Yes |
| `token_count` | Always 0 | Computed | Computed |
| Speed | \~100+ GB/s | Tokenizer-bound | Tokenizer-bound |
| Best for | High-throughput pipelines | Token-precise chunking | Semantic boundaries |
## When to Use FastChunker
**Use FastChunker when:**
* Processing large volumes of text (>100KB documents)
* Building high-throughput pipelines
* Byte-level precision is acceptable
* You don't need exact token counts
**Use other chunkers when:**
* You need precise token counts for LLM context limits
* Working with small documents (\< 1KB)
* Complex semantic boundaries are required
## Return Type
FastChunker returns chunks as `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting character position in original text
end_index: int # Ending character position in original text
token_count: int # Always 0 (not computed for speed)
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
The `token_count` field is always 0 in FastChunker output.
If you need token counts, use the tokenizer separately or choose a different chunker.
# Late Chunker
Source: https://docs.chonkie.ai/oss/chunkers/late-chunker
Split text into chunks based on a late-bound token count
The **LateChunker** implements the late chunking strategy described in the [Late Chunking](https://arxiv.org/abs/2409.04701) paper. It builds on top of the `RecursiveChunker` and uses document-level embeddings to create more semantically rich chunk representations.
Instead of generating embeddings for each chunk independently, the LateChunker first encodes the entire text into a single embedding.
It then splits the text using recursive rules and derives each chunk’s embedding by averaging relevant parts of the
full document embedding. This allows each chunk to carry broader contextual information,
improving retrieval performance in RAG systems.
## API Reference
To use the `LateChunker` via the API, check out the [API reference documentation](../../api/chunkers/late-chunker).
## Installation
LateChunker requires the `sentence-transformers` library to be installed, and currently only supports SentenceTransformer models.
You can install it with:
The LateChunker uses `RecursiveRules` to determine how to chunk the text.
The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree.
Find more information about the rules in the [Additional Information](#additional-information) section.
```bash theme={"system"}
pip install "chonkie[st]"
```
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import LateChunker
chunker = LateChunker(
embedding_model="nomic-ai/modernbert-embed-base",
chunk_size=2048,
rules=RecursiveRules(),
min_characters_per_chunk=24,
)
```
You can also initialize the LateChunker using a recipe. Recipes are pre-defined rules for common chunking tasks.
Find all available recipes on our Hugging Face Hub [here](https://huggingface.co/datasets/chonkie-ai/recipes).
```python theme={"system"}
from chonkie import LateChunker
# Initialize the late chunker to chunk Markdown
chunker = LateChunker.from_recipe("markdown", lang="en")
# Initialize the late chunker to chunk Hindi texts
chunker = LateChunker.from_recipe(lang="hi")
```
## Parameters
SentenceTransformer model to use for embedding
Maximum number of tokens per chunk
Rules to use for chunking
Minimum number of characters per sentence
## Usage
### Single Text Chunking
```python theme={"system"}
text = """First paragraph about a specific topic.
Second paragraph continuing the same topic.
Third paragraph switching to a different topic.
Fourth paragraph expanding on the new topic."""
chunks = chunker(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
print(f"Embedding shape: {chunk.embedding.shape}")
```
### Batch Chunking
```python theme={"system"}
texts = [
"First document about topic A...",
"Second document about topic B..."
]
batch_chunks = chunker(texts)
for chunk in batch_chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
print(f"Embedding shape: {chunk.embedding.shape}")
```
## Return Type
LateChunker returns chunks as `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[Context] = None # Optional context metadata
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
As of version 1.3.0, LateChunker returns the base `Chunk` type instead of the
specialized `LateChunk` type. The embedding is automatically populated by the
LateChunker during the chunking process.
## Additional Information
LateChunker uses the `RecursiveRules` class to determine the chunking rules.
The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree.
```python theme={"system"}
@dataclass
class RecursiveRules:
rules: list[RecursiveLevel]
@dataclass
class RecursiveLevel:
delimiters: Union[None, str, list[str]]
whitespace: bool = False
include_delim: Optional[Literal["prev", "next"]] # Whether to include the delimiter in the previous chunk or the next chunk.
```
You can pass in custom rules to the LateChunker, or use the default ones.
Default rules are designed to be a good starting point for most documents, but you can customize them to your needs.
`RecursiveLevel` expects the list of custom delimiters to **not** include
whitespace. If whitespace as a delimiter is required, you can set the
`whitespace` parameter in the `RecursiveLevel` class to True. Note that if
`whitespace = True`, you cannot pass a list of custom delimiters.
# Neural Chunker
Source: https://docs.chonkie.ai/oss/chunkers/neural-chunker
Split text using a fine-tuned BERT model to detect semantic shifts
The `NeuralChunker` leverages the power of deep learning! It uses a fine-tuned BERT model specifically trained to identify semantic shifts within text, allowing it to split documents at points where the topic or context changes significantly. This provides highly coherent chunks ideal for RAG.
## API Reference
To use the `NeuralChunker` via the API, check out the [API reference documentation](../../api/chunkers/neural-chunker).
## Installation
NeuralChunker requires specific dependencies for its deep learning model. You can install it with:
```bash theme={"system"}
pip install "chonkie[neural]"
```
For general installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import NeuralChunker
# Basic initialization with default parameters
chunker = NeuralChunker(
model="mirth/chonky_modernbert_base_1", # Default model
device_map="cpu", # Device to run the model on ('cpu', 'cuda', etc.)
min_characters_per_chunk=10, # Minimum characters for a chunk
)
# Specify a different model or device
chunker = NeuralChunker(
model="path/to/your/model",
device_map="cuda:0" # Use GPU if available
)
```
## Parameters
The identifier or path to the fine-tuned BERT model used for detecting
semantic shifts.
The tokenizer to use for the chunker
The device to run the inference on (e.g., "cpu", "cuda", "mps"). Chonkie will
try to auto-detect the best available device if not specified.
The minimum number of characters required for a text segment to be considered
a valid chunk.
Stride to use for the chunker. Will automatically select appropriate stride
for the model if not specified.
## Usage
### Single Text Chunking
```python theme={"system"}
text = """Topic one starts here and continues for a bit.
Suddenly, the context shifts to topic two, which is quite different.
Topic two carries on, discussing various aspects. Then topic one briefly returns.
Finally, we conclude with topic three."""
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}") # Note: token_count might be added post-hoc or not available depending on implementation
print(f"Start index: {chunk.start_index}")
print(f"End index: {chunk.end_index}")
```
### Batch Chunking
```python theme={"system"}
texts = [
"Document 1 discussing AI ethics. Then shifts to model training techniques.",
"Document 2 about pygmy hippos. Their habitat and diet. Then conservation efforts."
]
batch_chunks = chunker.chunk_batch(texts)
for doc_chunks in batch_chunks:
for chunk in doc_chunks:
print(f"Chunk: {chunk.text}")
```
### Using as a Callable
```python theme={"system"}
# Single text
chunks = chunker("Text discussing topic A... then topic B...")
# Multiple texts
batch_chunks = chunker(["Text 1...", "Text 2..."])
```
## Return Type
NeuralChunker returns chunks as `Chunk` objects.
```python theme={"system"}
from dataclasses import dataclass
from typing import Optional, Union
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
# Chunkers Overview
Source: https://docs.chonkie.ai/oss/chunkers/overview
Overview of the different chunkers available in Chonkie
Chonkie provides multiple chunking strategies to handle different text processing needs. Each chunker in Chonkie is designed to follow the same core principles outlined in the [concepts](/common/concepts) page.
Splits code based on its structure using ASTs. Ideal for chunking source
code files.
SIMD-accelerated byte-based chunking at 100+ GB/s. Best for high-throughput
pipelines where byte size limits are acceptable.
Chunks using Late Chunking algorithm, best for higher recall in your RAG
applications.
Uses a fine-tuned BERT model to split text based on semantic shifts. Great
for topic-coherent chunks.
Recursively chunks documents into smaller chunks. Best for long documents
with well-defined structure.
Groups content based on semantic similarity. Best for preserving context and
topical coherence.
Splits text at sentence boundaries. Perfect for maintaining semantic
completeness at the sentence level.
Agentic chunking using generative models (LLMs) via the Genie interface for
S-tier chunk quality. 🦛🧞
Splits large markdown tables into smaller, manageable chunks by row,
preserving headers. Great for tabular data in RAG and LLM pipelines.
Segments text using the TeraflopAI Segmentation API. Ideal for
domain-specific segmentation such as legal documents.
Splits text into fixed-size token chunks. Best for maintaining consistent
chunk sizes and working with token-based models.
## Availability
Different chunkers are available depending on your installation:
| Chunker | Default | embeddings | `"chonkie[all]"` | Chonkie JS | API Chunking |
| ---------------- | :-------------------: | :-------------------: | :-------------------: | :-------------------: | :-------------------: |
| TokenChunker | | | | | |
| FastChunker | | | | | |
| SentenceChunker | | | | | |
| RecursiveChunker | | | | | |
| TableChunker | | | | | |
| CodeChunker | | | | | |
| SemanticChunker | | | | | |
| LateChunker | | | | | |
| NeuralChunker | | | | | |
| SlumberChunker | | | | | |
## Common Interface
All chunkers share a consistent interface:
```python Python theme={"system"}
# Single text chunking
chunks = chunker.chunk(text)
# Batch processing
chunks = chunker.chunk_batch(texts)
# Direct calling
chunks = chunker(text) # or chunker([text1, text2])
# Async variants (all chunkers support these)
chunks = await chunker.achunk(text)
chunks = await chunker.achunk_batch(texts)
```
```javascript JavaScript theme={"system"}
// Single text chunking
const chunks = await chunker.chunk(text);
```
## Async Support
Every chunker supports async out of the box — no extra setup required.
| Method | Async Equivalent | Description |
| --------------------- | ---------------------- | ------------------------- |
| `chunk(text)` | `achunk(text)` | Chunk a single text |
| `chunk_batch(texts)` | `achunk_batch(texts)` | Chunk a list of texts |
| `chunk_document(doc)` | `achunk_document(doc)` | Chunk a `Document` object |
### Basic Usage
```python theme={"system"}
import asyncio
from chonkie import RecursiveChunker
async def main():
chunker = RecursiveChunker(chunk_size=512)
chunks = await chunker.achunk("Your document text here...")
all_chunks = await chunker.achunk_batch([
"First document...",
"Second document...",
"Third document...",
])
asyncio.run(main())
```
### Concurrent Chunking
Use `asyncio.gather` to chunk multiple texts concurrently:
```python theme={"system"}
import asyncio
from chonkie import SemanticChunker
async def process_documents(texts: list[str]):
chunker = SemanticChunker(chunk_size=512)
results = await asyncio.gather(
*[chunker.achunk(text) for text in texts]
)
return results
```
### How It Works
* **`achunk` and `achunk_batch`** run the synchronous methods in a thread pool via `asyncio.to_thread`, so CPU-bound chunking does not block your event loop.
* **`achunk_document`** goes further: if the document has pre-existing chunks, it dispatches a concurrent `asyncio.gather` over all of them.
Because `achunk` and `achunk_batch` use `asyncio.to_thread`, they are safe to use in async web frameworks (FastAPI, Starlette, aiohttp, etc.) without blocking the event loop.
## F.A.Q.
Yes, all the chunkers are thread-safe. Though, the performance might vary
since some chunkers use threading under the hood. So, monitor your
performance accordingly.
No. Async support is built into every chunker via `BaseChunker`. Any chunker you import from `chonkie` already has `achunk`, `achunk_batch`, and `achunk_document` available.
Yes, especially when chunking many texts concurrently. `achunk` offloads work to a thread pool, so multiple coroutines can chunk in parallel without blocking the event loop. For single-text chunking the overhead is minimal.
Yes. All chunkers are thread-safe, so sharing a single instance across concurrent `asyncio.gather` calls is fine and avoids redundant initialization costs.
Any framework that uses `asyncio` — FastAPI, Starlette, aiohttp, Sanic, Litestar, and others. The async methods use standard `asyncio` primitives with no framework-specific dependencies.
# Recursive Chunker
Source: https://docs.chonkie.ai/oss/chunkers/recursive-chunker
Recursively chunk documents into smaller chunks.
The RecursiveChunker is a chunker that recursively chunks documents into smaller chunks.
It is a good choice for documents that are long but well structured, for example, a book or a research paper.
## API Reference
To use the `RecursiveChunker` via the API, check out the [API reference documentation](../../api/chunkers/recursive-chunker).
## Installation
The RecursiveChunker is included in the base installation of Chonkie. No additional dependencies are required.
If you would like to use custom tokenizers in JavaScript, please install the
`@chonkiejs/token` library
## Initialization
The RecursiveChunker uses `RecursiveRules` to determine how to chunk the text.
The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree.
Find more information about the rules in the [Additional Information](#additional-information) section.
```python Python theme={"system"}
from chonkie import RecursiveChunker, RecursiveRules
chunker = RecursiveChunker(
tokenizer: Union[str, Callable, Any] = "character",
chunk_size: int = 2048,
rules: RecursiveRules = RecursiveRules(),
min_characters_per_chunk: int = 24,
)
```
```javascript JavaScript theme={"system"}
import { RecursiveChunker } from "@chonkiejs/core"
const chunker = await RecursiveChunker.create({
tokenizer: "character",
chunkSize: 2048,
rules: new RecursiveRules(),
minCharactersPerChunk: 24
});
```
You can also initialize the RecursiveChunker using a recipe. Recipes are pre-defined rules for common chunking tasks.
Find all available recipes on our Hugging Face Hub [here](https://huggingface.co/datasets/chonkie-ai/recipes).
Recipes are supported on Python only
```python theme={"system"}
from chonkie import RecursiveChunker
# Initialize the recursive chunker to chunk Markdown
chunker = RecursiveChunker.from_recipe("markdown", lang="en")
# Initialize the recursive chunker to chunk Hindi texts
chunker = RecursiveChunker.from_recipe(lang="hi")
```
## Parameters
Tokenizer to use. Can be a string identifier or a tokenizer instance
Maximum number of tokens per chunk
Rules to use for chunking.
Minimum number of characters per chunk
## Usage
### Single Text Chunking
```python Python theme={"system"}
text = """This is the first sentence. This is the second sentence.
And here's a third one with some additional context."""
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
```
```javascript JavaScript theme={"system"}
const text = "This is the first sentence. This is the second sentence \n. And here's a third one with some additional context."
chunks = await chunker.chunk(text)
for (const chunk of chunks):
console.log(`Chunk text: ${chunk.text}`)
console.log(`Tokens: ${chunk.tokenCount}`);
}
```
### Batch Chunking
```python theme={"system"}
texts = [
"This is the first sentence. This is the second sentence.
And here's a third one with some additional context.",
"This is the first sentence. This is the second sentence.
And here's a third one with some additional context.",
]
chunks = chunker.chunk_batch(texts)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
```
### Using as a Callable
```python theme={"system"}
# Single text
chunks = chunker("This is the first sentence. This is the second sentence.")
# Multiple texts
batch_chunks = chunker(["Text 1. More text.", "Text 2. More."])
```
## Return Type
The RecursiveChunker returns chunks as `Chunk` objects:
```python Python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
```javascript JavaScript theme={"system"}
class Chunk {
/** The text content of the chunk */
text: string;
/** The starting index of the chunk in the original text */
startIndex: number;
/** The ending index of the chunk in the original text */
endIndex: number;
/** The number of tokens in the chunk */
tokenCount: number;
/** Optional embedding vector for the chunk */
embedding?: number[];
/* Get a string representation of the chunk */
toString(): string;
}
```
## Additional Information
The RecursiveChunker uses the `RecursiveRules` class to determine the chunking rules. The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree.
```python Python theme={"system"}
@dataclass
class RecursiveRules:
rules: list[RecursiveLevel]
@dataclass
class RecursiveLevel:
delimiters: Optional[Union[str, list[str]]]
whitespace: bool = False
include_delim: Optional[Literal["prev", "next"]]) # Whether to include the delimiter in the previous chunk or the next chunk.
```
```javascript JavaScript theme={"system"}
class RecursiveRules {
levels: RecursiveLevel[];
}
class RecursiveLevel {
delimiters?: string | string[];
whitespace: boolean;
includeDelim: 'prev' | 'next';
}
```
You can pass in custom rules to the RecursiveChunker, or use the default rules. The default rules are designed to be a good starting point for most documents, but you can customize them to your needs.
`RecursiveLevel` expects the list of custom delimiters to **not** include
whitespace. If whitespace as a delimiter is required, you can set the
`whitespace` parameter in the `RecursiveLevel` class to True. Note that if
`whitespace = True`, you cannot pass a list of custom delimiters.
# SDPM Chunker (Legacy)
Source: https://docs.chonkie.ai/oss/chunkers/sdpm-chunker
Semantic Double-Pass Merging chunker - now integrated into SemanticChunker
**Deprecated as of v1.2.0**
The SDPM (Semantic Double-Pass Merging) functionality has been integrated into the main `SemanticChunker`.
**Recommended Migration:**
```python theme={"system"}
# Old way (deprecated)
from chonkie.legacy import SDPMChunker
chunker = SDPMChunker(skip_window=1)
# New way (recommended)
from chonkie import SemanticChunker
chunker = SemanticChunker(skip_window=1)
```
The new SemanticChunker provides all SDPM capabilities plus additional improvements like Savitzky-Golay filtering for better boundary detection.
The `SDPMChunker` extends semantic chunking by using a double-pass merging approach. It first groups content by semantic similarity, then merges similar groups within a skip window, allowing it to connect related content that may not be consecutive in the text.
## Why Use the New SemanticChunker Instead?
The new `SemanticChunker` includes all SDPM functionality plus:
* **Better performance**: Optimized C extensions for faster processing
* **Smoother boundaries**: Savitzky-Golay filtering for noise reduction
* **Cleaner API**: Simplified parameter names and improved defaults
* **Active development**: Ongoing improvements and bug fixes
## Legacy Installation
If you need to use the legacy version for compatibility:
```bash theme={"system"}
pip install "chonkie[semantic]"
```
Then import from the legacy module:
```python theme={"system"}
from chonkie.legacy import SDPMChunker
```
## Legacy Usage
This documentation is preserved for users who need to maintain existing code using SDPMChunker. For new projects, please use the main [SemanticChunker](./semantic-chunker).
### Basic Initialization
```python theme={"system"}
from chonkie.legacy import SDPMChunker
# Legacy initialization
chunker = SDPMChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.5,
chunk_size=2048,
min_sentences=1,
skip_window=1
)
```
### Legacy Parameters
The legacy SDPMChunker uses these parameters (many now renamed in the new SemanticChunker):
* `embedding_model`: Model identifier or embedding instance
* `mode`: "cumulative" or "window" (removed in new version)
* `threshold`: Similarity threshold (0-1) or "auto"
* `chunk_size`: Maximum tokens per chunk
* `similarity_window`: Sentences for threshold calculation
* `min_sentences`: Minimum sentences per chunk (now `min_sentences_per_chunk`)
* `min_chunk_size`: Minimum tokens per chunk (removed in new version)
* `min_characters_per_sentence`: Minimum characters per sentence
* `threshold_step`: Step size for threshold calculation (removed in new version)
* `skip_window`: Number of chunks to skip when merging
### Example Migration
#### Old Code (Legacy)
```python theme={"system"}
from chonkie.legacy import SDPMChunker
chunker = SDPMChunker(
embedding_model="minishlab/potion-base-32M",
mode="window",
threshold="auto",
chunk_size=512,
min_sentences=1,
min_chunk_size=2,
skip_window=1
)
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Sentences: {len(chunk.sentences)}")
```
#### New Code (Recommended)
```python theme={"system"}
from chonkie import SemanticChunker
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7, # Explicit threshold instead of "auto"
chunk_size=512,
min_sentences_per_chunk=1, # Renamed parameter
skip_window=1 # Same functionality
)
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Token count: {chunk.token_count}")
```
## Return Type Changes
### Legacy Return Type
The legacy SDPMChunker returns `SemanticChunk` objects with sentence details:
```python theme={"system"}
@dataclass
class SemanticChunk:
text: str
start_index: int
end_index: int
token_count: int
sentences: list[SemanticSentence] # Detailed sentence information
```
### New Return Type
The new SemanticChunker returns simpler `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str
start_index: int
end_index: int
token_count: int
# No sentence details - cleaner and more efficient
```
## Full Legacy Documentation
For users who must use the legacy version, the complete original functionality remains available:
```python theme={"system"}
from chonkie.legacy import SDPMChunker
# All original parameters still work
chunker = SDPMChunker(
embedding_model="minishlab/potion-base-32M",
mode="window",
threshold="auto",
chunk_size=2048,
similarity_window=1,
min_sentences=1,
min_chunk_size=2,
min_characters_per_sentence=12,
threshold_step=0.01,
delim=[". ", "! ", "? ", "\n"],
include_delim="prev",
skip_window=1
)
# Original methods preserved
chunks = chunker.chunk(text)
batch_chunks = chunker.chunk_batch(texts)
```
## Support
While the legacy SDPMChunker remains available for backward compatibility, it is no longer actively developed. Please consider migrating to the new SemanticChunker for:
* Better performance
* Active bug fixes
* New features
* Ongoing support
For migration assistance, see the [SemanticChunker documentation](./semantic-chunker) or open an issue on our [GitHub repository](https://github.com/chonkie-ai/chonkie).
# Semantic Chunker
Source: https://docs.chonkie.ai/oss/chunkers/semantic-chunker
Split text into chunks based on semantic similarity with advanced features
The `SemanticChunker` splits text into chunks based on semantic similarity, ensuring that related content stays together in the same chunk. This chunker now includes advanced features like Savitzky-Golay filtering for smoother boundary detection and skip-window merging for connecting related content that may not be consecutive. This chunker is inspired by the work of [Greg Kamradt](https://github.com/gkamradt).
## API Reference
To use the `SemanticChunker` via the API, check out the [API reference documentation](../../api/chunkers/semantic-chunker).
## Installation
SemanticChunker requires additional dependencies for semantic capabilities. You can install it with:
```bash Python theme={"system"}
pip install "chonkie[semantic]"
```
```bash JavaScript theme={"system"}
npm install @chonkiejs/core
```
For installation instructions, see the [Installation Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import SemanticChunker
# Basic initialization with default parameters
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M", # Default model
threshold=0.8, # Similarity threshold (0-1)
chunk_size=2048, # Maximum tokens per chunk
similarity_window=3, # Window for similarity calculation
skip_window=0 # Skip-and-merge window (0=disabled)
)
# With skip-and-merge enabled (similar to legacy SDPM behavior)
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7,
chunk_size=2048,
skip_window=1 # Enable merging of similar non-consecutive groups
)
```
```javascript theme={"system"}
import { SemanticChunker } from "@chonkiejs/core";
// Basic initialization with custom embedding function
const embedFn = async (texts) => {
// Your embedding logic here
// Return array of embeddings for each text
};
const chunker = await SemanticChunker.create({
embedFunction: embedFn, // Custom embedding function
threshold: 0.8, // Similarity threshold (0-1)
chunkSize: 2048, // Maximum tokens per chunk
similarityWindow: 3, // Window for similarity calculation
skipWindow: 0 // Skip-and-merge window (0=disabled)
});
```
## Parameters
Model identifier or embedding model instance
Similarity threshold for grouping sentences (0-1). Lower values create larger groups.
Maximum tokens per chunk
Number of sentences to consider for similarity calculation
Minimum number of sentences per chunk
Minimum number of characters per sentence
Number of groups to skip when looking for similar content to merge.
* `0` (default): No skip-and-merge, uses standard semantic grouping
* `1` or higher: Enables merging of semantically similar groups within the skip window
This feature allows the chunker to connect related content that may not be consecutive in the text.
Window length for the Savitzky-Golay filter used in boundary detection
Polynomial order for the Savitzky-Golay filter
Tolerance for the Savitzky-Golay filter boundary detection
Delimiters to split sentences on
Include delimiters in the chunk text. Specify whether to include with the previous or next sentence.
## Basic Usage
```python theme={"system"}
from chonkie import SemanticChunker
# Initialize with semantic similarity grouping
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7, # Similarity threshold
chunk_size=512
)
text = """Your document text with multiple topics and themes..."""
chunks = chunker.chunk(text)
# Process chunks
for chunk in chunks:
print(f"Chunk: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
```
```javascript theme={"system"}
import { SemanticChunker } from "@chonkiejs/core";
// Define custom embedding function
const embedFn = async (texts) => {
// Your embedding logic here (e.g., call to an API)
// Return array of embeddings for each text
};
// Initialize with semantic similarity grouping
const chunker = await SemanticChunker.create({
embedFunction: embedFn,
threshold: 0.7, // Similarity threshold
chunkSize: 512
});
const text = "Your document text with multiple topics and themes...";
const chunks = await chunker.chunk(text);
// Process chunks
for (const chunk of chunks) {
console.log(`Chunk: ${chunk.text.slice(0, 50)}...`);
console.log(`Tokens: ${chunk.tokenCount}`);
}
```
## Examples
```python theme={"system"}
from chonkie import SemanticChunker
text = """Artificial intelligence is transforming industries worldwide.
Machine learning algorithms can now process vast amounts of data efficiently.
Deep learning models have achieved remarkable accuracy in complex tasks.
Climate change poses significant challenges to our planet.
Rising temperatures affect ecosystems and biodiversity globally.
Sustainable practices are essential for environmental preservation.
Quantum computing represents a paradigm shift in computation.
These systems leverage quantum mechanical phenomena for processing.
Potential applications include cryptography and drug discovery."""
# Create semantic chunker
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.75, # Higher threshold = more similar content grouped
chunk_size=1024
)
chunks = chunker.chunk(text)
# Analyze semantic groupings
for i, chunk in enumerate(chunks):
print(f"\n--- Semantic Group {i+1} ---")
print(f"Content: {chunk.text[:100]}...")
print(f"Token count: {chunk.token_count}")
print(f"Theme: {chunk.text.split('.')[0]}") # First sentence as theme indicator
```
```python theme={"system"}
from chonkie import SemanticChunker
# Text with alternating topics
text = """Neural networks process information through interconnected nodes.
The stock market experienced significant volatility this quarter.
Deep learning models require substantial training data for optimization.
Economic indicators point to potential recession risks ahead.
GPU acceleration has revolutionized machine learning computations.
Federal reserve policies impact global financial markets.
Transformer architectures dominate modern NLP applications.
Cryptocurrency markets show correlation with traditional assets."""
# Enable skip-window to merge non-consecutive similar content
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.65,
chunk_size=512,
skip_window=2 # Look ahead 2 groups for similar content
)
chunks = chunker.chunk(text)
# AI-related content will be grouped together
# Financial content will be grouped separately
for i, chunk in enumerate(chunks):
print(f"\nGroup {i+1}: {len(chunk.text.split('.'))} sentences")
print(f"Preview: {chunk.text[:80]}...")
```
```python theme={"system"}
from chonkie import SemanticChunker
text = """Your comprehensive document with various topics..."""
# Experiment with different thresholds
thresholds = [0.5, 0.7, 0.9]
for threshold in thresholds:
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=threshold,
chunk_size=512,
similarity_window=3 # Consider 3 sentences for similarity
)
chunks = chunker.chunk(text)
print(f"\nThreshold {threshold}: {len(chunks)} chunks created")
# Lower threshold = larger, more diverse chunks
# Higher threshold = smaller, more focused chunks
avg_size = sum(c.token_count for c in chunks) / len(chunks)
print(f"Average chunk size: {avg_size:.1f} tokens")
```
```python theme={"system"}
from chonkie import SemanticChunker
# Initialize chunker once
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7,
chunk_size=1024,
min_sentences_per_chunk=2 # Ensure meaningful chunks
)
# Multiple documents with different topics
documents = [
"""Document about artificial intelligence and machine learning...""",
"""Document about climate change and environmental science...""",
"""Document about quantum computing and physics..."""
]
# Process all documents
batch_results = chunker.chunk_batch(documents)
# Analyze results
for doc_idx, chunks in enumerate(batch_results):
print(f"\nDocument {doc_idx + 1}:")
print(f" Total chunks: {len(chunks)}")
print(f" Total tokens: {sum(c.token_count for c in chunks)}")
# Show semantic boundaries
for i, chunk in enumerate(chunks):
first_sentence = chunk.text.split('.')[0]
print(f" Chunk {i+1}: {first_sentence[:50]}...")
```
```python theme={"system"}
from chonkie import SemanticChunker
from chonkie.embeddings import AutoEmbeddings
# Use AutoEmbeddings for automatic model selection
embeddings = AutoEmbeddings.get_embeddings(
model="sentence-transformers/all-MiniLM-L6-v2"
)
chunker = SemanticChunker(
embedding_model=embeddings,
threshold=0.8,
chunk_size=512
)
# Or use specific embedding providers
from chonkie.embeddings import OpenAIEmbeddings
openai_embeddings = OpenAIEmbeddings(
model="text-embedding-ada-002"
)
chunker = SemanticChunker(
embedding_model=openai_embeddings,
threshold=0.75,
chunk_size=1024
)
text = "Your text to chunk with custom embeddings..."
chunks = chunker.chunk(text)
```
```python theme={"system"}
from chonkie import SemanticChunker
# Configure Savitzky-Golay filter for smoother boundaries
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7,
chunk_size=512,
filter_window=7, # Larger window for smoother filtering
filter_polyorder=4, # Higher order polynomial
filter_tolerance=0.15 # Stricter boundary detection
)
text = """Complex document with subtle topic transitions..."""
chunks = chunker.chunk(text)
# The filtering helps identify more natural semantic boundaries
# especially in documents with gradual topic shifts
for chunk in chunks:
print(f"Smooth boundary chunk: {chunk.text[:60]}...")
```
```python theme={"system"}
from chonkie import SemanticChunker
# Customize sentence detection
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7,
chunk_size=1024,
min_sentences_per_chunk=3, # At least 3 sentences per chunk
min_characters_per_sentence=30, # Filter out short fragments
delim=[". ", "! ", "? ", "\n\n"], # Custom sentence delimiters
include_delim="prev" # Include delimiter with previous sentence
)
# Text with various sentence structures
text = """Short sentence. This is a much longer sentence with more detail.
Question here? Exclamation point! New paragraph starts here.
Another paragraph with different content..."""
chunks = chunker.chunk(text)
for chunk in chunks:
sentences = chunk.text.split('. ')
print(f"Chunk with {len(sentences)} sentences")
```
```python theme={"system"}
from chonkie import SemanticChunker
from chonkie.refinery import OverlapRefinery, EmbeddingsRefinery
# Create semantic chunker
chunker = SemanticChunker(
embedding_model="minishlab/potion-base-32M",
threshold=0.7,
chunk_size=512
)
# Add refineries for RAG optimization
overlap_refinery = OverlapRefinery(overlap_size=50)
embeddings_refinery = EmbeddingsRefinery(
embedding_model="minishlab/potion-base-32M"
)
# Process document
text = """Your document for RAG system..."""
chunks = chunker.chunk(text)
# Apply refinements
chunks = overlap_refinery.refine(chunks)
chunks = embeddings_refinery.refine(chunks) # Add embeddings
# Ready for vector database
for chunk in chunks:
print(f"Chunk ready for indexing: {chunk.text[:50]}...")
if chunk.embedding is not None:
print(f" Embedding shape: {chunk.embedding.shape}")
```
## Advanced Features
### Savitzky-Golay Filtering
The SemanticChunker uses Savitzky-Golay filtering for smoother boundary detection in similarity curves. This reduces noise in the semantic similarity signal and provides more stable chunk boundaries.
### Skip-Window Merging
When `skip_window > 0`, the chunker can merge semantically similar groups that are not consecutive. This is useful for:
* Documents with alternating topics
* Content with recurring themes
* Technical documents with distributed related sections
## Supported Embeddings
SemanticChunker supports multiple embedding providers through Chonkie's embedding system. See the [Embeddings Overview](/python-sdk/embeddings/overview) for more information.
## Return Type
SemanticChunker returns `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str
start_index: int
end_index: int
token_count: int
```
# Sentence Chunker
Source: https://docs.chonkie.ai/oss/chunkers/sentence-chunker
Split text into chunks while preserving sentence boundaries
The `SentenceChunker` splits text into chunks while preserving complete sentences, ensuring that each chunk maintains proper sentence boundaries and context.
## API Reference
To use the `SentenceChunker` via the API, check out the [API reference documentation](../../api/chunkers/sentence-chunker).
## Installation
SentenceChunker is included in the base installation of Chonkie. No additional dependencies are required.
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python Python theme={"system"}
from chonkie import SentenceChunker
# Basic initialization with default parameters
chunker = SentenceChunker(
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
chunk_size=2048, # Maximum tokens per chunk
chunk_overlap=128, # Overlap between chunks
min_sentences_per_chunk=1 # Minimum sentences in each chunk
)
```
```javascript JavaScript theme={"system"}
import { SentenceChunker } from "@chonkiejs/core";
// Basic initialization with default parameters
const chunker = await SentenceChunker.create({
tokenizer: "character", // Default tokenizer
chunkSize: 2048, // Maximum tokens per chunk
chunkOverlap: 128, // Overlap between chunks
minSentencesPerChunk: 1 // Minimum sentences in each chunk
});
```
## Parameters
Tokenizer to use. Can be a string identifier ("character", "word", "byte", "gpt2",
etc.) or a tokenizer instance
Maximum number of tokens per chunk
Number of overlapping tokens between chunks
Minimum number of sentences to include in each chunk
Minimum number of characters per sentence
Use approximate token counting for faster processing.
This field is deprecated and will be removed in future versions.
Delimiters to split sentences on
Specify whether to include the delimiter with the previous or next chunk.
## Usage
### Single Text Chunking
```python Python theme={"system"}
text = """This is the first sentence. This is the second sentence.
And here's a third one with some additional context."""
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
```
```javascript JavaScript theme={"system"}
const text = `This is the first sentence. This is the second sentence.
And here's a third one with some additional context.`;
const chunks = await chunker.chunk(text);
for (const chunk of chunks) {
console.log(`Chunk text: ${chunk.text}`);
console.log(`Token count: ${chunk.tokenCount}`);
}
```
### Batch Chunking
```python Python theme={"system"}
texts = [
"First document. With multiple sentences.",
"Second document. Also with sentences. And more context."
]
batch_chunks = chunker.chunk_batch(texts)
for doc_chunks in batch_chunks:
for chunk in doc_chunks:
print(f"Chunk: {chunk.text}")
```
```javascript JavaScript theme={"system"}
const texts = [
"First document. With multiple sentences.",
"Second document. Also with sentences. And more context."
];
const batchChunks = await chunker.chunkBatch(texts);
for (const docChunks of batchChunks) {
for (const chunk of docChunks) {
console.log(`Chunk: ${chunk.text}`);
}
}
```
### Using as a Callable
```python theme={"system"}
# Single text
chunks = chunker("First sentence. Second sentence.")
# Multiple texts
batch_chunks = chunker(["Text 1. More text.", "Text 2. More."])
```
## Supported Tokenizers
SentenceChunker supports multiple tokenizer backends:
* **TikToken** (Recommended)
```python theme={"system"}
import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
```
* **AutoTikTokenizer**
```python theme={"system"}
from autotiktokenizer import AutoTikTokenizer
tokenizer = AutoTikTokenizer.from_pretrained("gpt2")
```
* **Hugging Face Tokenizers**
```python theme={"system"}
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("gpt2")
```
* **Transformers**
```python theme={"system"}
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
```
## Return Type
SentenceChunker returns chunks as `Chunk` objects:
```python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
# Slumber Chunker
Source: https://docs.chonkie.ai/oss/chunkers/slumber-chunker
Agentic chunking powered by generative models via the Genie interface
Meet the `SlumberChunker` – Chonkie's first **agentic chunker**! This isn't your average chunker; it uses the reasoning power of large generative models (LLMs) to understand your text deeply and create truly S-tier chunks.
## API Reference
To use the `SlumberChunker` via the API, check out the [API reference documentation](../../api/chunkers/slumber-chunker).
## Introducing Genie! 🧞
The magic behind `SlumberChunker` is **Genie**, Chonkie's interface for integrating generative models and APIs. Genie allows `SlumberChunker` to intelligently analyze text structure, identify optimal split points, and even summarize or rephrase content for the best possible chunk quality.
**Available Genies:**
* `GeminiGenie` - Google Gemini APIs
* `OpenAIGenie` - OpenAI APIs (also works with OpenAI-compatible providers)
* `AzureOpenAIGenie` - Azure OpenAI APIs
* `GroqGenie` - Fast inference on Groq hardware
* `CerebrasGenie` - Fastest inference on Cerebras hardware
To unleash the power of SlumberChunker and Genie, you need the `[genie]`
optional install. This includes the necessary libraries to connect to various
generative model APIs.
```bash theme={"system"}
pip install "chonkie[genie]"
```
## Installation
As mentioned, SlumberChunker requires the `[genie]` optional install:
```bash theme={"system"}
pip install "chonkie[genie]"
```
For general installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python theme={"system"}
from chonkie import SlumberChunker
from chonkie.genie import GeminiGenie
# Optional: Initialize Genie
genie = GeminiGenie("gemini-3-pro-preview")
# Basic initialization
chunker = SlumberChunker(
genie=genie, # Genie interface to use
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
chunk_size=1024, # Maximum chunk size
candidate_size=128, # How many tokens Genie looks at for potential splits
min_characters_per_chunk=24, # Minimum number of characters per chunk
verbose=True # See the progress bar for the chunking process
)
# You can also rely on default Genie setup if configured globally
# chunker = SlumberChunker() # Uses default Genie if available
```
## Parameters
An instance of a Genie interface (e.g., `GeminiGenie`). If `None`, tries to
load a default Genie configuration, which is
`GeminiGenie("gemini-3-pro-preview")`
Tokenizer or token counting function used for initial splitting and size
estimation.
The target maximum number of tokens per chunk. Genie will try to adhere to
this.
Initial recursive rules used to generate candidate split points before Genie
refines them. See
[RecursiveChunker](/oss/chunkers/recursive-chunker#additional-information) for
details.
The number of tokens around a potential split point that Genie examines to
make its decision.
Minimum number of characters required for a chunk to be considered valid.
If `True`, prints detailed information about Genie's decision-making process
during chunking. Useful for debugging!
## Usage
### Single Text Chunking
```python theme={"system"}
text = """Complex document with interwoven ideas. Section 1 introduces concept A.
Section 2 discusses concept B, but references A frequently.
Section 3 concludes by merging A and B. Traditional chunkers might struggle here."""
# Assuming 'chunker' is initialized as shown above
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
print(f"Start index: {chunk.start_index}")
print(f"End index: {chunk.end_index}")
# SlumberChunk might have additional metadata from Genie
```
### Batch Chunking
```python theme={"system"}
texts = [
"First document requiring nuanced splitting...",
"Second document where agentic understanding helps..."
]
batch_chunks = chunker.chunk_batch(texts) # Note: Batch processing might be slower due to LLM calls
for doc_chunks in batch_chunks:
for chunk in doc_chunks:
print(f"Chunk: {chunk.text}")
```
### Using as a Callable
```python theme={"system"}
# Single text
chunks = chunker("Let Genie decide the best way to CHONK this...")
# Multiple texts
batch_chunks = chunker(["Text 1...", "Text 2..."])
```
## Return Type
SlumberChunker returns chunks as `Chunk` objects.
```python theme={"system"}
from dataclasses import dataclass
from typing import Optional, Union
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
# Table Chunker
Source: https://docs.chonkie.ai/oss/chunkers/table-chunker
Split markdown or HTML tables into manageable chunks by row, preserving headers.
The `TableChunker` splits large markdown or HTML tables into smaller, manageable chunks by row, always preserving the header. This is especially useful for processing, indexing, or embedding tabular data in LLM and RAG pipelines.
## API Reference
Use the `recursive` endpoint to access table chunking functionality.
On the API, the table chunker operates as part of the recursive chunker,
allowing you to process documents containing inline tables while ensuring
that table structures remain intact across chunk boundaries.
## Installation
TableChunker is included in the base installation of Chonkie. No additional dependencies are required.
For installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python row chunker theme={"system"}
from chonkie import TableChunker
# Basic initialization custom parameters
chunker = TableChunker(
tokenizer="row", # Chunk by rows, valid only for TableChunker
chunk_size=3 # Maximum number of rows per chunk (not including header)
)
```
```python token chunker theme={"system"}
from chonkie import TableChunker
# Basic initialization
chunker = TableChunker(
tokenizer="character", # using Character chunker (or you can use "gpt2", ...)
chunk_size=16 # Maximum number of tokens/characters per chunk
)
```
```javascript row chunker theme={"system"}
import { TableChunker } from "@chonkiejs/core";
// Basic initialization with custom parameters
const chunker = await TableChunker.create({
tokenizer: "row", // Chunk by rows, valid only for TableChunker
chunkSize: 3 // Maximum number of rows per chunk (not including header)
});
```
```javascript token chunker theme={"system"}
import { TableChunker } from "@chonkiejs/core";
// Basic initialization
const chunker = await TableChunker.create({
tokenizer: "character", // using Character chunker
chunkSize: 16 // Maximum number of tokens/characters per chunk
});
```
## Parameters
Tokenizer to use. Default is "row". Can be a string identifier ("row", "character", "word", "gpt2", "byte",
etc.) or a tokenizer instance.
Maximum number of rows (if tokenizer="row") or tokens/characters per chunk.
## Usage
```python Markdown (Row-Based) theme={"system"}
from chonkie import TableChunker
table = """
| Name | Age | City |
|--------|-----|----------|
| Alice | 30 | New York |
| Bob | 25 | London |
| Carol | 28 | Paris |
| Dave | 35 | Berlin |
"""
chunker = TableChunker(tokenizer="row", chunk_size=3)
chunks = chunker.chunk(table)
for chunk in chunks:
print(chunk.text)
# Each chunk is a valid markdown table segment, always including the header. For the example above and `chunk_size=3`, you might get:
# >>>
# | Name | Age | City |
# |--------|-----|----------|
# | Alice | 30 | New York |
# | Bob | 25 | London |
# | Carol | 28 | Paris |
# | Name | Age | City |
# |--------|-----|----------|
# | Dave | 35 | Berlin |
```
```python Markdown (Token-Based) theme={"system"}
from chonkie import TableChunker
table = """
| Name | Age | City |
|--------|-----|----------|
| Alice | 30 | New York |
| Bob | 25 | London |
| Carol | 28 | Paris |
| Dave | 35 | Berlin |
"""
chunker = TableChunker(tokenizer="character",chunk_size=16)
chunks = chunker.chunk(table)
for chunk in chunks:
print(chunk.text)
# Each chunk is a valid markdown table segment, always including the header. For the example above and `chunk_size=16`, you might get:
# >>>
# | Name | Age | City |
# | ----- | --- | -------- |
# | Alice | 30 | New York |
# | Bob | 25 | London |
# | Name | Age | City |
# | ----- | --- | ------ |
# | Carol | 28 | Paris |
# | Dave | 35 | Berlin |
```
```python HTML Tables theme={"system"}
from chonkie import TableChunker
html_table = """
ID
Status
1
Active
2
Pending
3
Inactive
4
Active
"""
# HTML tables are chunked while preserving
, , and tags
chunker = TableChunker(tokenizer="row", chunk_size=2)
chunks = chunker.chunk(html_table)
for chunk in chunks:
print(f"--- HTML Chunk ---\n{chunk.text}\n")
```
```javascript Markdown (Row-Based) theme={"system"}
import { TableChunker } from "@chonkiejs/core";
const table = `
| Name | Age | City |
|--------|-----|----------|
| Alice | 30 | New York |
| Bob | 25 | London |
| Carol | 28 | Paris |
| Dave | 35 | Berlin |
`;
const chunker = await TableChunker.create({ tokenizer: "row", chunkSize: 3 });
const chunks = await chunker.chunk(table);
for (const chunk of chunks) {
console.log(chunk.text);
}
```
```javascript Markdown (Token-Based) theme={"system"}
import { TableChunker } from "@chonkiejs/core";
const table = `
| Name | Age | City |
|--------|-----|----------|
| Alice | 30 | New York |
| Bob | 25 | London |
| Carol | 28 | Paris |
| Dave | 35 | Berlin |
`;
const chunker = await TableChunker.create({ tokenizer: "character", chunkSize: 16 });
const chunks = await chunker.chunk(table);
for (const chunk of chunks) {
console.log(chunk.text);
}
```
```javascript HTML Tables theme={"system"}
import { TableChunker } from "@chonkiejs/core";
const htmlTable = `
ID
Status
1
Active
2
Pending
3
Inactive
4
Active
`;
// HTML tables are chunked while preserving
, , and tags
const chunker = await TableChunker.create({ tokenizer: "row", chunkSize: 2 });
const chunks = await chunker.chunk(htmlTable);
for (const chunk of chunks) {
console.log(`--- HTML Chunk ---\n${chunk.text}\n`);
}
```
## Methods
* `chunk(table: str) -> list[Chunk]`: Chunk a markdown table string.
* `chunk_document(document: Document) -> Document`: Chunk all tables in a `MarkdownDocument`.
## Notes
* Supports both standard Markdown pipe tables and HTML `
` elements.
* Requires at least a header, separator, and one data row (for Markdown) or at least one `
` data row for HTML tables (with optional `` and `
` structure).
* If the table fits within the chunk size, it is returned as a single chunk.
* For advanced use, pass a custom tokenizer for token-based chunking.
***
See also: [Chunkers Overview](/oss/chunkers/overview)
# TeraflopAI Chunker
Source: https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker
Segment text using the TeraflopAI Segmentation API
The `TeraflopAIChunker` uses the [TeraflopAI](https://www.teraflopai.com/) Segmentation API to split text into semantically meaningful segments. It is especially useful for domain-specific segmentation such as legal documents.
## Installation
TeraflopAI Chunker requires the `teraflopai` Python package:
```bash theme={"system"}
pip install "chonkie[teraflopai]"
```
For general installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
```python using api_key theme={"system"}
from chonkie import TeraflopAIChunker
# Using an API key (or set the TERAFLOPAI_API_KEY environment variable)
chunker = TeraflopAIChunker(api_key="your_api_key_here")
```
```python using custom_url theme={"system"}
chunker = TeraflopAIChunker(
api_key="your_api_key_here",
url="https://api.segmentation.teraflopai.com/v1/segmentation/free",
)
```
```python using external client theme={"system"}
from teraflopai import TeraflopAI
client = TeraflopAI(
url="https://api.segmentation.teraflopai.com/v1/segmentation/free",
api_key="your_api_key_here",
)
chunker = TeraflopAIChunker(client=client)
```
## Parameters
An existing TeraflopAI client instance. If provided, `url` and `api_key` are
ignored.
The URL for the TeraflopAI segmentation API endpoint.
The API key for authentication. If not provided, it will be read from the
`TERAFLOPAI_API_KEY` environment variable.
The tokenizer used to compute token counts for returned chunks.
## Usage
### Single Text Chunking
```python theme={"system"}
text = """
Global warming refers to the long-term increase in Earth’s average surface temperature due to human activities, primarily the emission of greenhouse gases such as carbon dioxide and methane. These gases trap heat in the atmosphere, leading to significant changes in climate patterns across the globe.
Scientists have observed rising temperatures, melting polar ice caps, and increasing sea levels, all of which pose serious risks to ecosystems and human societies. Extreme weather events such as hurricanes, droughts, and heatwaves are becoming more frequent and intense as a result of these changes.
Governments and organizations around the world are working to reduce emissions, transition to renewable energy sources, and promote sustainable practices. However, global cooperation and immediate action are essential to mitigate the long-term impacts and protect future generations from the most severe consequences of climate change.
Public awareness and individual responsibility also play a crucial role in addressing global warming. Simple actions like reducing energy consumption, minimizing waste, and supporting environmentally friendly initiatives can collectively make a meaningful difference in slowing down this global crisis.
"""
chunks = chunker.chunk(text)
for chunk in chunks:
print(f"Chunk text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
print(f"Start index: {chunk.start_index}")
print(f"End index: {chunk.end_index}")
```
### Batch Chunking
```python theme={"system"}
texts = [
"First document to segment.",
"Second document with more content to segment.",
]
batch_results = chunker(texts)
for i, chunks in enumerate(batch_results):
print(f"Document {i}: {len(chunks)} chunks")
```
### Using with Environment Variable
```bash theme={"system"}
export TERAFLOPAI_API_KEY="your_api_key_here"
```
```python theme={"system"}
from chonkie import TeraflopAIChunker
# No need to pass api_key — it will be read from the environment
chunker = TeraflopAIChunker()
chunks = chunker.chunk("Your text here.")
```
## How It Works
1. The text is sent to the TeraflopAI Segmentation API endpoint.
2. The API returns a list of text segments.
3. Each segment is converted into a Chonkie `Chunk` object with proper `start_index`, `end_index`, and `token_count` fields.
The TeraflopAI Segmentation API performs the segmentation on the server side.
This chunker requires an active internet connection and a valid API key.
# Token Chunker
Source: https://docs.chonkie.ai/oss/chunkers/token-chunker
Split text into fixed-size token chunks with configurable overlap
The `TokenChunker` splits text into chunks based on token count, ensuring each chunk stays within specified token limits.
## API Reference
To use the `TokenChunker` via the API, check out the [API reference documentation](../../api/chunkers/token-chunker).
## Installation
TokenChunker is included in the base installation of Chonkie.
If you would like to use custom tokenizers in JavaScript, please install the
`@chonkiejs/token` library
## Initialization
```python Python theme={"system"}
from chonkie import TokenChunker
# Basic initialization with default parameters
chunker = TokenChunker(
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
chunk_size=2048, # Maximum tokens per chunk
chunk_overlap=128 # Overlap between chunks
)
# Using a custom tokenizer
from tokenizers import Tokenizer
custom_tokenizer = Tokenizer.from_pretrained("your-tokenizer")
chunker = TokenChunker(
tokenizer=custom_tokenizer,
chunk_size=2048,
chunk_overlap=128
)
```
```javascript JavaScript theme={"system"}
import { TokenChunker } from "@chonkiejs/core";
// Create a chunker
let chunker = await TokenChunker.create({
chunkSize: 2048,
chunkOverlap: 128,
});
// Using a custom tokenizer
// NOTE: Requires installation of `@chonkiejs/token`
chunker = TokenChunker.create({
tokenizer: "gpt2",
chunkSize: 2048,
chunkOverlap: 512
});
```
## Parameters
Tokenizer to use. Can be a string identifier ("character", "word", "byte", "gpt2",
etc.) or a tokenizer instance
Maximum number of tokens per chunk
Number or percentage of overlapping tokens between chunks
## Basic Usage
```python Python theme={"system"}
from chonkie import TokenChunker
# Initialize the chunker
chunker = TokenChunker(
tokenizer="gpt2",
chunk_size=512,
chunk_overlap=50
)
# Chunk your text
text = "Your long document text here..."
chunks = chunker.chunk(text)
# Access chunk information
for chunk in chunks:
print(f"Chunk: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
```
```javascript JavaScript theme={"system"}
import { TokenChunker } from "@chonkiejs/core";
// Create a chunker
const chunker = await TokenChunker.create({
chunkSize: 512,
chunkOverlap: 128,
});
// Chunk your text
const chunks = await chunker.chunk("Your text here...");
// Access chunk information
for (const chunk of chunks) {
console.log(chunk.text);
console.log(`Tokens: ${chunk.tokenCount}`);
}
```
## Examples
```python Python theme={"system"}
from chonkie import TokenChunker
# Create a chunker with specific parameters
chunker = TokenChunker(
tokenizer="gpt2",
chunk_size=1024,
chunk_overlap=128
)
text = """Natural language processing has revolutionized how we interact with computers.
Machine learning models can now understand context, generate text, and even translate
between languages with remarkable accuracy. This transformation has enabled applications
ranging from virtual assistants to automated content generation."""
# Chunk the text
chunks = chunker.chunk(text)
# Process each chunk
for i, chunk in enumerate(chunks):
print(f"\n--- Chunk {i+1} ---")
print(f"Text: {chunk.text}")
print(f"Token count: {chunk.token_count}")
print(f"Start index: {chunk.start_index}")
print(f"End index: {chunk.end_index}")
```
```javascript JavaScript theme={"system"}
import { TokenChunker } from "@chonkiejs/core";
// Create a chunker with specific parameters
const chunker = await TokenChunker.create({
chunkSize: 1024,
chunkOverlap: 128,
});
const text = `Natural language processing has revolutionized how we interact with computers.
Machine learning models can now understand context, generate text, and even translate
between languages with remarkable accuracy. This transformation has enabled applications
ranging from virtual assistants to automated content generation.`;
// Chunk the text
const chunks = await chunker.chunk(text);
// Process each chunk
for (let i = 0; i < chunks.length; i++) {
const chunk = chunks[i];
console.log(`\n--- Chunk ${i + 1} ---`);
console.log(`Text: ${chunk.text}`);
console.log(`Token count: ${chunk.tokenCount}`);
console.log(`Start index: ${chunk.startIndex}`);
console.log(`End index: ${chunk.endIndex}`);
}
```
Batch processing is only supported in Python
```python theme={"system"}
from chonkie import TokenChunker
# Initialize chunker for batch processing
chunker = TokenChunker(
tokenizer="gpt2",
chunk_size=512,
chunk_overlap=50
)
# Multiple documents to process
documents = [
"First document about machine learning fundamentals...",
"Second document discussing neural networks...",
"Third document on natural language processing..."
]
# Process all documents at once
batch_chunks = chunker.chunk_batch(documents)
# Iterate through results
for doc_idx, doc_chunks in enumerate(batch_chunks):
print(f"\nDocument {doc_idx + 1}: {len(doc_chunks)} chunks")
for chunk in doc_chunks:
print(f" - Chunk: {chunk.text[:50]}... ({chunk.token_count} tokens)")
```
Custom tokenizers are only supported in Python. See the Installation section for JavaScript tokenizer support.
```python theme={"system"}
from chonkie import TokenChunker
import tiktoken
# Using TikToken with a specific model encoding
tokenizer = tiktoken.get_encoding("cl100k_base") # GPT-4 encoding
chunker = TokenChunker(
tokenizer=tokenizer,
chunk_size=2048,
chunk_overlap=200
)
# Or using Hugging Face tokenizers
from transformers import AutoTokenizer
hf_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
chunker = TokenChunker(
tokenizer=hf_tokenizer,
chunk_size=512,
chunk_overlap=50
)
text = "Your text to chunk with custom tokenizer..."
chunks = chunker.chunk(text)
```
The callable interface is only supported in Python
```python theme={"system"}
from chonkie import TokenChunker
# Initialize once
chunker = TokenChunker(
tokenizer="gpt2",
chunk_size=1024,
chunk_overlap=100
)
# Use as a callable for single text
single_text = "This is a document that needs chunking..."
chunks = chunker(single_text)
print(f"Single text produced {len(chunks)} chunks")
# Use as a callable for multiple texts
multiple_texts = [
"First document text...",
"Second document text...",
"Third document text..."
]
batch_results = chunker(multiple_texts)
print(f"Processed {len(batch_results)} documents")
```
```python Python theme={"system"}
from chonkie import TokenChunker
# Fixed token overlap
chunker_fixed = TokenChunker(
tokenizer="gpt2",
chunk_size=1000,
chunk_overlap=100 # Exactly 100 tokens overlap
)
# Percentage-based overlap
chunker_percent = TokenChunker(
tokenizer="gpt2",
chunk_size=1000,
chunk_overlap=0.1 # 10% overlap (100 tokens for 1000 token chunks)
)
text = "Long document text that will be chunked with overlap..."
# Compare the results
fixed_chunks = chunker_fixed.chunk(text)
percent_chunks = chunker_percent.chunk(text)
print(f"Fixed overlap: {len(fixed_chunks)} chunks")
print(f"Percentage overlap: {len(percent_chunks)} chunks")
```
```javascript JavaScript theme={"system"}
import { TokenChunker } from "@chonkiejs/core";
// Fixed token overlap
const chunkerFixed = await TokenChunker.create({
chunkSize: 1000,
chunkOverlap: 100, // Exactly 100 tokens overlap
});
const text = "Long document text that will be chunked with overlap...";
// Compare the results
const fixedChunks = await chunkerFixed.chunk(text);
console.log(`Fixed overlap (100): ${fixedChunks.length} chunks`);
```
```python Python theme={"system"}
from chonkie import TokenChunker
# Configure for large documents
chunker = TokenChunker(
tokenizer="gpt2",
chunk_size=4096, # Larger chunks for efficiency
chunk_overlap=512 # Maintain context between chunks
)
# Read a large document
with open("large_document.txt", "r") as f:
large_text = f.read()
# Process efficiently
chunks = chunker.chunk(large_text)
print(f"Document statistics:")
print(f" Original length: {len(large_text)} characters")
print(f" Number of chunks: {len(chunks)}")
print(f" Average chunk size: {sum(c.token_count for c in chunks) / len(chunks):.1f} tokens")
# Save chunks for further processing
for i, chunk in enumerate(chunks):
with open(f"chunk_{i:03d}.txt", "w") as f:
f.write(chunk.text)
```
```javascript JavaScript theme={"system"}
import { TokenChunker } from "@chonkiejs/core";
import { readFile, writeFile } from "fs/promises";
// Configure for large documents
const chunker = await TokenChunker.create({
chunkSize: 4096, // Larger chunks for efficiency
chunkOverlap: 512, // Maintain context between chunks
});
// Read a large document
const largeText = await readFile("large_document.txt", "utf-8");
// Process efficiently
const chunks = await chunker.chunk(largeText);
console.log("Document statistics:");
console.log(` Original length: ${largeText.length} characters`);
console.log(` Number of chunks: ${chunks.length}`);
const avgTokenCount =
chunks.reduce((sum, c) => sum + c.tokenCount, 0) / chunks.length;
console.log(` Average chunk size: ${avgTokenCount.toFixed(1)} tokens`);
// Save chunks for further processing
for (let i = 0; i < chunks.length; i++) {
const filename = `chunk_${i.toString().padStart(3, "0")}.txt`;
await writeFile(filename, chunks[i].text);
}
```
## Supported Tokenizers
Changing tokenizer backend is only supported on Python
TokenChunker supports multiple tokenizer backends:
* **TikToken** (Recommended)
```python theme={"system"}
import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
```
* **AutoTikTokenizer**
```python theme={"system"}
from autotiktokenizer import AutoTikTokenizer
tokenizer = AutoTikTokenizer.from_pretrained("gpt2")
```
* **Hugging Face Tokenizers**
```python theme={"system"}
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("gpt2")
```
* **Transformers**
```python theme={"system"}
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
```
## Return Type
TokenChunker returns chunks as `Chunk` objects.
```python Python theme={"system"}
@dataclass
class Chunk:
text: str # The chunk text
start_index: int # Starting position in original text
end_index: int # Ending position in original text
token_count: int # Number of tokens in chunk
context: Optional[str] = None # Optional overlap context text
embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector
```
```javascript JavaScript theme={"system"}
class Chunk {
/** The text content of the chunk */
text: string;
/** The starting index of the chunk in the original text */
startIndex: number;
/** The ending index of the chunk in the original text */
endIndex: number;
/** The number of tokens in the chunk */
tokenCount: number;
/** Optional embedding vector for the chunk */
embedding?: number[];
/* Get a string representation of the chunk */
toString(): string;
}
```
# AutoEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/auto-embeddings
Automatically select the best embeddings handler for your use case
AutoEmbeddings is a class that automatically selects the appropriate embeddings handler for you, based on the model name you provide.
## Installation
Embeddings require the appropriate library to be installed. See the [Installation Guide](/oss/installation) for more information.
## Usage
Load the embeddings handler for the model you want to use.
```python theme={"system"}
from chonkie import AutoEmbeddings
# Get the embeddings handler for SentenceTransformer
embeddings = AutoEmbeddings.get_embeddings("all-MiniLM-L6-v2")
# Get the embeddings handler for OpenAI
embeddings = AutoEmbeddings.get_embeddings("text-embedding-3-large")
# Get the embeddings handler for Model2Vec
embeddings = AutoEmbeddings.get_embeddings("minishlab/potion-base-32M")
```
After loading the embeddings handler, you can use it in the same way you would use any other embeddings handler.
```python theme={"system"}
from chonkie import SemanticChunker
chunker = SemanticChunker(embedding_model=embeddings, threshold=0.7)
# Chunk the text
chunks = chunker(text)
```
SemanticChunkers interally call upon the AutoEmbeddings class to get the
embeddings handler. So you can directly pass in a string to the `embeddings`
parameter as well, as long as it matches one of the models supported by
AutoEmbeddings, and its dependencies are installed.
## Method: `get_embeddings`
The `get_embeddings` method is a factory method that returns an instance of the appropriate embeddings handler.
The name of the embeddings model to use.
An instance of the appropriate embeddings handler of type `BaseEmbeddings`.
# AzureOpenAIEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/azure-embeddings
Embed text using Azure OpenAI embeddings
Embeddings are handled by the `AzureOpenAIEmbeddings` class, which wraps the Azure OpenAI service.
## Installation
Embeddings require the `openai`, `azure-identity`, `numpy`, and `tiktoken` libraries. See the [Installation Guide](/oss/installation) for more information.
## Usage
```python theme={"system"}
from chonkie import AzureOpenAIEmbeddings
# Initialize Azure OpenAI embeddings
embeddings = AzureOpenAIEmbeddings(
azure_endpoint="https://.openai.azure.com/",
azure_api_key="",
model="text-embedding-3-small", # or other supported model
deployment=""
)
# Single embedding
emb = embeddings.embed("your text here")
# Batch embedding
embs = embeddings.embed_batch(["text1", "text2"])
```
## Example
```python theme={"system"}
embeddings = AzureOpenAIEmbeddings(
azure_endpoint="https://my-resource.openai.azure.com/",
azure_api_key="my-key",
model="text-embedding-3-small",
deployment="embedding-deployment"
)
```
# CohereEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/cohere-embeddings
Embed text using Cohere embeddings
Embeddings are handled by the `CohereEmbeddings` class, which is a wrapper around the Cohere API.
## Installation
Embeddings require the `cohere` library. See the [Installation Guide](/oss/installation) for more information.
## Usage
```python theme={"system"}
from chonkie import CohereEmbeddings
# Initialize Cohere embeddings
embeddings = CohereEmbeddings()
# Specify model and API key
embeddings = CohereEmbeddings(model="embed-english-light-v3.0", api_key="YOUR_API_KEY")
```
## Example
```python theme={"system"}
embeddings = CohereEmbeddings()
vectors = embeddings.embed("your text here")
# Or you can
vectors = embeddings.embed_batch(["text1", "text2"])
```
# Create your own custom embeddings handler
Source: https://docs.chonkie.ai/oss/embeddings/custom-embeddings
Chonkie allows you to use your own embeddings handler by creating a child class of the `BaseEmbeddings` class, and implementing the necessary methods. It's quite simple!
## Example
First, we create a child class of the `BaseEmbeddings` class, and implement the necessary methods.
```python theme={"system"}
from chonkie.embeddings import BaseEmbeddings
class CustomEmbeddings(BaseEmbeddings):
@property
def dimension(self) -> int:
...
def embed(self, text: str) -> "np.ndarray":
...
def embed_batch(self, texts: list[str]) -> list["np.ndarray"]:
...
def count_tokens(self, text: str) -> int:
...
def count_tokens_batch(self, texts: list[str]) -> list[int]:
...
def get_tokenizer(self):
...
@classmethod
def is_available(cls) -> bool:
...
def __repr__(self) -> str:
...
```
At this point, we have a custom embeddings handler, we can use it like this:
```python theme={"system"}
embeddings = CustomEmbeddings()
```
But let's say we want to use this together with the `AutoEmbeddings` class, for the sake of convenience. We can do this by registering it with the `EmbeddingsRegistry`.
```python theme={"system"}
from chonkie.embeddings import EmbeddingsRegistry
# Register with the embeddings registry
EmbeddingsRegistry.register(
"custom",
CustomEmbeddings,
pattern=r"^custom/|^model-name",
valid_types=["CustomEmbeddings"]
)
```
Now we can use our custom embeddings handler with the `AutoEmbeddings` class.
```python theme={"system"}
embeddings = AutoEmbeddings.get_embeddings("custom/my-custom-embeddings")
```
Finally, we can use our custom embeddings handler in the same way we would use any other embeddings handler.
```python theme={"system"}
chunker = SemanticChunker(embedding_model=embeddings, threshold=0.7)
chunks = chunker(text)
```
# GeminiEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/gemini-embeddings
Embed text using Google Gemini embeddings via GenAI API
Embeddings are handled by the `GeminiEmbeddings` class, which is a wrapper around the Google GenAI API.
## Installation
Gemini embeddings require the `google-genai` and `numpy` libraries. See the [Installation Guide](/oss/installation) for more information.
```bash theme={"system"}
pip install "chonkie[gemini]"
```
## Usage
```python theme={"system"}
from chonkie import GeminiEmbeddings
# Initialize Gemini embeddings
embeddings = GeminiEmbeddings(
model="gemini-embedding-exp-03-07", # Optional: specify model
api_key="YOUR_GEMINI_API_KEY", # Optional: or set GEMINI_API_KEY env var
task_type="SEMANTIC_SIMILARITY", # Optional: task type
)
# Embed a single text
vector = embeddings.embed("Your text here")
```
## Example
```python theme={"system"}
texts = ["Hello world", "Goodbye world"]
embeddings = GeminiEmbeddings()
vectors = embeddings.embed_batch(texts)
```
# JinaEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/jina-embeddings
JinaEmbeddings is a utility for embedding chunks.
JinaEmbeddings is a utility class to use JinaAI's API for Chonkie's semantic chunking.
## Installation
Embeddings require the `jina` library. See the [Installation Guide](/oss/installation) for more information.
```bash theme={"system"}
pip install "chonkie[jina]"
```
## Usage
```python theme={"system"}
from chonkie import JinaEmbeddings
# Initialize the Jina embeddings
embeddings = JinaEmbeddings()
# Initialize the semantic chunker
chunker = SemanticChunker(embeddings)
# Chunk the text
text = ... # Your text string
# CHONK!
chunks = chunker(text)
```
# Model2VecEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings
Embed text using Model2Vec embeddings
Embeddings are handled by the `Model2VecEmbeddings` class, which is a wrapper around the `model2vec` library.
## Installation
Embeddings require the `model2vec` library. See the [Installation Guide](/oss/installation) for more information.
## Usage
```python theme={"system"}
from chonkie import Model2VecEmbeddings
embeddings = Model2VecEmbeddings()
```
# OpenAIEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/openai-embeddings
Embed text using OpenAI embeddings
Embeddings are handled by the `OpenAIEmbeddings` class, which is a wrapper around the `openai` library.
## Installation
Embeddings require the `openai` library. See the [Installation Guide](/oss/installation) for more information.
## Usage
```python theme={"system"}
from chonkie import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
```
# Embeddings Overview
Source: https://docs.chonkie.ai/oss/embeddings/overview
Overview of the different embeddings available in Chonkie
Chonkie provides a variety of embeddings handlers to handle different embedding models in a consistent manner.
Embeddings handlers are used in conjunction with chunkers to embed chunks of text.
Only few chunkers require embeddings, see the [Chunkers Overview](/oss/chunkers/overview) for more information.
## Installation
Embeddings handlers require additional dependencies. See the [Installation Guide](/oss/installation) for more information.
By default, Chonkie `semantic` installation includes `Model2VecEmbeddings`,
which is the current default embeddings handler
## Available Embeddings
Automatically select the best embeddings handler for your use case.
Embed text using Cohere embeddings (requires `cohere`).
Embed text using SentenceTransformer embeddings (requires
`sentence-transformers`).
Embed text using OpenAI embeddings (requires `openai`).
Embed text using Model2Vec embeddings (requires `model2vec`).
Embed text using Google Gemini embeddings (requires `google-genai`).
Embed text using JinaAI embeddings (requires `jina`).
Embed text using Azure OpenAI embeddings (requires `openai`,
`azure-identity`).
Embed text using VoyageAI embeddings (requires `voyageai`).
## Common Interface
All embeddings handlers share a consistent interface:
```python theme={"system"}
# Single text embedding
emb = embeddings.embed(text)
# Batch processing
emb = embeddings.embed_batch(texts)
# Direct calling
emb = embeddings(text) # or embeddings([text1, text2])
```
# SentenceTransformerEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings
Embed text using SentenceTransformer embedding models
Embeddings are handled by the `SentenceTransformer` class, which is a wrapper around the `sentence-transformers` library.
## Installation
Embeddings require the `sentence-transformers` library. See the [Installation Guide](/oss/installation) for more information.
## Usage
```python theme={"system"}
from chonkie import SentenceTransformerEmbeddings
embeddings = SentenceTransformerEmbeddings("all-MiniLM-L6-v2")
```
# VoyageAIEmbeddings
Source: https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings
Embed text using VoyageAI embeddings
Embeddings are handled by the `VoyageAIEmbeddings` class, which is a wrapper around the VoyageAI API.
## Installation
Embeddings require the `voyageai`, `numpy`, and `tokenizers` libraries. See the [Installation Guide](/oss/installation) for more information.
```bash theme={"system"}
pip install "chonkie[voyageai]"
```
## Usage
```python theme={"system"}
from chonkie import VoyageAIEmbeddings
# Initialize VoyageAI embeddings
embeddings = VoyageAIEmbeddings(
model="voyage-3-large", # Optional: specify model
api_key="YOUR_VOYAGE_API_KEY", # Optional: or set VOYAGE_API_KEY env var
output_dimension=1024, # Optional: set output dimension
batch_size=64 # Optional: set batch size
)
# Embed a single text
vector = embeddings.embed("Your text here")
# Embed a batch of texts
vectors = embeddings.embed_batch(["Text 1", "Text 2"])
```
## Example
```python theme={"system"}
embeddings = VoyageAIEmbeddings()
vectors = embeddings.embed("your text here")
# or you can
vectors = embeddings.embed_batch(["text1", "text2"])
```
# CLI
Source: https://docs.chonkie.ai/oss/experimental/chonkie-cli
Chonkie Command Line Interface
# Chonkie CLI
Chonkie provides a powerful Command Line Interface (CLI) to perform chunking and run pipelines directly from your terminal.
## Installation
The CLI is included with the default `chonkie` installation:
```bash theme={"system"}
pip install chonkie
```
## Basic Usage
The CLI provides a single `chonkie` command with two primary subcommands:
1. **`chunk`** – Quickly chunk text or files.
2. **`pipeline`** – Run full Chonkie pipelines (fetch → chef → chunk → refine → handbook).
To see available options and usage details, use the help flags:
```bash main theme={"system"}
chonkie --help
# Usage: chonkie [OPTIONS] COMMAND [ARGS]...
#
# > 🦛 CHONK your texts with Chonkie
#
# ╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ --install-completion Install completion for the current shell. │
# │ --show-completion Show completion for the current shell, to copy it or customize the installation. │
# │ --help Show this message and exit. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
#
# ╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ chunk Chunk text using a specified chunker and optionally store it. │
# │ pipeline Run a processing pipeline on text or files. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
```bash chunk theme={"system"}
chonkie chunk --help
# Usage: chonkie chunk [OPTIONS] TEXT
#
# ╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ * text TEXT Text to chunk or path to file [required] │
# ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ --chunker TEXT Chunking method to use. Options: code, fast, late, neural, recursive, semantic, sentence, slumber, table, token │
# │ [default: semantic] │
# │ --chunk-size INTEGER Maximum number of tokens per chunk │
# │ --chunk-overlap INTEGER Number of tokens to overlap between chunks │
# │ --threshold FLOAT Threshold for semantic similarity (0-1) │
# │ --chunker-params TEXT Additional parameters for the chunker as key=value pairs (e.g., --chunker-params tokenizer=gpt2 min_characters_per_chunk=50) │
# │ --handshaker TEXT Where to store the chunks. Options: chroma, elastic, milvus, mongodb, pgvector, pinecone, qdrant, turbopuffer, weaviate │
# │ --help Show this message and exit. │
# ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
```bash pipeline theme={"system"}
chonkie pipeline --help
# Usage: chonkie pipeline [OPTIONS] [TEXT]
#
# Run a processing pipeline on text or files.
#
# ╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ text [TEXT] Text to process or path to file │
# ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# ╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
# │ --fetcher TEXT Fetcher method to use (e.g., file) [default: file] │
# │ --d TEXT directory to process, if text is not a file │
# │ --ext TEXT file extensions to process, if d is specified, example ['.md', '.txt'] │
# │ --chef TEXT Chef method to use (e.g., text, markdown) │
# │ --chef-params TEXT Parameters for the chef as key=value pairs (e.g., --chef-params │
# │ clean_whitespace=true) │
# │ --chunker TEXT Chunking method to use [default: semantic] │
# │ --chunk-size INTEGER Maximum number of tokens per chunk │
# │ --chunk-overlap INTEGER Number of tokens to overlap between chunks │
# │ --threshold FLOAT Threshold for semantic similarity (0-1) │
# │ --chunker-params TEXT Additional parameters for the chunker as key=value pairs (e.g., --chunker-params │
# │ tokenizer=gpt2 min_characters_per_chunk=50) │
# │ --refiner TEXT Refiner method to use │
# │ --refiner-params TEXT Parameters for the refiner as key=value pairs (e.g., --refiner-params │
# │ context_size=50) │
# │ --handshaker TEXT Handshaker method to use │
# │ --handshaker-params TEXT Parameters for the handshaker as key=value pairs (e.g., --handshaker-params │
# │ collection_name=documents) │
# │ --help Show this message and exit. │
# ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
### Chunking Texts or Files
Use the `chunk` command to quickly chunk text or a single file.
**Syntax**:
```bash theme={"system"}
chonkie chunk [TEXT_OR_PATH] [OPTIONS]
```
**Options**:
* `--chunker`: The chunking method to use (default: `semantic`). Options: `semantic`, `token`, `sentence`, `recursive`, etc.
* `--chunk-size`: Maximum number of tokens per chunk (e.g., `512`, `1024`).
* `--chunk-overlap`: Number of tokens to overlap between chunks (e.g., `50`, `100`).
* `--threshold`: Threshold for semantic similarity (0-1), used by semantic chunkers.
* `--chunker-params`: Additional chunker parameters as `key=value` pairs. Can be used multiple times.
* `--handshaker`: Optional storage backend to export chunks.
**Examples**:
```bash theme={"system"}
# Chunk raw text with default settings
chonkie chunk "This is a long text that needs chunking..." --chunker token
# Chunk with explicit chunk size
chonkie chunk "Long text..." --chunker recursive --chunk-size 512
# Chunk with overlap
chonkie chunk document.txt --chunker token --chunk-size 1024 --chunk-overlap 100
# Chunk with semantic threshold
chonkie chunk document.txt --chunker semantic --threshold 0.8
# Chunk with additional parameters using key=value pairs
chonkie chunk document.txt \
--chunker recursive \
--chunk-size 512 \
--chunker-params min_characters_per_chunk=50 \
--chunker-params tokenizer=gpt2
# Chunk and store in a vector DB (e.g., Chroma)
chonkie chunk document.txt --handshaker chroma
```
***
### Running Pipelines
The `pipeline` command is more powerful and supports processing directories, applying chefs/refiners, and exporting data.
**Syntax**:
```bash theme={"system"}
chonkie pipeline [TEXT_OR_PATH] [OPTIONS]
```
**Core Options**:
* `--d`: Directory to process (mutually exclusive with text/file argument).
* `--ext`: File extensions to include when processing a directory (e.g., `.md`, `.txt`). Can be used multiple times.
* `--chef`: Preprocessor to use (e.g., `text`, `markdown`).
* `--chef-params`: Parameters for the chef as `key=value` pairs. Can be used multiple times.
* `--chunker`: Chunking method (default: `semantic`).
* `--chunk-size`: Maximum number of tokens per chunk.
* `--chunk-overlap`: Number of tokens to overlap between chunks.
* `--threshold`: Threshold for semantic similarity (0-1).
* `--chunker-params`: Additional chunker parameters as `key=value` pairs. Can be used multiple times.
* `--refiner`: Optional refinement strategy (e.g., `overlap`).
* `--refiner-params`: Parameters for the refiner as `key=value` pairs. Can be used multiple times.
* `--handshaker`: Optional destination storage.
* `--handshaker-params`: Parameters for the handshaker as `key=value` pairs. Can be used multiple times.
**Examples**:
#### 1. Process a Directory
Process all markdown and text files in the `docs` directory:
```bash theme={"system"}
chonkie pipeline --d docs --ext .md --ext .txt --chunker recursive
```
#### 2. Process a Single File
Run a pipeline on a single file:
```bash theme={"system"}
chonkie pipeline README.md --chunker token --chef text
```
#### 3. Pipeline with Custom Chunking Parameters
Use explicit parameters and additional chunker options:
```bash theme={"system"}
chonkie pipeline document.txt \
--chunker recursive \
--chunk-size 512 \
--chunker-params min_characters_per_chunk=50
```
#### 4. Pipeline with Multiple Component Parameters
Configure chef, chunker, and refiner with custom parameters:
```bash theme={"system"}
chonkie pipeline document.txt \
--chef text \
--chunker token \
--chunk-size 1024 \
--chunk-overlap 100 \
--refiner overlap \
--refiner-params context_size=50
```
#### 5. Full RAG Pipeline
Run a full RAG pipeline: fetch from directory -> process markdown -> chunk recursively -> export to ChromaDB.
```bash theme={"system"}
chonkie pipeline \
--d ./knowledge_base \
--ext .md \
--chef markdown \
--chunker recursive \
--chunk-size 512 \
--handshaker chroma \
--handshaker-params collection_name=documents
```
## Parameter Configuration
### Explicit Parameters
For commonly used parameters, you can use dedicated options:
* `--chunk-size`: Set the maximum tokens per chunk
* `--chunk-overlap`: Set overlap between chunks
* `--threshold`: Set semantic similarity threshold
### Key-Value Parameters
For additional or component-specific parameters, use the `*_params` options with `key=value` syntax:
```bash theme={"system"}
# Single parameter
--chunker-params tokenizer=gpt2
# Multiple parameters (repeat the option)
--chunker-params tokenizer=gpt2 --chunker-params min_characters_per_chunk=50
# Boolean parameters
--chunker-params verbose=true
# Numeric parameters (automatically converted)
--chunker-params chunk_size=512
--chunker-params threshold=0.8
```
**Type Conversion**: Parameters are automatically converted:
* `true`/`false` → boolean
* `none`/`null` → None
* Numeric strings → int or float
* Other strings → string
**Parameter Precedence**: Explicit options (like `--chunk-size`) override values in `--chunker-params` if both are provided.
## Tips
* Use `--help` on any command to see full options: `chonkie pipeline --help`.
* Directory processing recursively walks subdirectories.
* Output is printed to stdout by default unless a handshaker is specified.
* Combine explicit parameters with `*_params` for maximum flexibility.
* Check component documentation for available parameters for each chunker, chef, refiner, or handshaker.
# Code Chunker
Source: https://docs.chonkie.ai/oss/experimental/code-chunker
Advanced AST-based code chunking with intelligent semantic preservation
The experimental CodeChunker provides advanced AST-based code parsing that goes beyond simple line-based splitting to understand and preserve code structure and semantics.
**Experimental Feature**: This CodeChunker is experimental and may change significantly between versions. Use with caution in production environments.
## Key Features
* **AST-based parsing** using tree-sitter for accurate code understanding
* **Automatic language detection** using Magika for seamless multi-language handling
* **Language-specific rules** for optimal chunking based on programming language
* **Intelligent grouping** of related code elements (imports, comments, classes)
* **Semantic preservation** prioritizes code coherence over strict size limits
* **Multi-language support** for popular programming languages
* **Recursive splitting** for large code constructs when chunk size is specified
## Installation
To use the experimental CodeChunker, you need the code dependencies:
```bash theme={"system"}
pip install chonkie[code]
```
## Supported Languages
The experimental CodeChunker supports the following programming languages:
* **Python** - Classes, functions, imports, docstrings
* **TypeScript** - Functions, classes, interfaces, modules
* **JavaScript** - Functions, classes, modules, JSX
* **Rust** - Functions, structs, modules, traits
* **Go** - Functions, structs, packages, interfaces
* **Java** - Classes, methods, packages, interfaces
* **C** - Functions, structs, headers
* **C++** - Functions, classes, namespaces, structs
* **C#** - Classes, methods, namespaces, properties
* **HTML** - Tags, elements, attributes
* **CSS** - Rules, selectors, properties
* **Markdown** - Headers, sections, code blocks
## Basic Usage
```python theme={"system"}
from chonkie.experimental import CodeChunker
# Create a code chunker for Python
chunker = CodeChunker(language="python")
# Chunk some Python code
code = '''
import os
from typing import List
def process_files(directory: str) -> list[str]:
"""Process all files in a directory."""
files = []
for filename in os.listdir(directory):
if filename.endswith('.py'):
files.append(filename)
return files
class FileProcessor:
def __init__(self, base_dir: str):
self.base_dir = base_dir
self.processed_count = 0
def process(self, filename: str) -> bool:
"""Process a single file."""
# Processing logic here
self.processed_count += 1
return True
'''
chunks = chunker.chunk(code)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}:")
print(chunk.text)
print("---")
```
## Advanced Configuration
### With Chunk Size Limit
```python theme={"system"}
# Set a chunk size limit (chunks may exceed this to preserve semantics)
chunker = CodeChunker(
language="python",
chunk_size=2048, # Target chunk size in characters
tokenizer="character"
)
```
### Language Auto-Detection
The experimental CodeChunker can automatically detect the programming language using Magika, Google's deep learning-based language detection model:
```python theme={"system"}
# Let the chunker detect the language automatically
chunker = CodeChunker(language="auto")
# Chunk different types of code - language is detected automatically
python_code = '''
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
'''
javascript_code = '''
function fibonacci(n) {
if (n <= 1) return n;
return fibonacci(n-1) + fibonacci(n-2);
}
'''
rust_code = '''
fn fibonacci(n: u32) -> u32 {
if n <= 1 { n } else { fibonacci(n-1) + fibonacci(n-2) }
}
'''
# All will be chunked with appropriate language-specific rules
python_chunks = chunker.chunk(python_code) # Detected as Python
js_chunks = chunker.chunk(javascript_code) # Detected as JavaScript
rust_chunks = chunker.chunk(rust_code) # Detected as Rust
```
**Performance Consideration**: When using `language="auto"`, the chunker will show a warning that auto-detection may affect performance. For better performance in production, specify the language explicitly when known.
### Split Context Control
```python theme={"system"}
# Control whether to add split context information
chunker = CodeChunker(
language="typescript",
add_split_context=True # Include context about split locations
)
```
## Understanding Chunk Behavior
### Semantic Preservation
The experimental CodeChunker prioritizes semantic coherence over strict size limits:
```python theme={"system"}
chunker = CodeChunker(language="python", chunk_size=100)
# This class will likely stay together even if it exceeds 100 characters
code = '''
class SmallButImportant:
def __init__(self):
self.value = "important"
def get_value(self):
return self.value
'''
chunks = chunker.chunk(code)
# The class will typically be kept as one chunk for semantic coherence
```
### Language-Specific Grouping
Different languages have different grouping behaviors:
```python theme={"system"}
# Python code is grouped by logical units
python_code = '''
import numpy as np
import pandas as pd
def data_processor():
"""Process data using pandas."""
return pd.DataFrame()
class DataAnalyzer:
def analyze(self, data):
return np.mean(data)
'''
# Likely chunks:
# 1. Import statements together
# 2. Function definition
# 3. Class definition
```
```javascript theme={"system"}
// JavaScript/TypeScript grouping
const code = `
import { Component } from 'react';
import { useState } from 'react';
export const MyComponent = () => {
const [state, setState] = useState(null);
return
{state}
;
};
export class DataService {
async fetchData() {
return fetch('/api/data');
}
}
`;
// Likely chunks:
// 1. Import statements
// 2. Component definition
// 3. Class definition
```
```rust theme={"system"}
// Rust code grouping
let rust_code = r#"
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct User {
id: u32,
name: String,
}
impl User {
pub fn new(id: u32, name: String) -> Self {
Self { id, name }
}
}
"#;
// Likely chunks:
// 1. Use statements
// 2. Struct definition with derives
// 3. Implementation block
```
## Best Practices
### Choose Appropriate Chunk Sizes
```python theme={"system"}
# For code analysis tasks
chunker = CodeChunker(language="python", chunk_size=1024)
# For embedding generation (smaller chunks often work better)
chunker = CodeChunker(language="python", chunk_size=2048)
# No size limit (preserve all semantic units)
chunker = CodeChunker(language="python", chunk_size=None)
```
### Language-Specific Considerations
```python theme={"system"}
# For web development files with mixed content
html_chunker = CodeChunker(language="html", chunk_size=800)
# For documentation with code examples
md_chunker = CodeChunker(language="markdown", chunk_size=600)
# For system-level code that needs precise structure
c_chunker = CodeChunker(language="c", chunk_size=1200)
```
## Output Format
Each chunk contains detailed metadata about the code structure:
```python theme={"system"}
chunks = chunker.chunk(code)
for chunk in chunks:
print(f"Text: {chunk.text}")
print(f"Start: {chunk.start_index}")
print(f"End: {chunk.end_index}")
print(f"Token count: {chunk.token_count}")
```
## Limitations
**Current Limitations**:
* **Experimental status**: APIs may change between versions
* **Performance**: AST parsing may be slower than simple text splitting
* **Language support**: Not all programming languages are supported yet
* **Size flexibility**: Chunks may significantly exceed specified size limits
* **Dependencies**: Requires tree-sitter and language packs
## Migration from Stable CodeChunker
If migrating from the stable CodeChunker to the experimental version:
```python theme={"system"}
# Old stable version
from chonkie import CodeChunker
# New experimental version
from chonkie.experimental import CodeChunker
# The API is similar but with enhanced capabilities
chunker = CodeChunker(language="python", chunk_size=2048)
```
## Feedback and Support
Since this is an experimental feature, your feedback is valuable:
* **Report issues** on [GitHub](https://github.com/chonkie-inc/chonkie)
* **Share use cases** to help improve the chunker
* **Test with your code** and let us know what works well or needs improvement
The experimental CodeChunker will eventually replace or supplement the stable CodeChunker based on community feedback and testing results.
# Overview
Source: https://docs.chonkie.ai/oss/experimental/overview
Explore cutting-edge chunking capabilities with Chonkie's experimental features
# Experimental Features
Welcome to Chonkie's experimental features! This section contains advanced, cutting-edge functionality that's currently in development and testing phases.
**Experimental Notice**: Features in this section are experimental and may change significantly between versions. They are provided for early testing and feedback. Use with caution in production environments.
## What's Experimental?
Experimental features in Chonkie represent:
* **Advanced algorithms** that are still being refined
* **New chunking strategies** that may not be fully optimized
* **Innovative approaches** to text processing that need real-world validation
* **Features with evolving APIs** that may change based on user feedback
* **CLI improvements** allowing for directory processing and pipeline execution from the terminal
## Getting Started
To use experimental features, import them from the `chonkie.experimental` module:
```python theme={"system"}
from chonkie.experimental import CodeChunker
# Create an experimental chunker
chunker = CodeChunker(language="python", chunk_size=2048)
```
## Providing Feedback
Your feedback is crucial for graduating experimental features to stable status. If you encounter issues or have suggestions:
1. **Open an issue** on our [GitHub repository](https://github.com/chonkie-inc/chonkie)
2. **Join our Discord** to discuss with the community
3. **Share your use cases** to help us understand real-world applications
## Migration to Stable
When experimental features become stable, they will:
* Move to the main Chonkie namespace
* Receive API stability guarantees
* Include comprehensive documentation and examples
* Be covered by semantic versioning promises
We recommend using experimental features in development and testing environments first, and carefully evaluating their performance before production use.
# FileFetcher
Source: https://docs.chonkie.ai/oss/fetchers/file-fetcher
Fetch files from local filesystem for pipeline processing
The `FileFetcher` retrieves files from your local filesystem. It supports two modes: fetching a single file or fetching multiple files from a directory with optional extension filtering.
## Installation
FileFetcher is included with the base Chonkie installation:
```bash theme={"system"}
pip install chonkie
```
## Usage
### Single File Mode
Fetch a single file by providing the `path` parameter:
```python theme={"system"}
from chonkie.pipeline import Pipeline
# Fetch and process a single file
doc = (Pipeline()
.fetch_from("file", path="document.txt")
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
print(f"Chunked into {len(doc.chunks)} chunks")
```
### Directory Mode
Fetch multiple files from a directory using the `dir` parameter:
```python theme={"system"}
# Fetch all files from a directory
docs = (Pipeline()
.fetch_from("file", dir="./documents")
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
print(f"Processed {len(docs)} documents")
for doc in docs:
print(f" - {len(doc.chunks)} chunks")
```
### Extension Filtering
Filter files by extension when using directory mode:
```python theme={"system"}
# Fetch only .txt and .md files
docs = (Pipeline()
.fetch_from("file", dir="./documents", ext=[".txt", ".md"])
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
```
## Parameters
Path to a single file. Cannot be used with `dir`.
Directory to fetch files from. Cannot be used with `path`.
List of file extensions to filter (e.g., `[".txt", ".md"]`). Only used with `dir` parameter.
## Return Values
* **Single file mode** (`path` provided): Returns a single `Path` object
* **Directory mode** (`dir` provided): Returns `list[Path]` containing all matching files
## Standalone Usage
You can also use FileFetcher directly without the pipeline:
```python theme={"system"}
from chonkie import FileFetcher
fetcher = FileFetcher()
# Single file
file_path = fetcher.fetch(path="document.txt")
print(file_path) # PosixPath('document.txt')
# Directory with extension filter
files = fetcher.fetch(dir="./docs", ext=[".txt", ".md"])
for file in files:
print(file)
```
## Error Handling
FileFetcher validates inputs and provides clear error messages:
```python theme={"system"}
# FileNotFoundError if file doesn't exist
fetcher.fetch(path="nonexistent.txt") # Raises FileNotFoundError
# ValueError if both path and dir are provided
fetcher.fetch(path="file.txt", dir="./docs") # Raises ValueError
# ValueError if neither is provided
fetcher.fetch() # Raises ValueError
```
## Best Practices
When working with directories containing many files, always specify `ext` to avoid processing unwanted files:
```python theme={"system"}
# Good - only processes markdown files
.fetch_from("file", dir="./docs", ext=[".md"])
# Potentially slow - processes ALL files
.fetch_from("file", dir="./docs")
```
While relative paths work, absolute paths make your pipeline more portable:
```python theme={"system"}
from pathlib import Path
docs_dir = Path(__file__).parent / "documents"
.fetch_from("file", dir=str(docs_dir), ext=[".txt"])
```
## What's Next?
After fetching files, you'll typically want to:
1. **Process** them with a [Chef](/oss/chefs/overview) to parse content
2. **Chunk** them with a [Chunker](/oss/chunkers/overview) to split into manageable pieces
3. **Refine** chunks with [Refineries](/oss/refinery/overview) for better quality
See the [Pipeline Guide](/oss/pipelines) for complete examples.
# Fetchers Overview
Source: https://docs.chonkie.ai/oss/fetchers/overview
Overview of the different fetchers available in Chonkie
Fetchers connect different data sources to Chonkie's pipeline system, enabling seamless data ingestion from various sources.
## What are Fetchers?
Fetchers are the first step in the CHOMP pipeline (CHef -> CHunker -> Refinery -> Porter/Handshake). They retrieve data from different sources and pass it to the next pipeline stage for processing. Fetchers make it easy to:
* Load files from local storage
* Fetch documents from cloud storage (coming soon)
* Retrieve data from databases (coming soon)
* Connect to APIs and web sources (coming soon)
## Installation
Fetchers are included with the base Chonkie installation:
```bash theme={"system"}
pip install chonkie
```
## Using Fetchers in Pipelines
Fetchers integrate seamlessly with the Pipeline API:
```python theme={"system"}
from chonkie.pipeline import Pipeline
# Single file
doc = (Pipeline()
.fetch_from("file", path="document.txt")
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
# Directory with multiple files
docs = (Pipeline()
.fetch_from("file", dir="./docs", ext=[".txt", ".md"])
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
```
## Available Fetchers
Fetch files from local filesystem - single files or entire directories.
More fetchers are coming soon! We're working on cloud storage, database, and API fetchers.
# Chroma Handshake
Source: https://docs.chonkie.ai/oss/handshakes/chroma-handshake
Export Chonkie's Chunks into a Chroma collection.
The `ChromaHandshake` class provides seamless integration between Chonkie's chunking system and ChromaDB, a popular vector database.
Embed and store your Chonkie chunks in ChromaDB without ever leaving the Chonkie SDK.
## Installation
Before using the Chroma handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[chroma]
```
## Basic Usage
### Initialization
```python theme={"system"}
from chonkie import ChromaHandshake
# Initialize with default settings (in-memory ChromaDB)
handshake = ChromaHandshake()
# Or specify a persistent storage path
handshake = ChromaHandshake(path="./chroma_db")
# Or use an existing Chroma client
import chromadb
client = chromadb.Client()
handshake = ChromaHandshake(client=client, collection_name="my_collection")
```
### Writing Chunks to ChromaDB
```python theme={"system"}
from chonkie import ChromaHandshake, SemanticChunker
handshake = ChromaHandshake() # Initializes a new Chroma client
chunker = SemanticChunker()
chunks = chunker("Chonkie is the best chonker ever!")
handshake.write(chunks)
```
### Searching Chunks in ChromaDB
You can retrieve the most similar chunks from your ChromaDB collection using the `search` method:
```python search using a query theme={"system"}
from chonkie import ChromaHandshake
# Initialize the handshake
handshake = ChromaHandshake(collection_name="my_documents")
results = handshake.search(query="best chonker", limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using embedding theme={"system"}
from chonkie import ChromaHandshake
# Initialize the handshake
handshake = ChromaHandshake(collection_name="my_documents")
# Generate an embedding using the handshake's configured embedding function
embedding = handshake.embedding_function("best chonker").tolist()
results = handshake.search(embedding=embedding, limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using chonkie chunks theme={"system"}
from chonkie import ChromaHandshake, SemanticChunker
# Initialize the handshake
handshake = ChromaHandshake(collection_name="my_documents")
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie is the best chonker ever!")
# The chunker creates chunks but doesn't store their embeddings on the object.
# We can generate an embedding from a chunk's text to use for the search.
if chunks:
search_embedding = handshake.embedding_function(chunks.text).tolist()
# Search the handshake using the generated embedding
results = handshake.search(
embedding=search_embedding,
limit=2,
)
for result in results:
print(result["score"], result["text"])
```
## Parameters
Chroma client instance. If not provided, a new client will be created.
Name of the collection to use. If "random", a random name will be generated.
Embedding model to use.
If provided, creates a persistent Chroma client at the specified path.
# Elasticsearch Handshake
Source: https://docs.chonkie.ai/oss/handshakes/elastic-handshake
Export Chonkie's Chunks into an Elasticsearch index.
The `ElasticHandshake` class provides seamless integration between Chonkie's chunking system and Elasticsearch, allowing you to leverage its powerful vector search capabilities.
Embed and store your Chonkie chunks in an Elasticsearch index without ever leaving the Chonkie SDK. The handshake automatically handles index creation and the necessary vector field mapping.
## Installation
Before using the Elasticsearch handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[elastic]
```
## Basic Usage
### Initialization
```python Initialize for a Local Instance theme={"system"}
from chonkie import ElasticHandshake
# Connects to http://localhost:9200 by default
handshake = ElasticHandshake()
```
```python Initialize for Elastic Cloud theme={"system"}
from chonkie import ElasticHandshake
handshake = ElasticHandshake(
hosts="YOUR_CLOUD_ID",
api_key="YOUR_API_KEY",
index_name="my_document_index"
)
```
```python Initialize with an Existing Client theme={"system"}
from elasticsearch import Elasticsearch
from chonkie import ElasticHandshake
# Create and configure your client
client = Elasticsearch(
"https://your-domain.elastic-cloud.com:9243",
api_key="YOUR_API_KEY"
)
handshake = ElasticHandshake(client=client, index_name="my_document_index")
```
### Parameters
An existing `elasticsearch.Elasticsearch` client instance. If not provided, a new client will be created based on other parameters.
Name of the Elasticsearch index to use. If "random", a unique name will be generated.
The embedding model to use for creating vectors. Can be a model name from Hugging Face or a `BaseEmbeddings` instance.
The URL(s) of the Elasticsearch instance(s) to connect to.
The Cloud ID for connecting to an Elastic Cloud deployment.
The API key for authenticating with Elasticsearch, commonly used for Elastic Cloud.
### Writing Chunks to Elasticsearch
```python theme={"system"}
from chonkie import ElasticHandshake, SentenceChunker
# Initialize the handshake for your deployment
handshake = ElasticHandshake(
cloud_id="YOUR_CLOUD_ID",
api_key="YOUR_API_KEY",
index_name="my_documents",
)
# Create some chunks
chunker = SentenceChunker()
chunks = chunker.chunk("Chonkie uses the bulk API for efficient indexing. It's fast and reliable!")
# Write chunks to Elasticsearch
handshake.write(chunks)
```
### Searching Chunks in Elasticsearch
You can retrieve the most similar chunks from your Elasticsearch index using the `search` method, which performs a k-Nearest Neighbor (kNN) vector search.
```python Search using a Text Query theme={"system"}
from chonkie import ElasticHandshake
# Initialize the handshake to connect to your index
handshake = ElasticHandshake(
hosts="YOUR_CLOUD_ID",
api_key="YOUR_API_KEY",
index_name="my_documents",
)
results = handshake.search(query="fast and efficient indexing", limit=2)
```
```python Search using an Embedding Vector theme={"system"}
from chonkie import ElasticHandshake
# Initialize the handshake
handshake = ElasticHandshake(
hosts="YOUR_CLOUD_ID",
api_key="YOUR_API_KEY",
index_name="my_documents",
)
# Generate an embedding vector for your query
embedding = handshake.embedding_model.embed("fast and efficient indexing").tolist()
results = handshake.search(embedding=embedding, limit=2)
```
# LanceDB Handshake
Source: https://docs.chonkie.ai/oss/handshakes/lancedb-handshake
Export Chonkie's Chunks into a LanceDB table.
The `LanceDBHandshake` class provides seamless integration between Chonkie's chunking system and LanceDB, a serverless vector database built on Apache Arrow.
Embed and store your Chonkie chunks in LanceDB — locally or in the cloud — without ever leaving the Chonkie SDK.
## Installation
Before using the LanceDB handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[lancedb]
```
## Basic Usage
### Initialization
```python in-memory (ephemeral) theme={"system"}
from chonkie import LanceDBHandshake
# Default: in-memory LanceDB, auto-generated table name
handshake = LanceDBHandshake()
```
```python local persistent storage theme={"system"}
from chonkie import LanceDBHandshake
# Persist to a local directory
handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_chunks")
```
```python existing connection theme={"system"}
import lancedb
from chonkie import LanceDBHandshake
conn = lancedb.connect("./my_lancedb")
handshake = LanceDBHandshake(connection=conn, table_name="my_chunks")
```
```python LanceDB Cloud theme={"system"}
from chonkie import LanceDBHandshake
handshake = LanceDBHandshake(
uri="db://my-project",
table_name="my_chunks",
)
```
### Writing Chunks to LanceDB
```python theme={"system"}
from chonkie import LanceDBHandshake, SemanticChunker
# Initialize the handshake
handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_documents")
# Create some chunks
chunker = SemanticChunker()
chunks = chunker("Chonkie loves to chonk your texts!")
# Write chunks to LanceDB
handshake.write(chunks)
```
### Searching Chunks in LanceDB
You can retrieve the most similar chunks from your LanceDB table using the `search` method:
```python search using a query theme={"system"}
from chonkie import LanceDBHandshake
handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_documents")
results = handshake.search(query="chonk your texts", limit=5)
for result in results:
print(result["score"], result["text"])
```
```python search using an embedding theme={"system"}
from chonkie import LanceDBHandshake
handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_documents")
embedding = handshake.embedding_model.embed("chonk your texts").tolist()
results = handshake.search(embedding=embedding, limit=5)
for result in results:
print(result["score"], result["text"])
```
## Parameters
An existing LanceDB connection. If not provided, a new connection is created using `uri`.
URI of the LanceDB database. Use `"memory://"` for an ephemeral in-memory database, a local directory path for persistent storage, or a `db://` URI for LanceDB Cloud.
Name of the table to write chunks to. If `"random"`, a unique name is auto-generated.
Embedding model to use. Can be a model name string or a `BaseEmbeddings` instance.
# Milvus Handshake
Source: https://docs.chonkie.ai/oss/handshakes/milvus-handshake
Export Chonkie's Chunks into a Milvus collection.
The `MilvusHandshake` class provides seamless integration between Chonkie's chunking system and Milvus, a powerful, open-source vector database.
Embed and store your Chonkie chunks in a Milvus collection, with automatic schema and index creation, without ever leaving the Chonkie SDK.
## Installation
Before using the Milvus handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[milvus]
```
## Basic Usage
### Initialization
```python Initialize for a Local Instance theme={"system"}
from chonkie import MilvusHandshake
# Connects to Milvus at http://localhost:19530 by default
handshake = MilvusHandshake()
```
```python Initialize using a URI theme={"system"}
from chonkie import MilvusHandshake
# Recommended for connecting to remote or secured instances
handshake = MilvusHandshake(
uri=os.getenv("MILVUS_URI"),
user=os.getenv("MILVUS_USER"),
api_key=os.getenv("MILVUS_API_KEY"),
collection_name="test_collection",
)
```
### Parameters
The name of the Milvus collection to use. If "random", a unique name is generated.
The embedding model to use for creating vectors.
The full URI to connect to Milvus. This is the preferred method for specifying connection details.
The host of the Milvus instance. Used if `uri` is not provided.
The port of the Milvus instance. Used if `uri` is not provided.
The connection alias to use for this Milvus connection.
### Writing Chunks to Milvus
```python theme={"system"}
from chonkie import MilvusHandshake, SentenceChunker
# Initialize the handshake for your deployment
handshake = MilvusHandshake(
uri="http://localhost:19530",
collection_name="my_documents",
)
# Create some chunks
chunker = SentenceChunker()
chunks = chunker.chunk("Milvus stores data in collections. Chonkie makes ingestion easy!")
# Write chunks to the Milvus collection
handshake.write(chunks)
```
### Searching Chunks in Milvus
You can retrieve the most similar chunks from your Milvus collection using the `search` method.
```python Search using a Text Query theme={"system"}
from chonkie import MilvusHandshake
# Initialize the handshake to connect to your collection
handshake = MilvusHandshake(
uri="http://localhost:19530",
collection_name="my_documents",
)
results = handshake.search(query="easy data ingestion", limit=2)
```
```python Search using an Embedding Vector theme={"system"}
from chonkie import MilvusHandshake
# Initialize the handshake
handshake = MilvusHandshake(
uri="http://localhost:19530",
collection_name="my_documents",
)
# Generate an embedding vector for your query
embedding = handshake.embedding_model.embed("easy data ingestion")
results = handshake.search(embedding=embedding, limit=2)
```
# MongoDB Handshake
Source: https://docs.chonkie.ai/oss/handshakes/mongodb-handshake
Export Chonkie's Chunks into a MongoDB collection.
The `MongoDBHandshake` class provides integration between Chonkie's chunking system and MongoDB, a popular NoSQL database.
Embed and store your Chonkie chunks in MongoDB directly from the Chonkie SDK.
## Installation
Before using the MongoDB handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[mongodb]
```
## Initialization
```python Initialize using chonkie theme={"system"}
from chonkie import MongoDBHandshake
# Initialize with default settings (auto-generated database/collection)
handshake = MongoDBHandshake(uri="mongodb://localhost:27017")
```
```python initialize using the client theme={"system"}
import pymongo
client = pymongo.MongoClient("mongodb://localhost:27017")
handshake = MongoDBHandshake(
client=client,
db_name="my_db",
collection_name="my_collection"
)
```
```python specify connection and embedding theme={"system"}
handshake = MongoDBHandshake(
uri="mongodb://localhost:27017",
db_name="my_db",
collection_name="my_collection",
embedding_model="minishlab/potion-retrieval-32M"
)
```
### Parameters
MongoDB client instance. If not provided, a new client will be created based on other parameters.
MongoDB connection URI.
MongoDB username for authentication.
MongoDB password for authentication.
MongoDB host address.
MongoDB port number.
Name of the database to use. If "random", a unique name will be generated.
Name of the collection to use. If "random", a unique name will be generated.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
Additional keyword arguments to pass to the MongoDB client or collection creation.
## Writing Chunks to MongoDB
```python theme={"system"}
from chonkie import MongoDBHandshake, SemanticChunker
# Initialize the handshake
handshake = MongoDBHandshake(
uri="mongodb://localhost:27017",
db_name="my_documents",
collection_name="my_collection"
)
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to MongoDB
handshake.write(chunks)
```
## Searching Chunks in MongoDB
You can retrieve the most similar chunks from your MongoDB collection using the `search` method:
```python search using a query theme={"system"}
from chonkie import MongoDBHandshake
# Initialize the handshake
handshake = MongoDBHandshake(
uri="mongodb://localhost:27017",
db_name="my_documents",
collection_name="my_collection"
)
results = handshake.search(query="chonk your texts", limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using embedding theme={"system"}
from chonkie import MongoDBHandshake
# Initialize the handshake
handshake = MongoDBHandshake(
uri="mongodb://localhost:27017",
db_name="my_documents",
collection_name="my_collection"
)
embedding = handshake.embedding_model.embed("chonk your texts").tolist()
results = handshake.search(embedding=embedding, limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using chonkie chunks theme={"system"}
from chonkie import MongoDBHandshake, SemanticChunker
# Initialize the handshake
handshake = MongoDBHandshake(
uri="mongodb://localhost:27017",
db_name="my_documents",
collection_name="my_collection"
)
# Create some chunks
chunker = SemanticChunker(embedding_model=handshake.embedding_model)
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Search the handshake
results = handshake.search(
embedding=chunks[0].sentences[0].embedding,
limit=2,
)
for result in results:
print(result["score"], result["text"])
```
# Handshakes Overview
Source: https://docs.chonkie.ai/oss/handshakes/overview
Overview of the different handshakes available in Chonkie
Handshakes allow you to easily connect Chonkie to a vector database of your choice.
Embed your chunks and write them to your database in just a few lines of code.
Connect Chonkie to your ephemeral or persistent ChromaDB instance
Connect Chonkie to your Elasticsearch index
Connect Chonkie to your local or cloud LanceDB table
Connect Chonkie to your Milvus collection
Connect Chonkie to your MongoDB collection
Connect Chonkie to your Pgvector database
Connect Chonkie to your Pinecone index
Connect Chonkie to your Qdrant database
Connect Chonkie to your Turbopuffer database
Connect Chonkie to your Weaviate database
# Pgvector Handshake
Source: https://docs.chonkie.ai/oss/handshakes/pgvector-handshake
Export Chonkie's Chunks into a PostgreSQL database with pgvector.
The `PgvectorHandshake` class provides seamless integration between Chonkie's chunking system and PostgreSQL with pgvector. It uses the vecs client library from Supabase underneath to provide a higher-level API with automatic indexing, metadata filtering, and simplified connection management.
Store your Chonkie chunks in PostgreSQL with vector embeddings and perform semantic search without ever leaving the Chonkie SDK.
## Installation
Before using the Pgvector handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[pgvector]
```
You'll also need PostgreSQL with the pgvector extension installed:
```sql theme={"system"}
-- Connect to your database and enable pgvector
CREATE EXTENSION IF NOT EXISTS vector;
```
## Initialization
```python theme={"system"}
from chonkie import PgvectorHandshake
# Initialize with individual connection parameters
handshake = PgvectorHandshake(
host="localhost",
port=5432,
database="your_database",
user="your_user",
password="your_password",
collection_name="chonkie_chunks"
)
# Or use a connection string
handshake = PgvectorHandshake(
connection_string="postgresql://user:password@localhost:5432/database"
)
# Or use an existing vecs client
import vecs
client = vecs.create_client("postgresql://user:password@localhost:5432/database")
handshake = PgvectorHandshake(client=client, collection_name="my_collection")
```
## Usage
```python theme={"system"}
from chonkie import PgvectorHandshake, RecursiveChunker
# Initialize the handshake
handshake = PgvectorHandshake(
host="localhost",
database="my_database",
user="my_user",
password="my_password"
)
# Create some chunks
chunker = RecursiveChunker(chunk_size=2048)
chunks = chunker.chunk("Chonkie makes PostgreSQL vector search easy!")
# Write chunks to PostgreSQL
handshake.write(chunks)
```
```python theme={"system"}
# Search for similar chunks
results = handshake.search(
query="PostgreSQL vector search",
limit=5
)
for result in results:
print(f"Text: {result['text']}")
print(f"Similarity: {result['similarity']:.3f}")
print("---")
```
```python theme={"system"}
# Create an HNSW index for better performance
handshake.create_index(method="hnsw")
# Or create with custom parameters
handshake.create_index(
method="hnsw",
m=16,
ef_construction=64
)
```
## Parameters
An existing vecs.Client instance. If provided, other connection parameters are ignored.
PostgreSQL host address.
PostgreSQL port number.
PostgreSQL database name.
PostgreSQL username.
PostgreSQL password.
Full PostgreSQL connection string. If provided, individual connection parameters are ignored.
Name of the collection to store chunks in.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
Number of dimensions for the vector embeddings. If not provided, will be inferred from the embedding model.
# Pinecone Handshake
Source: https://docs.chonkie.ai/oss/handshakes/pinecone-handshake
Export Chonkie's Chunks into a Pinecone index.
The `PineconeHandshake` class provides seamless integration between Chonkie's chunking system and Pinecone, a managed vector database.
Embed and store your Chonkie chunks in Pinecone directly from the Chonkie SDK.
## Installation
Before using the Pinecone handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[pinecone]
```
## Initialization
```python Initialize using chonkie theme={"system"}
from chonkie import PineconeHandshake
handshake = PineconeHandshake(api_key="YOUR_API_KEY")
```
```python initialize using the client theme={"system"}
import pinecone
client = pinecone.Pinecone(api_key="YOUR_API_KEY")
handshake = PineconeHandshake(client=client, index_name="my_index")
```
```python specify embedding model theme={"system"}
handshake = PineconeHandshake(
api_key="YOUR_API_KEY",
index_name="my_index",
embedding_model="minishlab/potion-retrieval-32M",
)
```
### Parameters
Pinecone client instance. If not provided, a new client will be created based on other parameters.
Pinecone API key for authentication.
Name of the index to use. If "random", a unique name will be generated.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
Dimension of the embeddings. If not provided, will be inferred from the embedding model.
Additional keyword arguments to pass to the Pinecone client or index creation.
## Writing Chunks to Pinecone
```python theme={"system"}
from chonkie import PineconeHandshake, SemanticChunker
# Initialize the handshake
handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents")
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to Pinecone
handshake.write(chunks)
```
## Searching Chunks in Pinecone
You can retrieve the most similar chunks from your Pinecone index using the `search` method:
```python search using a query theme={"system"}
from chonkie import PineconeHandshake
# Initialize the handshake
handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents")
results = handshake.search(query="chonk your texts", limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using embedding theme={"system"}
from chonkie import PineconeHandshake
# Initialize the handshake
handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents")
embedding = handshake.embedding_model.embed("chonk your texts").tolist()
results = handshake.search(embedding=embedding, limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using chonkie chunks theme={"system"}
from chonkie import PineconeHandshake, SemanticChunker
# Initialize the handshake
handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents")
# Create some chunks
chunker = SemanticChunker(embedding_model=handshake.embedding_model)
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Search the handshake
results = handshake.search(
embedding=chunks[0].sentences[0].embedding,
limit=2,
)
for result in results:
print(result["score"], result["text"])
```
# Qdrant Handshake
Source: https://docs.chonkie.ai/oss/handshakes/qdrant-handshake
Export Chonkie's Chunks into a Qdrant collection.
The `QdrantHandshake` class provides seamless integration between Chonkie's chunking system and Qdrant, a high-performance vector database.
Embed and store your Chonkie chunks in Qdrant without ever leaving the Chonkie SDK.
## Installation
Before using the Qdrant handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[qdrant]
```
## Basic Usage
### Initialization
```python Initialize using chonkie theme={"system"}
from chonkie import QdrantHandshake
handshake = QdrantHandshake(url="http://localhost:6333")
```
```python initialize using the client theme={"system"}
from qdrant_client import QdrantClient
client = QdrantClient(":memory:")
handshake = QdrantHandshake(client=client, collection_name="my_collection")
```
```python qdrant cloud Initialization theme={"system"}
from qdrant_client import QdrantClient
handshake = QdrantHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
)
```
# Parameters
Qdrant client instance. If not provided, a new client will be created based on other parameters.
Name of the collection to use. If "random", a unique name will be generated.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
URL of the Qdrant server. If provided, will connect to this server.
If provided, creates a persistent Qdrant client at the specified path.
API key for Qdrant Cloud authentication.
### Writing Chunks to Qdrant
```python theme={"system"}
from chonkie import QdrantHandshake, SemanticChunker
# Initialize the handshake
handshake = QdrantHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents",
)
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to Qdrant
handshake.write(chunks)
```
### Searching Chunks in Qdrant
You can retrieve the most similar chunks from your Qdrant collection using the `search` method:
```python search using a query theme={"system"}
from chonkie import QdrantHandshake
# Initialize the handshake
handshake = QdrantHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents",
)
results = handshake.search(query="chonk your texts", limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using embedding theme={"system"}
from chonkie import QdrantHandshake
# Initialize the handshake
handshake = QdrantHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents",
)
embedding = handshake.embedding_model.embed("chonk your texts").tolist()
results = handshake.search(embedding=embedding, limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using chonkie chunks theme={"system"}
from chonkie import QdrantHandshake, SemanticChunker
# Initialize the handshake
handshake = QdrantHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents",
)
# Create some chunks
chunker = SemanticChunker(embedding_model=handshake.embedding_model)
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Search the handshake
results = handshake.search(
embedding=chunks[0].sentences[0].embedding,
limit=2,
)
for result in results:
print(result["score"], result["text"])
```
# Turbopuffer Handshake
Source: https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake
Export Chonkie's Chunks into a Turbopuffer database.
The `TurbopufferHandshake` class provides seamless integration between Chonkie's chunking system and Turbopuffer, a high-performance vector database.
Embed and store your Chonkie chunks in Turbopuffer without ever leaving the Chonkie SDK.
The Turbopuffer Handshake requires a Turbopuffer API key. You can get one by signing up for a [Turbopuffer account](https://turbopuffer.com).
## Installation
Before using the Turbopuffer handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[turbopuffer]
```
## Basic Usage
### Initialization
```python theme={"system"}
from chonkie import TurbopufferHandshake
# Initialize with default settings (requires TURBOPUFFER_API_KEY environment variable)
handshake = TurbopufferHandshake()
# Or provide an API key directly
handshake = TurbopufferHandshake(api_key="your_turbopuffer_api_key")
# Use a specific namespace
handshake = TurbopufferHandshake(namespace_name="my_documents")
# Or use an existing Turbopuffer namespace
import turbopuffer as tpuf
ns = tpuf.Namespace("existing_namespace")
handshake = TurbopufferHandshake(namespace=ns)
```
### Writing Chunks to Turbopuffer
```python theme={"system"}
from chonkie import TurbopufferHandshake, SemanticChunker
handshake = TurbopufferHandshake(namespace_name="my_documents")
chunker = SemanticChunker()
chunks = chunker("Chonkie chunks, turbopuffer puffs!")
handshake.write(chunks)
```
## Parameters
An existing Turbopuffer Namespace instance to use. If not provided, a new namespace will be created.
Name of the namespace to use. If "random", a unique name will be generated.
Only used if `namespace` parameter is not provided.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
Turbopuffer API key. If not provided, will look for TURBOPUFFER\_API\_KEY environment variable.
## Authentication
You can authenticate with Turbopuffer in one of two ways:
1. **Environment Variable** (Recommended for development):
```bash theme={"system"}
export TURBOPUFFER_API_KEY='your-api-key-here'
```
2. **Directly in code** (Not recommended for production):
```python theme={"system"}
handshake = TurbopufferHandshake(api_key="your-api-key-here")
```
For production environments, it's recommended to use environment variables or a secure secret management system to handle your API keys.
# Weaviate Handshake
Source: https://docs.chonkie.ai/oss/handshakes/weaviate-handshake
Export Chonkie's Chunks into a Weaviate collection.
The `WeaviateHandshake` class provides seamless integration between Chonkie's chunking system and [Weaviate](https://weaviate.io/), a powerful vector database.
Embed and store your Chonkie chunks in Weaviate without ever leaving the Chonkie SDK.
## Installation
Before using the Weaviate handshake, make sure to install the required dependencies:
```bash theme={"system"}
pip install chonkie[weaviate]
```
## Basic Usage
### Initialization
```python Initialize using chonkie theme={"system"}
from chonkie import WeaviateHandshake
# Initialize with default settings (local Weaviate)
handshake = WeaviateHandshake()
# Or connect to a Weaviate server
handshake = WeaviateHandshake(url="http://localhost:8080", api_key= "YOUR_API_KEY")
```
```python initialize using the client theme={"system"}
import weaviate
from chonkie import WeaviateHandshake
client = weaviate.connect_to_local()
handshake = WeaviateHandshake(client=client, collection_name="my_collection")
```
```python weaviate cloud initialization theme={"system"}
from chonkie import WeaviateHandshake
handshake = WeaviateHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY"
)
```
## Parameters
Weaviate client instance. If not provided, a new client will be created based on other parameters.
Name of the collection to use. If "random", a unique name will be generated.
Embedding model to use. Can be a model name or a BaseEmbeddings instance.
URL of the Weaviate server. If provided, will connect to this server.
API key for Weaviate Cloud authentication.
OAuth configuration for authentication (optional).
Batch size for batch operations.
Whether to use dynamic batching.
Number of retries for batch timeouts.
Additional headers for the Weaviate client.
## Writing Chunks to Weaviate
```python theme={"system"}
from chonkie import WeaviateHandshake, SemanticChunker
# Initialize the handshake
handshake = WeaviateHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents"
)
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to Weaviate
handshake.write(chunks)
```
### Searching Chunks in Weaviate
You can retrieve the most similar chunks from your Weaviate collection using the `search` method:
```python search using a query theme={"system"}
from chonkie import WeaviateHandshake
# Initialize the handshake
handshake = WeaviateHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents"
)
results = handshake.search(query="chonk your texts", limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using embedding theme={"system"}
from chonkie import WeaviateHandshake
# Initialize the handshake
handshake = WeaviateHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents"
)
embedding = handshake.embedding_model.embed("chonk your texts").tolist()
results = handshake.search(embedding=embedding, limit=2)
for result in results:
print(result["score"], result["text"])
```
```python search using chonkie chunks theme={"system"}
from chonkie import WeaviateHandshake, SemanticChunker
# Initialize the handshake
handshake = WeaviateHandshake(
url="YOUR_CLOUD_URL",
api_key="YOUR_API_KEY",
collection_name="my_documents"
)
# Create some chunks
chunker = SemanticChunker(embedding_model=handshake.embedding_model)
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Search the handshake
results = handshake.search(
embedding=chunks[0].sentences[0].embedding,
limit=2,
)
for result in results:
print(result["score"], result["text"])
```
# Installation
Source: https://docs.chonkie.ai/oss/installation
Installing Chonkie and its various components
Chonkie follows a modular approach to dependencies, keeping the base installation lightweight while allowing you to add extra features as needed.
## Basic Installation
For basic token and sentence chunking capabilities.
### Python
```bash pip theme={"system"}
pip install chonkie
```
```bash uv theme={"system"}
uv add chonkie
```
This installs our basic chunkers, plus the Python API SDK.
To use advanced features locally, skip ahead to [Installation Options](#installation-options)
### JavaScript
Install the core package for local chunking
```bash npm theme={"system"}
npm install @chonkiejs/core
```
```bash pnpm theme={"system"}
pnpm add @chonkiejs/core
```
```bash bun theme={"system"}
bun add @chonkiejs/core
```
```bash yarn theme={"system"}
yarn add @chonkiejs/core
```
To use custom tokenizers, install the `@chonkiejs/token` package
```bash npm theme={"system"}
npm install @chonkiejs/token
```
```bash pnpm theme={"system"}
pnpm add @chonkiejs/token
```
```bash bun theme={"system"}
bun add @chonkiejs/token
```
```bash yarn theme={"system"}
yarn add @chonkiejs/token
```
To use the API, install the `@chonkiejs/cloud` package
```bash npm theme={"system"}
npm install @chonkiejs/cloud
```
```bash pnpm theme={"system"}
pnpm add @chonkiejs/cloud
```
```bash bun theme={"system"}
bun add @chonkiejs/cloud
```
```bash yarn theme={"system"}
yarn add @chonkiejs/cloud
```
## Installation Options
Chonkie provides several installation options to match your specific needs:
```bash Python theme={"system"}
# Basic installation (TokenChunker, SentenceChunker, RecursiveChunker)
pip install chonkie
# For Hugging Face Hub support
pip install "chonkie[hub]"
# For visualization support (e.g., rich text output)
pip install "chonkie[viz]"
# For the default semantic provider support (includes Model2Vec)
pip install "chonkie[semantic]"
# For OpenAI embeddings support
pip install "chonkie[openai]"
# For Cohere embeddings support
pip install "chonkie[cohere]"
# For Jina embeddings support
pip install "chonkie[jina]"
# For SentenceTransformer embeddings support (required by LateChunker)
pip install "chonkie[st]"
# For CodeChunker support
pip install "chonkie[code]"
# For NeuralChunker support (BERT-based)
pip install "chonkie[neural]"
# For SlumberChunker support (Genie/LLM interface)
pip install "chonkie[genie]"
# For Groq Genie support (fast inference)
pip install "chonkie[groq]"
# For Cerebras Genie support (fastest inference)
pip install "chonkie[cerebras]"
# For installing multiple features together
pip install "chonkie[st, code, genie]"
# For all features
pip install "chonkie[all]"
```
```bash JavaScript theme={"system"}
# Basic installation for local chunking
npm install @chonkiejs/core
# To use the API
npm install @chonkiejs/cloud
```
## Chunker Availability
The following table shows which chunkers are available with different installation options:
| Chunker | Default | embeddings | 'all' | Chonkie JS | API |
| ---------------- | :-------------------: | :-------------------: | :-------------------: | :-------------------: | :-------------------: |
| TokenChunker | | | | | |
| FastChunker | | | | | |
| RecursiveChunker | | | | | |
| SentenceChunker | | | | | |
| TableChunker | | | | | |
| SemanticChunker | | | | | |
| LateChunker | | | | | |
| CodeChunker | | | | | |
| NeuralChunker | | | | | |
| SlumberChunker | | | | | |
## Embeddings Availability
Different embedding providers are available with different installation options:
| Embeddings Provider | Default | 'model2vec' | 'st' | 'openai' | 'semantic' | 'all' |
| ----------------------------- | :---------------: | :-------------------: | :-------------------: | :-------------------: | :-------------------: | :-------------------: |
| Model2VecEmbeddings | | | | | | |
| SentenceTransformerEmbeddings | | | | | | |
| OpenAIEmbeddings | | | | | | |
## Dependencies
Here's what each installation option adds:
| Installation Option | Additional Dependencies |
| ------------------- | ------------------------------------------------ |
| Default | tqdm, numpy, chonkie-core, tenacity |
| 'hub' | + huggingface-hub, jsonschema |
| 'viz' | + rich |
| 'model2vec' | + tokenizers, model2vec, numpy |
| 'st' | + tokenizers, sentence-transformers, accelerate |
| 'openai' | + openai, tiktoken, pydantic |
| 'cohere' | + tokenizers, cohere |
| 'jina' | + tokenizers |
| 'semantic' | + tokenizers, model2vec |
| 'code' | + tree-sitter, tree-sitter-language-pack, magika |
| 'neural' | + transformers, torch |
| 'genie' | + pydantic, google-genai |
| 'groq' | + pydantic, groq |
| 'cerebras' | + pydantic, cerebras-cloud-sdk |
| 'litellm' | + litellm, tiktoken, tokenizers |
| 'all' | all above dependencies |
## Important Notes
* We provide separate `semantic` and `all` installs pre-packaged that might match other installation options breeding redundancy. This redundancy is intentional to provide users with the best experience and freedom to choose their preferred means.
* The `semantic` and `all` optional installs may change in future versions, so what you download today may not be the same for tomorrow.
* Installing either 'semantic' or 'openai' extras will enable SemanticChunker, as it can work with any embeddings provider. The difference is in which embedding providers are available for use with this chunker.
## Logging
Chonkie logs warnings and errors by default. Control logging with the `CHONKIE_LOG` environment variable:
```bash theme={"system"}
export CHONKIE_LOG=off # Disable logging
export CHONKIE_LOG=warning # Warnings and errors (default)
export CHONKIE_LOG=info # More verbose
export CHONKIE_LOG=debug # Everything
```
See [Logging](/oss/utils/logging) for more details.
# Building Pipelines
Source: https://docs.chonkie.ai/oss/pipelines
Build powerful text processing workflows with Chonkie's Pipeline API
Chonkie's Pipeline API provides a fluent, chainable interface for building text processing workflows. Pipelines follow the **CHOMP architecture**, automatically orchestrating components in the correct order.
## What is CHOMP?
CHOMP (CHOnkie's Multi-step Pipeline) is our standardized architecture for document processing:
```
Fetcher → Chef → Chunker → Refinery → Porter/Handshake
```
Retrieve raw data from files, APIs, or databases
Preprocess and transform raw data into Documents
Split documents into manageable chunks
Post-process and enhance chunks
Export or store chunks
Pipelines automatically reorder components to follow CHOMP, so you can add them in any order.
## Quick Start
### Single File Processing
```python theme={"system"}
from chonkie import Pipeline
# Build and execute pipeline
doc = (Pipeline()
.fetch_from("file", path="document.txt")
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
# Access chunks
print(f"Created {len(doc.chunks)} chunks")
for chunk in doc.chunks:
print(f"Chunk: {chunk.text[:50]}...")
```
### Directory Processing
Process multiple files at once:
```python theme={"system"}
# Process all markdown files in a directory
docs = (Pipeline()
.fetch_from("file", dir="./documents", ext=[".md", ".txt"])
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
# Process each document
for doc in docs:
print(f"Document has {len(doc.chunks)} chunks")
```
### Direct Text Input
Skip the fetcher and provide text directly:
```python theme={"system"}
# No fetcher needed
doc = (Pipeline()
.process_with("text")
.chunk_with("semantic", threshold=0.8)
.run(texts="Your text here"))
# Multiple texts
docs = (Pipeline()
.chunk_with("recursive", chunk_size=512)
.run(texts=["Text 1", "Text 2", "Text 3"]))
```
### Asynchronous Execution
For high-throughput applications (e.g., web servers, batch processing), use `arun()`:
```python theme={"system"}
import asyncio
async def process_docs():
pipe = Pipeline().chunk_with("recursive")
# Run pipeline asynchronously
doc = await pipe.arun(texts="Async processing is fast!")
# Process multiple concurrently
docs = await pipe.arun(texts=["Doc 1", "Doc 2"])
return docs
```
## Pipeline Methods
### fetch\_from()
Fetch data from a source:
```python theme={"system"}
# Single file
.fetch_from("file", path="document.txt")
# Directory with extension filter
.fetch_from("file", dir="./docs", ext=[".txt", ".md"])
```
### process\_with()
Process data with a chef:
```python theme={"system"}
# Text processing
.process_with("text")
# Markdown processing
.process_with("markdown")
# Table processing
.process_with("table")
```
### chunk\_with()
Chunk documents (required):
```python theme={"system"}
# Recursive chunking
.chunk_with("recursive", chunk_size=512)
# Semantic chunking
.chunk_with("semantic", threshold=0.8, chunk_size=1024)
# Code chunking
.chunk_with("code", chunk_size=512)
```
### refine\_with()
Refine chunks (optional, can chain multiple):
```python theme={"system"}
# Add overlap context
.refine_with("overlap", context_size=100, method="prefix")
# Add embeddings
.refine_with("embedding", model="text-embedding-3-small")
```
### export\_with()
Export chunks to formats (optional):
```python theme={"system"}
# Export to JSON
.export_with("json", file="chunks.json")
# Export to Hugging Face Datasets
.export_with("datasets", name="my-dataset")
```
### store\_in()
Store in vector databases (optional):
```python theme={"system"}
# Store in Chroma
.store_in("chroma", collection_name="documents")
# Store in Qdrant
.store_in("qdrant", collection_name="docs", url="http://localhost:6333")
```
## Advanced Examples
### RAG Knowledge Base
Build a complete RAG ingestion pipeline:
```python theme={"system"}
# Ingest documents into vector database
docs = (Pipeline()
.fetch_from("file", dir="./knowledge_base", ext=[".txt", ".md"])
.process_with("text")
.chunk_with("semantic", threshold=0.8, chunk_size=1024)
.refine_with("overlap", context_size=100)
.store_in("qdrant",
collection_name="knowledge",
url="http://localhost:6333")
.run())
print(f"Ingested {len(docs)} documents")
```
### Semantic Search Pipeline
Process documents with embeddings for search:
```python theme={"system"}
# Chunk with embeddings
doc = (Pipeline()
.fetch_from("file", path="research_paper.txt")
.process_with("text")
.chunk_with("semantic",
threshold=0.8,
chunk_size=1024,
similarity_window=3)
.refine_with("overlap", context_size=100)
.refine_with("embedding", model="minishlab/potion-base-32M")
.run())
# All chunks now have embeddings
for chunk in doc.chunks:
if chunk.embedding is not None:
print(f"Chunk: {chunk.text[:30]}... | Embedding shape: {chunk.embedding.shape}")
```
### Code Documentation
Process code with specialized chunking:
```python theme={"system"}
# Chunk Python files
docs = (Pipeline()
.fetch_from("file", dir="./src", ext=[".py"])
.chunk_with("code", chunk_size=512)
.export_with("json", file="code_chunks.json")
.run())
print(f"Processed {len(docs)} Python files")
```
### Markdown Processing
Handle markdown with table and code awareness:
```python theme={"system"}
# Process markdown documentation
doc = (Pipeline()
.fetch_from("file", path="README.md")
.process_with("markdown")
.chunk_with("recursive", chunk_size=512)
.run())
# Access markdown metadata
print(f"Found {len(doc.tables)} tables")
print(f"Found {len(doc.code)} code blocks")
print(f"Created {len(doc.chunks)} chunks")
```
## Recipe-Based Pipelines
Load pre-configured pipelines from the Chonkie Hub:
```python theme={"system"}
# Load markdown processing recipe
pipeline = Pipeline.from_recipe("markdown")
# Run with your content
doc = pipeline.run(texts="# My Markdown\n\nContent here")
# Load custom local recipe
pipeline = Pipeline.from_recipe("custom", path="./my_recipe.json")
```
Recipes are stored in the [chonkie-ai/recipes](https://huggingface.co/datasets/chonkie-ai/recipes) repository.
## Best Practices
Explicitly set `chunk_size` for predictable behavior:
```python theme={"system"}
# Good - explicit size
.chunk_with("recursive", chunk_size=512)
# Avoid - uses defaults that may change
.chunk_with("recursive")
```
Choose chunkers appropriate for your content:
```python theme={"system"}
# Code files → Code chunker
.chunk_with("code")
# Need semantic similarity → Semantic chunker
.chunk_with("semantic", threshold=0.8)
# General text → Recursive chunker
.chunk_with("recursive")
```
Add overlap refineries for better retrieval context:
```python theme={"system"}
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=100)
```
Always specify file extensions to avoid unwanted files:
```python theme={"system"}
# Good - filtered
.fetch_from("file", dir="./docs", ext=[".txt", ".md"])
# Bad - processes everything including binaries
.fetch_from("file", dir="./docs")
```
Multiple refineries can be chained:
```python theme={"system"}
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=50)
.refine_with("embedding", model="text-embedding-3-small")
```
## Pipeline Validation
Pipelines validate configuration before execution:
✅ **Must have**: At least one chunker
✅ **Must have**: Fetcher OR text input via `run(texts=...)`
❌ **Cannot have**: Multiple chefs (only one allowed)
```python theme={"system"}
# ❌ Invalid - no chunker
Pipeline().fetch_from("file", path="doc.txt").run()
# ❌ Invalid - multiple chefs
Pipeline()
.process_with("text")
.process_with("markdown") # Error!
.chunk_with("recursive")
# ✅ Valid - has chunker and input source
Pipeline()
.fetch_from("file", path="doc.txt")
.chunk_with("recursive", chunk_size=512)
.run()
# ✅ Valid - text input, no fetcher needed
Pipeline()
.chunk_with("recursive")
.run(texts="Hello world")
```
## Return Values
Pipeline behavior depends on input:
* **Single file/text**: Returns `Document`
* **Multiple files/texts**: Returns `list[Document]`
```python theme={"system"}
# Single file → Document
doc = Pipeline().fetch_from("file", path="doc.txt").chunk_with("recursive").run()
assert isinstance(doc, Document)
# Directory → list[Document]
docs = Pipeline().fetch_from("file", dir="./docs").chunk_with("recursive").run()
assert isinstance(docs, list)
# Multiple texts → list[Document]
docs = Pipeline().chunk_with("recursive").run(texts=["t1", "t2"])
assert isinstance(docs, list)
```
## Error Handling
Pipelines provide clear error messages:
```python theme={"system"}
from pathlib import Path
try:
doc = Pipeline()
.fetch_from("file", path="missing.txt")
.chunk_with("recursive")
.run()
except FileNotFoundError as e:
print(f"File not found: {e}")
except ValueError as e:
print(f"Configuration error: {e}")
except RuntimeError as e:
print(f"Pipeline execution failed: {e}")
```
## Component Overview
### Available Components
Explore each component type:
Connect to data sources (files, APIs, databases)
Preprocess text, markdown, tables, etc.
Split text with various strategies
Add overlap, embeddings, and more
Export to JSON, Datasets, etc.
Store in Chroma, Qdrant, Pinecone, etc.
## What's Next?
Learn how to connect different data sources in [Fetchers](/oss/fetchers/overview)
Find the right chunking strategy in [Chunkers](/oss/chunkers/overview)
Improve chunk quality in [Refineries](/oss/refinery/overview)
Ingest into vector databases with [Handshakes](/oss/handshakes/overview)
# DatasetsPorter
Source: https://docs.chonkie.ai/oss/porters/datasets-porter
Export Chonkie's Chunks into a Hugging Face Dataset.
The `DatasetsPorter` exports a list of `Chunk` objects into a Hugging Face `Dataset` object. This is particularly useful for saving your processed chunks in a standardized format for training models, sharing, or archiving.
## Installation
The `DatasetsPorter` requires the `datasets` library. You can install it with:
```bash theme={"system"}
pip install "chonkie[datasets]"
```
For general installation instructions, see the [Installation
Guide](/oss/installation).
## Initialization
To get started, simply import and initialize the porter.
```python theme={"system"}
from chonkie import DatasetsPorter
porter = DatasetsPorter()
```
## Parameters
The list of `Chunk` objects to be exported.
If `True`, the dataset will be saved to the location specified in the `path`
parameter.
The local directory path where the dataset should be saved. This is only used
if `save_to_disk` is `True`.
Additional keyword arguments to be passed directly to the
`datasets.Dataset.save_to_disk` method. This allows you to control aspects
like the number of shards or processes.
## Usage
The `DatasetsPorter` can either return a `Dataset` object directly for in-memory use or save it to disk.
### Return a Dataset Object
By default, the porter returns a `Dataset` object without writing any files.
```python theme={"system"}
from chonkie import Chunk
chunks = [
Chunk(text="This is the first chunk.", start_index=0, end_index=25, token_count=5),
Chunk(text="This is the second chunk.", start_index=26, end_index=52, token_count=5),
]
# Get the dataset in memory
dataset = porter.export(chunks)
print(dataset)
# Expected output:
# Dataset({
# features: ['text', 'start_index', 'end_index', 'token_count', 'context'],
# num_rows: 2
# })
```
### Save a Dataset to Disk
To save the dataset, set `save_to_disk=True` and provide a `path`. The method will still return the `Dataset` object.
```python theme={"system"}
# Save the dataset to a directory named "my_exported_chunks"
dataset = porter.export(chunks, save_to_disk=True, path="my_exported_chunks")
# You can now find the dataset files in the "my_exported_chunks" directory
```
### Using as a Callable
The porter can also be used as a callable, which is an alias for the `export` method.
```python theme={"system"}
# Get the dataset in memory
dataset = porter(chunks)
# Save the dataset to disk
porter(chunks, save_to_disk=True, path="my_exported_chunks")
```
## Return Type
The `export` method (and the `__call__` method) will always return a `datasets.Dataset` object, regardless of whether it is saved to disk. This allows you to immediately work with the dataset after exporting.
# JSONPorter
Source: https://docs.chonkie.ai/oss/porters/json-porter
Export Chonkie's Chunks into a JSON file.
Port your chunks to a JSON file with the `JSONPorter`. This is useful for exporting your chunked data for use in other applications or for archiving.
## Initialization
```python theme={"system"}
from chonkie import JSONPorter
from chonkie.types.base import Chunk
chunks = [
Chunk(
id="chunk1",
text="This is the first chunk.",
metadata={"source": "document1.txt"}
),
Chunk(
id="chunk2",
text="This is the second chunk.",
metadata={"source": "document2.txt"}
)
]
porter = JSONPorter()
porter.export(chunks)
```
# Porters Overview
Source: https://docs.chonkie.ai/oss/porters/overview
Overview of the different porters available in Chonkie
Porters allow you to easily Port your chunks to any format or destination.
Port your chunks to a JSON file.
Port your chunks to a Hugging Face Datasets.
# Get Started with Chonkie
Source: https://docs.chonkie.ai/oss/quick-start
Get started with Chonkie
Using Chonkie takes two simple steps: First, install the package. Next, start chunking!
This page covers Chonkie Open Source. To get started with our API, visit the
[API Reference](/api/common/introduction).
## Installation
### Python
```bash pip theme={"system"}
pip install chonkie
```
```bash uv theme={"system"}
uv add chonkie
```
Want more features? Install everything with `pip install "chonkie[all]"`. See
[Installation](/oss/installation) for more options.
### JavaScript
Install the core package for local chunking
```bash npm theme={"system"}
npm install @chonkiejs/core
```
```bash pnpm theme={"system"}
pnpm add @chonkiejs/core
```
```bash bun theme={"system"}
bun add @chonkiejs/core
```
```bash yarn theme={"system"}
yarn add @chonkiejs/core
```
Chonkie JS provides local support for TokenChunker, SentenceChunker, RecursiveChunker, FastChunker, TableChunker, SemanticChunker, and CodeChunker. Other chunkers are available through the API.
## CHONK! 🦛✨
```python Python theme={"system"}
# First import the chunker you want from Chonkie
from chonkie import TokenChunker
# Initialize the chunker
chunker = TokenChunker() # defaults to using character tokenizer
# Here's some text to chunk
text = """Woah! Chonkie, the chunking library is so cool!"""
# Chunk some text
chunks = chunker(text)
# Access chunks
for chunk in chunks:
print(f"Chunk: {chunk.text}")
print(f"Tokens: {chunk.token_count}")
```
```javascript JavaScript theme={"system"}
// First import the chunker you want from Chonkie
import { RecursiveChunker } from "@chonkiejs/core";
// Create a chunker
const chunker = await RecursiveChunker.create({
chunkSize: 512,
minCharactersPerChunk: 24,
});
// Chunk your text
const chunks = await chunker.chunk(
"Woah! Chonkie, the chunking library is so cool!"
);
// Use the chunks
for (const chunk of chunks) {
console.log(chunk.text);
console.log(`Tokens: ${chunk.tokenCount}`);
}
```
# Embeddings Refinery
Source: https://docs.chonkie.ai/oss/refinery/embeddings-refinery
Embed Chunked Texts
The `EmbeddingsRefinery` allows you to add more more information
to your chunks by adding embeddings to them. This is useful for
downstream tasks like semantic search, clustering, or vector database insertions.
## API Reference
To use the `EmbeddingsRefinery` via the API, check out the [API reference documentation](../../api/refineries/embeddings).
## Initialization
To use the `EmbeddingsRefinery`, you need to initialize it with an embedding model.
```python theme={"system"}
from chonkie import EmbeddingsRefinery
# Initialize with string model identifier
# or an embedding model instance
em_refinery = EmbeddingsRefinery(
embedding_model="minishlab/potion-base-32M", # Required
)
```
## Usage
Use the `EmbeddingsRefinery` object as a callable or the
`refine` method to add embeddings to your chunks.
```python theme={"system"}
from chonkie import TokenChunker, EmbeddingsRefinery
test_string = "This is a test string. It will be chunked and embedded."
chunker = TokenChunker()
chunks = chunker(test_string)
# Add embeddings to the chunks
em_refinery = EmbeddingsRefinery(
embedding_model="minishlab/potion-base-32M", # Model string or BaseEmbeddings instance
)
chunks_with_embeddings = em_refinery(chunks)
```
## Parameters
Model identifier or embedding model instance
# Overlap Refinery
Source: https://docs.chonkie.ai/oss/refinery/overlap-refinery
Refine chunks by adding overlapping context from adjacent chunks.
The `OverlapRefinery` enhances chunks by incorporating context from neighboring chunks. This is useful for tasks where maintaining contextual continuity between chunks is important, such as question answering or summarization over long documents. It can add context as a prefix (from the preceding chunk) or a suffix (from the next chunk).
## API Reference
To use the `OverlapRefinery` via the API, check out the [API reference documentation](../../api/refineries/overlap).
## Initialization
To use the `OverlapRefinery`, initialize it with the desired parameters. You can specify a tokenizer, context size, overlap mode, method, and other options.
```python theme={"system"}
from chonkie import OverlapRefinery
# Initialize with default character-level overlap (25% context size)
overlap_refinery = OverlapRefinery()
# Initialize with a specific tokenizer and context size
overlap_refinery_token = OverlapRefinery(
tokenizer="character", # Default tokenizer (or use "gpt2", etc.)
context_size=0.25, # The size of the context to add to the chunks.
method="prefix", # Add context from the previous chunk
merge=True # Merge context directly into chunk text
)
# Initialize with justified method (context from both sides)
overlap_refinery_justified = OverlapRefinery(
tokenizer="character",
context_size=0.25,
method="justified", # Add context from both previous and next chunks
merge=True
)
# Initialize for recursive overlap based on rules
from chonkie import RecursiveRules, RecursiveLevel
rules = RecursiveRules(
levels=[
RecursiveLevel(delimiters=["\n\n"], include_delim="prev"),
RecursiveLevel(delimiters=["."], include_delim="prev"),
RecursiveLevel(whitespace=True)
]
)
overlap_refinery_recursive = OverlapRefinery(
tokenizer="character",
context_size=0.25,
mode="recursive",
rules=rules,
method="suffix"
)
```
## Usage
Use the `OverlapRefinery` object as a callable or use the `refine` method to add overlapping context to your chunks.
```python theme={"system"}
from chonkie import TokenChunker, OverlapRefinery
test_string = "This is the first sentence. This is the second sentence, providing context. This is the third sentence, which needs context from the second."
chunker = TokenChunker()
chunks = chunker(test_string)
# Initialize refinery to add suffix overlap
overlap_refinery = OverlapRefinery(
tokenizer="character",
context_size=0.5,
method="suffix",
merge=True
)
refined_chunks = overlap_refinery(chunks)
```
## Parameters
The tokenizer to use for calculating overlap size. Can be a
string identifier (e.g., "character", "word", "gpt2"), a callable, or a
`chonkie.Tokenizer` instance. Defaults to "character".
The size of the overlap context. If an `int`, it's the absolute number of
tokens. If a `float` (between 0 and 1), it's the fraction of the maximum chunk
token count.
The mode for calculating overlap. `"token"` uses the tokenizer directly.
`"recursive"` uses hierarchical splitting based on `rules`.
The method for adding context. `"suffix"` adds context from the *next* chunk
to the end of the current chunk. `"prefix"` adds context from the *previous*
chunk to the beginning of the current chunk. `"justified"` adds context
from both the previous and next chunks - for middle chunks it includes
context from both sides, while first and last chunks get context
from the single adjacent chunk.
The rules used for splitting text when `mode` is `"recursive"`. Defines
delimiters and behavior at different hierarchical levels. See
`chonkie.types.RecursiveRules`.
If `True`, the calculated context is directly prepended (for `prefix`) or
appended (for `suffix`) to the `chunk.text`. If `False`, the context is stored
in `chunk.context` attribute without modifying `chunk.text`.
If `True`, modifies the input list of chunks directly. If `False`, returns a
new list of modified chunks.
# Refinery Overview
Source: https://docs.chonkie.ai/oss/refinery/overview
Overview of the different refinery available in Chonkie
Refinery is a module in Chonkie that allows you to refine chunks.
Refineries help you to add additional `context` to your chunks which are useful to improve the quality of your embeddings and keyword indexing.
Refines chunks by adding overlapping chunks to the original chunk.
Refines chunks by adding embeddings to the original chunk.
# Hubbie
Source: https://docs.chonkie.ai/oss/utils/hubbie
Hubbie is a utility for accessing Chonkie's saved recipes.
Hubbie is a utility for accessing Chonkie's saved recipes. Recipes are pre-defined chunking rules for different languages and document types. When initializing a chunker with `from_recipe`, you can pass in the recipe name and language to use the recipe.
## Installation
To make use of Hubbie, you'll need to install the `hub` optional install.
```bash theme={"system"}
pip install "chonkie[hub]"
```
## Usage
```python theme={"system"}
from chonkie import RecursiveChunker
# Initialize the recursive chunker with the recipe name and language
chunker = RecursiveChunker.from_recipe("markdown", lang="en")
# Chunk the text
text = ... # Your text string
# CHONK!
chunks = chunker(text)
```
## Recipes
You can access Chonkie's saved recipes on [Hugging Face](https://huggingface.co/datasets/chonkie-ai/recipes).
# Logging
Source: https://docs.chonkie.ai/oss/utils/logging
Control Chonkie's log output
By default, Chonkie logs warnings and errors. You can control what gets logged using the `CHONKIE_LOG` environment variable or configure it programmatically.
## Quick Setup
Set the `CHONKIE_LOG` environment variable before importing chonkie:
```bash theme={"system"}
export CHONKIE_LOG=debug # Show everything
export CHONKIE_LOG=info # Show info, warnings, and errors
export CHONKIE_LOG=warning # Show warnings and errors (default)
export CHONKIE_LOG=error # Show only errors
export CHONKIE_LOG=off # Disable all logging
```
## Programmatic Configuration
Configure logging from your Python code:
```python theme={"system"}
import chonkie
# Enable debug logging
chonkie.logger.configure("debug")
# Disable logging completely
chonkie.logger.configure("off")
# Back to warnings and errors
chonkie.logger.configure("warning")
```
## Log Levels
* `debug` - Everything (chunker initialization, text processing, chunk counts)
* `info` - High-level operations (chunk creation, file operations)
* `warning` - Potential issues (default)
* `error` - Only errors
* `off` - Silent
## Custom Format
Change the log format to suit your needs:
```python theme={"system"}
import chonkie
chonkie.logger.configure("info")
```
Output:
```
2025-11-13 14:18:58,714 | INFO | chonkie.chunker.token:chunk:143 - Created 5 chunks
```
```python theme={"system"}
import chonkie
chonkie.logger.configure("info", format="%(levelname)s - %(message)s")
```
Output:
```
INFO - Created 5 chunks
```
```python theme={"system"}
import chonkie
chonkie.logger.configure("info", format="%(message)s")
```
Output:
```
Created 5 chunks
```
```python theme={"system"}
import chonkie
chonkie.logger.configure("info", format="[%(name)s] %(levelname)s: %(message)s")
```
Output:
```
[chonkie.chunker.token] INFO: Created 5 chunks
```
```python theme={"system"}
import chonkie
import logging
# Custom formatter that includes extra fields
class StructuredFormatter(logging.Formatter):
def format(self, record):
msg = super().format(record)
# Add any extra fields
extras = []
for key in ['chunk_count', 'tokens']:
if hasattr(record, key):
extras.append(f"{key}={getattr(record, key)}")
if extras:
msg += f" [{', '.join(extras)}]"
return msg
# Apply custom formatter
logger = logging.getLogger("chonkie")
handler = logging.StreamHandler()
handler.setFormatter(StructuredFormatter("%(levelname)s - %(message)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
```
Output:
```
INFO - Created chunks [chunk_count=5, tokens=128]
```
That's it. Logging is simple and stays out of your way.
# Visualizer
Source: https://docs.chonkie.ai/oss/utils/visualizer
Visualize your chunks and embeddings
The `Visualizer` helps you visualize your chunks properly and compare different chunkers and settings with ease, either on the terminal or via HTML.
## Installation
To make use of the `Visualizer`, you'll need to install the `viz` optional install.
```bash theme={"system"}
pip install "chonkie[viz]"
```
## Usage
To use the `Visualizer`, you'll need to import it from `chonkie.utils`.
```python theme={"system"}
# Import the Visualizer
from chonkie import Visualizer
# Initialize the Visualizer
viz = Visualizer()
```
The `Visualizer` has two main methods: `print` and `save`.
The `print` method will print the chunks to the terminal. It accepts a list of `Chunk` objects or plain strings.
```python theme={"system"}
viz.print(chunks)
# Or you can directly call the Visualizer object
viz(chunks)
# You can also pass a list of plain strings
viz(["First text segment", "Second text segment", "Third text segment"])
```
The `save` method will save the chunks to a HTML file. Like `print`, it accepts `Chunk` objects or strings.
```python theme={"system"}
viz.save("chonkie.html", chunks)
```