Overview of the different refinery available in Chonkie

Overview

Refinery Overview

Chonkie

The no-nonsense RAG chunking library that's lightweight, fast, and ready to CHONK your texts!

Introduction

🦛 Chonkie ✨

Installing Chonkie and its various components

Installation

Concepts

Chonkie Cloud

Pricing

Overview of the different chunkers available in Chonkie

Chunkers Overview

Split text into fixed-size token chunks with configurable overlap

TokenChunker

Split text into chunks while preserving sentence boundaries

SentenceChunker

Recursively chunk documents into smaller chunks.

RecursiveChunker

Split code into chunks based on code structure

CodeChunker

Split text into chunks based on semantic similarity

SemanticChunker

Split text using Semantic Double-Pass Merging for improved context preservation

SDPMChunker

Split text into chunks based on a late-bound token count

LateChunker

Split text using a fine-tuned BERT model to detect semantic shifts

NeuralChunker

Agentic chunking powered by generative models via the Genie interface

SlumberChunker

Overview of the different embeddings available in Chonkie

Embeddings Overview

Automatically select the best embeddings handler for your use case

AutoEmbeddings

Embed text using SentenceTransformer embedding models

SentenceTransformer

SentenceTransformerEmbeddings

OpenAI

OpenAIEmbeddings

Model2Vec

Model2VecEmbeddings

Custom

Create your own custom embeddings handler

Refine chunks by adding overlapping context from adjacent chunks.

OverlapRefinery

EmbeddingsRefinery

Visualizer

Hubbie is a utility for accessing Chonkie's saved recipes.

Hubbie

Validates the provided API key. Use this endpoint to check if your API key is valid and active.

Validate API Key

Splits text into chunks based on token count, ensuring each chunk stays within specified token limits.

Token Chunker

Splits text into chunks while preserving complete sentences, ensuring that each chunk maintains proper sentence boundaries and context.

Sentence Chunker

Recursively chunks documents into smaller chunks. It is a good choice for documents that are long but well structured.

Recursive Chunker

Splits text into chunks based on semantic similarity, ensuring that related content stays together in the same chunk.

Semantic Chunker

Extends semantic chunking using a double-pass merging approach (Semantic Double-Pass Merging). Connects related content even if not consecutive.

SDPM Chunker

Implements the late chunking strategy. Encodes the entire text, then splits recursively, deriving chunk embeddings from the full document embedding.

Late Chunker

Splits code into chunks based on its structure, leveraging Abstract Syntax Trees (ASTs) to create contextually relevant segments.

Code Chunker

Splits text using a fine-tuned BERT model to detect semantic shifts, creating topic-coherent chunks.

Neural Chunker

Agentic chunking powered by generative models (via Genie interface) for intelligent text splitting.

Slumber Chunker

Adds embeddings to a list of provided chunks using a specified embedding model.

Embeddings Refinery

Adds overlapping context (prefix or suffix) to a list of provided chunks based on specified arguments.

Overlap Refinery

Chonkie's Release Notes and Updates 🦛✨

Changelog

The no-nonsense RAG chunking library. Lightweight, fast, and ready to CHONK your texts

Split text into chunks while maintaining word boundaries

WordChunker

Refines chunks by adding overlapping chunks to the original chunk.

Getting Started

Chunkers

Embeddings

Refinery

Utils

Refinery Overview

OverlapRefinery

EmbeddingsRefinery