Chonkie Documentation

The CodeChunker splits code into chunks based on its structure, leveraging Abstract Syntax Trees (ASTs) to create contextually relevant segments.

Overview

Supports 165+ languages
Powered by tree-sitter-language-pack
Auto language detection via Magika, a language detection library made by Google

Supported Languages

Show all supported languages

Each language is identified by the key used with get_language(key) and get_parser(key).

General-Purpose Programming Languages

Language	Key	License
ActionScript	`actionscript`	MIT
Ada	`ada`	MIT
Agda	`agda`	MIT
C	`c`	MIT
C++	`cpp`	MIT
C#	`csharp`	MIT
Dart	`dart`	MIT
Go	`go`	MIT
Java	`java`	MIT
JavaScript	`javascript`	MIT
Julia	`julia`	MIT
Kotlin	`kotlin`	MIT
Nim	`nim`	MPL-2.0
OCaml	`ocaml/ocaml_interface`	MIT
Perl	`perl`	Artistic-2.0
Python	`python`	MIT
R	`r`	MIT
Ruby	`ruby`	MIT
Rust	`rust`	MIT
Scala	`scala`	MIT
Swift	`swift`	MIT
TypeScript	`typescript`	MIT
Zig	`zig`	MIT

Web, UI & Markup

Language	Key	License
HTML	`html`	MIT
CSS	`css`	MIT
SCSS	`scss`	MIT
Astro	`astro`	MIT
Vue	`vue`	MIT
Svelte	`svelte`	MIT
TSX	`tsx`	MIT
Markdown	`markdown`	MIT
Markdown Inline	`markdown_inline`	MIT
Mermaid	`mermaid`	MIT
XML	`xml`	MIT
YAML	`yaml`	MIT

Config, Build & DevOps

Language	Key	License
Bash	`bash`	MIT
Dockerfile	`dockerfile`	MIT
Git Ignore	`gitignore`	MIT
Git Commit	`gitcommit`	WTFPL
Make	`make`	MIT
Ninja	`ninja`	MIT
Meson	`meson`	MIT
Prisma	`prisma`	MIT
Requirements	`requirements`	MIT

Systems, GPU & Low-level

Language	Key	License
ASM	`asm`	MIT
CUDA	`cuda`	MIT
GLSL	`glsl`	MIT
HLSL	`hlsl`	MIT
LLVM	`llvm`	MIT
Verilog	`verilog`	MIT
VHDL	`vhdl`	MIT
WGSL	`wgsl`	MIT
WAST / WAT	`wasm`	Apache-2.0 + LLVM

API Reference

To use the CodeChunker via the API, check out the API reference documentation.

Installation

CodeChunker requires additional dependencies for code parsing. You can install it with:

pip install "chonkie[code]"

For installation instructions, see the Installation Guide.

Initialization

from chonkie import CodeChunker

chunker = CodeChunker(
    language="python",      # Specify the programming language
    tokenizer="character",  # Default tokenizer (or use "gpt2", etc.)
    chunk_size=2048,        # Maximum tokens per chunk
    include_nodes=False     # Optionally include AST nodes in output
)

from chonkie import CodeChunker

chunker = CodeChunker(
    language="auto",        # Auto detects programming language via Magika
    tokenizer="character",  # Default tokenizer (or use "gpt2", etc.)
    chunk_size=2048,        # Maximum tokens per chunk
    include_nodes=False     # Optionally include AST nodes in output
)

from chonkie import CodeChunker
from tokenizers import Tokenizer

custom_tokenizer = Tokenizer.from_pretrained("your-tokenizer")
chunker = CodeChunker(
    language="javascript",
    tokenizer=custom_tokenizer,
    chunk_size=2048
)

Automatic language detection using Magika can impact performance. For best results, specify the language directly via the language parameter.

Parameters

str

required

The programming language of the code. Accepts languages supported by tree-sitter-language-pack.

Union[str, Callable, Any]

default:"character"

Tokenizer or token counting function to use for measuring chunk size.

int

default:"2048"

Maximum number of tokens per chunk.

bool

default:"False"

Whether to include AST node information (Note: with the base Chunk type, node information is not stored).

Usage

Single Code Chunking

Python
JavaScript

code = """
def hello_world():
    print("Hello, Chonkie!")

class MyClass:
    def __init__(self):
        self.value = 42
"""
chunks = chunker.chunk(code)

for chunk in chunks:
    print(f"Chunk text: {chunk.text}")
    print(f"Token count: {chunk.token_count}")

const code = `
def hello_world():
    print("Hello, Chonkie!")

class MyClass:
    def __init__(self):
        self.value = 42
`;
const chunks = await chunker.chunk(code);

for (const chunk of chunks) {
  console.log(`Chunk text: ${chunk.text}`);
  console.log(`Token count: ${chunk.tokenCount}`);
}

Batch Chunking

Python
JavaScript

codes = [
    "def func1():\n    pass",
    "const x = 10;\nfunction add(a, b) { return a + b; }"
]
batch_chunks = chunker.chunk_batch(codes)

for doc_chunks in batch_chunks:
    for chunk in doc_chunks:
        print(f"Chunk: {chunk.text}")

const codes = [
  "def func1():\n    pass",
  "const x = 10;\nfunction add(a, b) { return a + b; }"
];
const batchChunks = await chunker.chunkBatch(codes);

for (const docChunks of batchChunks) {
  for (const chunk of docChunks) {
    console.log(`Chunk: ${chunk.text}`);
  }
}

Using as a Callable

# Single code string
chunks = chunker("def greet(name):\n    print(f'Hello, {name}')")

# Multiple code strings
batch_chunks = chunker(["int main() { return 0; }", "package main\nimport \"fmt\"\nfunc main() { fmt.Println(\"Hi\") }"])

Return Type

CodeChunker returns chunks as Chunk objects:

@dataclass
class Chunk:
    text: str           # The chunk text (code snippet)
    start_index: int    # Starting position in original code
    end_index: int      # Ending position in original code
    token_count: int    # Number of tokens in chunk
    context: Optional[Context] = None    # Optional context metadata
    embedding: Union[list[float], "np.ndarray", None] = None  # Optional embedding vector

As of version 1.3.0, CodeChunker returns the base Chunk type instead of the specialized CodeChunk type. This simplifies integration with other chunkers and refineries.

Getting Started

API Server

Chefs

Fetchers

Chunkers

Embeddings

Refinery

Handshakes

Porters

Utils

Experimental

Changelog

Code Chunker

Overview

Supported Languages

General-Purpose Programming Languages

Web, UI & Markup

Config, Build & DevOps

Systems, GPU & Low-level

API Reference

Installation

Initialization

Parameters

Usage

Single Code Chunking

Batch Chunking

Using as a Callable

Return Type

​Overview

​Supported Languages

​General-Purpose Programming Languages

​Web, UI & Markup

​Config, Build & DevOps

​Systems, GPU & Low-level

​API Reference

​Installation

​Initialization

​Parameters

​Usage

​Single Code Chunking

​Batch Chunking

​Using as a Callable

​Return Type

Overview

Supported Languages

General-Purpose Programming Languages

Web, UI & Markup

Config, Build & DevOps

Systems, GPU & Low-level

API Reference

Installation

Initialization

Parameters

Usage

Single Code Chunking

Batch Chunking

Using as a Callable

Return Type