Chonkie Documentation

The SemanticChunker splits text into chunks based on semantic similarity, ensuring that related content stays together in the same chunk. This approach is particularly useful for RAG applications where context preservation is crucial.

Installation

SemanticChunker requires additional dependencies for semantic capabilities. You can install it with:

pip install "chonkie[semantic]"

For installation instructions, see the Installation Guide.

Initialization

Here’s how to initialize the SemanticChunker with default parameters. In most cases, you can use the default parameters without any issues.

from chonkie import SemanticChunker

# Basic initialization with default parameters
chunker = SemanticChunker(
    embedding_model="minishlab/potion-base-8M",  # Default model
    mode="window",                               # Mode for grouping sentences, either "cumulative" or "window"
    threshold="auto",                            # Similarity threshold (0-1) or (1-100) or "auto"
    chunk_size=512,                              # Maximum tokens per chunk
    similarity_window=1,                         # Number of sentences to consider while windowing for similarity
    min_sentences=1                              # Minimum number of sentences per chunk
    min_characters_per_sentence=12,              # Minimum number of characters per sentence
    min_chunk_size=2,                            # Minimum tokens per chunk
    threshold_step=0.01,                         # Step size for similarity threshold calculation
    delim=['.', '!', '?', '\n']                  # Delimiters to split sentences on
    return_type="chunks"                         # Return type, either "chunks" or "texts"
)

Parameters

embedding_model

Union[str, BaseEmbeddings]

default:"minishlab/potion-base-8M"

Model identifier or embedding model instance

mode

Optional[str]

default:"window"

Mode for grouping sentences, either “cumulative” or “window” Window mode will group sentences based on similarity within a window of sentences. Cumulative dynamically adjusts the window size based on the similarity of the sentences, but will take longer to process.

threshold

Optional[float, int, str]

default:"auto"

When in the range [0,1], denotes the similarity threshold to consider sentences similar. When in the range (1,100], interprets the given value as a percentile threshold. When set to “auto”, the threshold is automatically calculated.

chunk_size

int

default:"512"

Maximum tokens per chunk

similarity_window

Optional[int]

default:"1"

Number of sentences to consider for similarity threshold calculation

min_sentences

Optional[int]

default:"1"

Minimum number of sentences per chunk

min_characters_per_sentence

Optional[int]

default:"12"

Minimum number of characters per sentence

min_chunk_size

Optional[int]

default:"2"

Minimum tokens per chunk

threshold_step

Optional[float]

default:"0.01"

Step size for similarity threshold calculation

delim

Union[str, List[str]]

default:"['.', '!', '?', '\\n']"

Delimiters to split sentences on

return_type

Optional[str]

default:"chunks"

Return type, either “chunks” or “texts”

Methods

The SemanticChunker has the following methods:

`call`

The __call__ method allows you to call the chunker like a function, which uses the .chunk or .chunk_batch method internally, depending on the arguments passed.

Arguments:

text

Union[str, List[str]]

default:"None"

Text to chunk.

show_progress_bar

bool

default:"True"

Whether to show a progress bar.

Returns:

Result

Union[List[SemanticChunk], List[str], List[List[SemanticChunk]], List[List[str]]]

default:"None"

Result of the chunking process.

`.chunk`

The .chunk method chunks a single text into chunks.

Arguments:

text

str

default:"None"

Text to chunk.

Returns:

Result

Union[List[SemanticChunk], List[str]]

default:"None"

Result of the chunking process.

`.chunk_batch`

The .chunk_batch method chunks a batch of texts into chunks.

Arguments:

texts

List[str]

default:"None"

List of texts to chunk.

show_progress_bar

bool

default:"True"

Whether to show a progress bar.

Returns:

Result

Union[List[List[SemanticChunk]], List[List[str]]]

default:"None"

Result of the chunking process.

Usage Examples

Even with the default parameters, the SemanticChunker will do a good job at chunking the text.

# Import the SemanticChunker
from chonkie import SemanticChunker 

# Initialize the chunker
chunker = SemanticChunker()

# Chunk a single text
text = """First paragraph about a specific topic.
Second paragraph continuing the same topic.
Third paragraph switching to a different topic.
Fourth paragraph expanding on the new topic."""

chunks = chunker(text)
chunks = chunker.chunk(text) # OR use the chunk method

for chunk in chunks:
    print(chunk.text)
    print(chunk.token_count)
    print(f"{chunk.start_index} - {chunk.end_index}")

The SemanticChunker can also chunk a batch of texts at once.

# Import the SemanticChunker
from chonkie import SemanticChunker

# Initialize the chunker   
chunker = SemanticChunker()

texts = [
    "First text about a specific topic.",   
    "Second text about a different topic.",
    "Third text about a third topic."
]

# Chunk the batch of texts
chunks = chunker(texts)
chunks = chunker.chunk_batch(texts) # OR use the chunk_batch method

for chunks in batch_chunks:
    for chunk in chunks:
        print(chunk.text)
        print(chunk.token_count)
        print(f"{chunk.start_index} - {chunk.end_index}")

You can also get texts instead of chunks by passing in “texts” in the return_type argument.

# Import the SemanticChunker
from chonkie import SemanticChunker

# Initialize the chunker
chunker = SemanticChunker(return_type="texts")

# Get your text
text = "Woah! Chonkie is a great library! I can't believe how easy it is to use. It's so much fun!"

# Chunk a single text
chunks = chunker(text)

# Print the chunks
for chunk in chunks:
    print(chunk) # This will print the chunk text (str)

You can also use a custom embedding model by passing in an embedding model instance or identifier.

# Import the SemanticChunker
from chonkie import SemanticChunker
from chonkie import AutoEmbeddings, SentenceTransformersEmbeddings

# Initialize the embedding model
embedding_model = AutoEmbeddings(model="all-minilm-l6-v2")
# OR
embedding_model = SentenceTransformersEmbeddings(model="all-minilm-l6-v2")

# Initialize the chunker
chunker = SemanticChunker(embedding_model=embedding_model)

# Chunk a single text
text = "Woah! Chonkie is a great library! I can't believe how easy it is to use. It's so much fun!"
chunks = chunker(text)

for chunk in chunks:
    print(chunk.text)
    print(chunk.token_count)
    print(f"{chunk.start_index} - {chunk.end_index}")

You can also use a custom threshold by passing in a float or int.

# Import the SemanticChunker
from chonkie import SemanticChunker

# Initialize the chunker
chunker = SemanticChunker(threshold=0.5) # Set to value of 0.5 (0-1)
# OR
chunker = SemanticChunker(threshold=80) # Set to 80 percentile threshold (1-100)

# Chunk a single text
text = "Woah! Chonkie is a great library! I can't believe how easy it is to use. It's so much fun!"
chunks = chunker(text)

for chunk in chunks:
    print(chunk.text)
    print(chunk.token_count)
    print(f"{chunk.start_index} - {chunk.end_index}")

You can also use custom delimiters by passing in a list of delimiters.

# Import the SemanticChunker
from chonkie import SemanticChunker

# Initialize the chunker
chunker = SemanticChunker(delim=[".", "!", "?", "\n"])

# Chunk a single text
text = "Woah! Chonkie is a great library! I can't believe how easy it is to use. It's so much fun!"
chunks = chunker(text)

for chunk in chunks:
    print(chunk.text)
    print(chunk.token_count)
    print(f"{chunk.start_index} - {chunk.end_index}")

Supported Embeddings

SemanticChunker supports multiple embedding providers through Chonkie’s embedding system. See the Embeddings Overview for more information.

Associated Return Type

SemanticChunker returns SemanticChunk objects:

@dataclass
class SemanticSentence(Sentence):
    text: str
    start_index: int
    end_index: int
    token_count: int
    embedding: Optional[np.ndarray]  # Sentence embedding vector

@dataclass
class SemanticChunk(SentenceChunk):
    text: str
    start_index: int
    end_index: int
    token_count: int
    sentences: List[SemanticSentence]

Getting Started

Chunkers

Embeddings

Refinery

SemanticChunker

Installation

Initialization

Parameters

Methods

`call`

`.chunk`

`.chunk_batch`

Usage Examples

Supported Embeddings

Associated Return Type

Getting Started

Chunkers

Embeddings

Refinery

​Installation

​Initialization

​Parameters

​Methods

​__call__

​.chunk

​.chunk_batch

​Usage Examples

​Supported Embeddings

​Associated Return Type

Installation

Initialization

Parameters

Methods

`call`

`.chunk`

`.chunk_batch`

Usage Examples

Supported Embeddings

Associated Return Type