The WordChunker splits text into chunks while preserving word boundaries, ensuring that words stay intact and readable.

Installation

WordChunker is included in the base installation of Chonkie. No additional dependencies are required.

For installation instructions, see the Installation Guide.

Initialization

Here’s how to initialize the WordChunker with the default parameters. Have a look at the Usage Examples section for more examples on how to use the chunker.

from chonkie import WordChunker

# Basic initialization with default parameters
chunker = WordChunker(
    tokenizer_or_token_counter="gpt2",        # Supports string identifiers
    chunk_size=512,                           # Maximum tokens per chunk
    chunk_overlap=0,                          # Overlap between chunks
    return_type="chunks"                      # Return type of the chunker; "chunks" or "texts"
)

Parameters

tokenizer_or_token_counter
Union[str, tokenizers.Tokenizer, tiktoken.Encoding, Callable]
default:"gpt2"

Tokenizer to use. Can be a string identifier or a tokenizer instance. If a callable is passed, it will be used as the token counter. If “character” is passed, the character tokenizer will be used. If “word” is passed, the word tokenizer will be used.

chunk_size
int
default:"512"

Maximum number of tokens per chunk

chunk_overlap
int
default:"0"

Number of overlapping tokens between chunks

return_type
str
default:"chunks"

Return type of the chunker; “chunks” or “texts”

Methods

The WordChunker class provides the following methods.

__call__

The __call__ method allows you to call the chunker like a function, which uses the .chunk or .chunk_batch method internally, depending on the arguments passed.

Arguments:

text
Union[str, List[str]]
default:"None"

Text to chunk.

show_progress_bar
bool
default:"True"

Whether to show a progress bar (only works if text is a list).

Returns:

Result
Union[List[Chunk], List[str], List[List[Chunk]], List[List[str]]]
default:"None"

Result of the chunking process.

.chunk

The .chunk method chunks a single text into chunks.

Arguments:

text
str
default:"None"

Text to chunk.

Returns:

Result
Union[List[Chunk], List[str]]

Result of the chunking process.

.chunk_batch

The .chunk_batch method chunks a batch of texts into chunks.

Arguments:

texts
List[str]
default:"None"

List of texts to chunk.

show_progress_bar
bool
default:"True"

Whether to show a progress bar.

Returns:

Result
Union[List[Chunk], List[str]]

Result of the chunking process.

Usage Examples

Associated Return Types

WordChunker returns chunks as Chunk objects with the following attributes:

@dataclass
class Chunk:
    text: str           # The chunk text
    start_index: int    # Starting position in original text
    end_index: int      # Ending position in original text
    token_count: int    # Number of tokens in chunk