SentenceChunker

Sentence-aware document chunker.

Splits text at sentence boundaries to preserve semantic coherence. Uses pattern matching for sentence detection (periods, question marks, etc.) while handling edge cases like abbreviations and decimal numbers.

This chunker produces higher quality chunks than simple character-based splitting because it never breaks in the middle of a sentence.

A sentence ends at ., ! or ?, optionally followed by closing quotes or brackets, when whitespace and then an uppercase letter (optionally after opening quotes or brackets) follow: He said "Hi." Then is two sentences, He said "Hi." and Then. Abbreviations such as Dr. and e.g. (as whole words) and decimal numbers such as 3.14 do not end a sentence. Sentences in a chunk keep the whitespace that separated them in the input, so with no overlap and no sentence longer than maxSize, every chunk is a slice of the input.

A sentence longer than maxSize is split at whitespace. A single word (a run without whitespace) longer than maxSize is kept whole as its own chunk, so such a chunk can exceed maxSize: the chunker never cuts inside a word, and so never between the two halves of a surrogate pair (#1711). For text that may contain long unbroken runs, such as URLs, base64 or CJK without spaces, use SimpleChunker, which cuts at a fixed size.

Usage:

val chunker = SentenceChunker()
val chunks = chunker.chunk(text, ChunkingConfig(targetSize = 800))

// Sentences are kept intact
chunks.foreach { c =>
 println(s"[$${c.index}] $${c.content}")
}