llm4s-docs
API
Generated with
llm4s-docs/org.llm4s/org.llm4s.llmconnect/org.llm4s.llmconnect.utils/ChunkingUtils

ChunkingUtils

org.llm4s.llmconnect.utils.ChunkingUtils
object ChunkingUtils

Utilities for splitting text, audio, and video data into fixed-size overlapping chunks.

Each chunking method produces a sequence of windows with configurable size and overlap, suitable for embedding pipelines and multimodal processing.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Value members

Concrete methods

def chunkAudio(samples: Array[Float], sampleRate: Int, window: FiniteDuration, overlapRatio: Double, padToWindow: Boolean): Seq[Array[Float]]

Window an audio signal into fixed-length segments with overlap. Optionally right-pad the final window with zeros so all windows have equal length.

Window an audio signal into fixed-length segments with overlap. Optionally right-pad the final window with zeros so all windows have equal length.

Value parameters

overlapRatio

Overlap ratio in [0, 1). For example, 0.25 = 25% overlap.

padToWindow

If true, pad the last segment with zeros to full window length.

sampleRate

Samples per second (> 0).

samples

Mono PCM samples in [-1, 1].

window

Window length (> 0); at least one sample.

Attributes

Returns

Sequence of audio windows (each Array[Float] of length windowSamples if padded).

def chunkText(text: String, size: Int, overlap: Int): Seq[String]

Splits a long text into chunks with specified size and overlap.

Splits a long text into chunks with specified size and overlap.

Throws IllegalArgumentException on an invalid size or overlap; use chunkTextValidated when they come from input.

Sizes are in UTF-16 units (String.length), but no chunk starts or ends between the two halves of a surrogate pair (#1711): a window end that would fall there moves back by one, and so does an overlap start. The one exception is size = 1 at an astral character, where moving back would leave the chunk empty: that chunk is the whole character, two units long. Text without astral characters is chunked exactly as by character index.

Value parameters

overlap

Number of overlapping characters between chunks (0 <= overlap < size).

size

Maximum characters per chunk (> 0).

text

Input string.

Attributes

Returns

Sequence of text chunks.

def chunkTextValidated(text: String, size: Int, overlap: Int): Result[Seq[String]]

As chunkText, but an invalid size or overlap is a Left(ValidationError) rather than a throw.

As chunkText, but an invalid size or overlap is a Left(ValidationError) rather than a throw.

Attributes

def chunkVideo[T](frames: Seq[T], fps: Int, clip: FiniteDuration, overlapRatio: Double): Seq[Seq[T]]

Chunk a sequence of frames into clips of fixed duration with overlap. Generic over frame type T (e.g., BufferedImage).

Chunk a sequence of frames into clips of fixed duration with overlap. Generic over frame type T (e.g., BufferedImage).

Value parameters

clip

Clip duration (> 0); at least one frame.

fps

Frames per second (> 0).

frames

Sequence of frames.

overlapRatio

Overlap ratio in [0, 1).

Attributes

Returns

Sequence of frame clips (each is a Seq[T]).

Generated with