PgDocumentRegistry

org.llm4s.rag.loader.PgDocumentRegistry
See thePgDocumentRegistry companion object
@Experimental

A DocumentRegistry in a PostgreSQL table, so that RAG.sync survives a restart: the next sync finds the versions the last one registered, re-embeds only what changed, and deletes what the source dropped while the process was down.

One row per document: its id, version (content_hash, version_timestamp, etag) and the source that owns it. Several pipelines can share one table only if their document ids do not collide; give each its own table otherwise.

Build it over a pool the application already has with PgDocumentRegistry.fromDataSource - the same one a PgVectorStore or PgSearchIndex uses (#2300) - or let it open its own with PgDocumentRegistry.apply. close closes only a pool the registry opened. Calls are blocking JDBC.

Value parameters

dataSource

the connection pool

ownsDataSource

whether close closes dataSource

tableName

the registry table, created if it does not exist

Attributes

Companion
object
Source
PgDocumentRegistry.scala
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

Get all registered document IDs, of every source.

Get all registered document IDs, of every source.

Attributes

Returns

Set of all document IDs currently registered

Source
PgDocumentRegistry.scala
def clear(): Result[Unit]

Clear all registrations, of every source.

Clear all registrations, of every source.

Attributes

Source
PgDocumentRegistry.scala
override def close(): Unit

Closes the connection pool only if this registry opened it; a caller-owned DataSource is left open.

Closes the connection pool only if this registry opened it; a caller-owned DataSource is left open.

Attributes

Definition Classes
Source
PgDocumentRegistry.scala
override def count(): Result[Int]

Get the count of registered documents.

Get the count of registered documents.

Attributes

Definition Classes
Source
PgDocumentRegistry.scala
def documentIds(sourceId: String): Result[Set[String]]

Get the IDs of the documents sourceId owns.

Get the IDs of the documents sourceId owns.

Value parameters

sourceId

The source

Attributes

Returns

The IDs of its registered documents

Source
PgDocumentRegistry.scala

Check if document exists and get its version.

Check if document exists and get its version.

Value parameters

docId

Document identifier

Attributes

Returns

Version if document is registered, None otherwise

Source
PgDocumentRegistry.scala
def register(docId: String, version: DocumentVersion, sourceId: String): Result[Unit]

Register a document as indexed, owned by sourceId. Registering a document again replaces its version and moves it to sourceId.

Register a document as indexed, owned by sourceId. Registering a document again replaces its version and moves it to sourceId.

Value parameters

docId

Document identifier

sourceId

The source the document belongs to

version

Document version at time of indexing

Attributes

Source
PgDocumentRegistry.scala
def unregister(docId: String): Result[Unit]

Unregister a document (deleted from source).

Unregister a document (deleted from source).

Value parameters

docId

Document identifier to remove

Attributes

Source
PgDocumentRegistry.scala

Inherited methods

def contains(docId: String): Result[Boolean]

Check if a document is registered.

Check if a document is registered.

Attributes

Inherited from:
DocumentRegistry
Source
DocumentRegistry.scala
def register(docId: String, version: DocumentVersion): Result[Unit]

Register a document as indexed, under DocumentRegistry.DefaultSourceId.

Register a document as indexed, under DocumentRegistry.DefaultSourceId.

Value parameters

docId

Document identifier

version

Document version at time of indexing

Attributes

Inherited from:
DocumentRegistry
Source
DocumentRegistry.scala

Concrete fields

Attributes

Source
PgDocumentRegistry.scala