Search & Discovery

Hematite provides two complementary search engines: Tantivy for full-text search across structured seeds and unstructured documents, and Qdrant for semantic similarity search via vector embeddings. Both are kept in sync with PostgreSQL CDC events.

Tantivy — Full-Text Search

Built in Rust, Tantivy provides fast full-text search indexing. Hematite's noetic-search crate manages the index lifecycle, while noetic-sync drives updates from PostgreSQL CDC events.

Indexing Pipeline

  1. CDC Event — A write to PostgreSQL (new seed record, updated document) triggers a CDC event.
  2. Event Queue — The sync service queues the event for index processing without blocking the original write.
  3. Index Update — Tantivy processes the event, adding or updating the document in the search index.
  4. Search Ready — The updated content is immediately available via search queries.

Search API

// Full-text search across all indexed content
curl https://myorg.hematite.io/api/v1/search?q=hematite \
  -H "Authorization: Bearer $TOKEN"

// Search with filters
curl 'https://myorg.hematite.io/api/v1/search?q=governance&content_type=blog-post&tags=ai' \
  -H "Authorization: Bearer $TOKEN"

// Paginated results
curl 'https://myorg.hematite.io/api/v1/search?q=hematite&page=2&per_page=20' \
  -H "Authorization: Bearer $TOKEN"

Index Management

// Force index rebuild from PostgreSQL
curl -X POST https://myorg.hematite.io/api/v1/search/rebuild \
  -H "Authorization: Bearer $TOKEN"

// Check index health and statistics
curl https://myorg.hematite.io/api/v1/search/stats \
  -H "Authorization: Bearer $TOKEN"

// Reindex a specific content type
curl -X POST https://myorg.hematite.io/api/v1/content-types/blog-post/reindex \
  -H "Authorization: Bearer $TOKEN"

Qdrant — Vector Embeddings

Semantic similarity search via Qdrant vector embeddings. Used for agent memory retrieval, document similarity, and content discovery.

Collection Structure

agentmem Agent long-term memory entries with tag-based ACL filtering
documents Vector embeddings for uploaded documents and content
content Semantic representations of structured content (seeds, pages)

Semantic Search API

// Semantic similarity search
curl https://myorg.hematite.io/api/v1/search/vector \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "query": "AI governance frameworks",
    "collection": "documents",
    "limit": 5,
    "similarity_threshold": 0.75
  }'

// ACL-filtered search (agent-specific)
curl https://myorg.hematite.io/api/v1/search/vector \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "query": "session context",
    "collection": "agentmem",
    "agent_id": "general-01",
    "limit": 3
  }'

LangCache — Embedding Cache

The LangCache service (:8091) caches embedding results to avoid redundant computation. Configurable thresholds with LRU eviction at Redis level.

// Check cache statistics
curl https://myorg.hematite.io/api/v1/langcache/stats \
  -H "Authorization: Bearer $TOKEN"

// Clear cache for a specific collection
curl -X POST https://myorg.hematite.io/api/v1/langcache/clear \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"collection": "documents"}'

Key principle: Two search engines, one API. Full-text for exact matching and filtering, vector embeddings for semantic discovery — both kept in sync with real-time data changes.

Related Pages