Data Platform Overview

Hematite manages both structured and unstructured data through a unified stack. Structured content (seeds, templates, org configs) lives in Redis for fast access and PostgreSQL for persistence. Unstructured assets (documents, media, raw text) are stored with metadata indexing and full-text search.

Data Types

Structured Data Content type definitions, seed records, template variables, org configurations, task records, project metadata
Unstructured Data Documents (PDF, DOCX), images, audio/video files, raw text blobs, CSV/JSON imports
Vector Embeddings Semantic representations of documents, agent memory entries, and content stored in Qdrant for similarity search
Search Indexes Tantivy full-text indexes synced from PostgreSQL via CDC — kept current without blocking writes

Storage Architecture

+------------------+     +------------------+     +------------------+
|   Redis          |     |   PostgreSQL     |     |   Object Store   |
|   (State/Cache)  |     |   (Persistence)  |     |   (Files/Media)  |
+--------+---------+     +--------+---------+     +--------+---------+
         |                         |                         |
         v                         v                         v
+---------------------------------------------------------------+
|                    Hematite Data Layer                        |
|                                                               |
|  +-----------+  +-----------+  +-----------+                  |
|  | Structured|->| Unstructured|->| Search    |                 |
|  | (seeds,   |  | (docs,     |  | (Tantivy  |                 |
|  | templates)|  | media)     |  | + Qdrant) |                 |
|  +-----------+  +-----------+  +-----------+                  |
|                                                               |
|  Ingestion → Validation → Transformation → Published         |
+---------------------------------------------------------------+

Structured Data Lifecycle

  1. DEFINE — Content types are defined with fields, types, and validation rules. Seeds populate the content type with actual data records.
  2. STORE — Structured data is cached in Redis for fast access (substrate cache, session state) and persisted in PostgreSQL for durability.
  3. INDEX — Tantivy search indexes are updated via PostgreSQL CDC sync events, keeping full-text search current without blocking writes.
  4. RENDER — Content is rendered through templates into published pages. The visual builder maps structured fields to visual blocks.
  5. PUBLISH — Generated HTML is served from CDN with cache-busting via content hashes in filenames.

Unstructured Data Lifecycle

  1. INGEST — Files are uploaded or imported from external sources (APIs, databases, file drops). Metadata is extracted automatically.
  2. STORE — Binary content goes to object storage with metadata indexed in PostgreSQL. Hot data cached in Redis.
  3. INDEX — Text extraction (OCR for images, PDF parsing) feeds Tantivy full-text index. Embeddings generated and stored in Qdrant.
  4. SEARCH — Full-text search across documents via Tantivy. Semantic similarity search via Qdrant vector embeddings.
  5. SERVE — Files served from CDN with org-scoped access control. Metadata available through REST API.

Data Access Patterns

// Search structured content (Tantivy)
curl https://myorg.hematite.io/api/v1/search?q=hematite \
  -H "Authorization: Bearer $TOKEN"

// Semantic search via Qdrant
curl https://myorg.hematite.io/api/v1/search/vector \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"query": "AI governance frameworks", "limit": 5}'

// Upload unstructured file
curl -X POST https://myorg.hematite.io/api/v1/files/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@document.pdf" \
  -F "metadata={\"content_type\": \"report\", \"tags\": [\"governance\"]}"

// Ingest from external API
curl -X POST https://myorg.hematite.io/api/v1/ingest/pipeline \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "source": "api",
    "url": "https://external-api.com/data",
    "schema_detection": true,
    "target_type": "blog-post"
  }'

Key principle: One platform manages all data types — structured seeds, unstructured documents, vector embeddings, search indexes — through a unified ingestion and transformation pipeline.

Related Pages