Troubleshooting

Common issues and resolution procedures for Hematite deployments.

Service Won't Start

Symptom systemctl start noetic-kernel fails with exit code 1
Cause Redis connection refused, PostgreSQL unreachable, or port conflict
Fix 
# Check Redis connectivity
redis-cli ping

# Check PostgreSQL connectivity
psql -h localhost -U postgres -c "SELECT 1"

# View kernel logs
journalctl -u noetic-kernel --since "5 min ago"

Endpoint Failover Not Triggering

Symptom Sessions stuck on a degraded endpoint beyond the failover timeout
Cause router.failover_timeout set too high, or standby endpoints not responding to health checks
Fix 
# Check current failover timeout
hematite config get router.failover_timeout

# Lower the timeout (seconds)
hematite config set router.failover_timeout=30

# Force endpoint health check refresh
curl -X POST https://myorg.hematite.io/api/v1/router/refresh \
  -H "Authorization: Bearer $TOKEN"

Tantivy Index Out of Sync

Symptom Search returns stale or missing results after content updates
Cause CDC sync service not running, or index refresh interval too long
Fix 
# Check sync service status
systemctl status noetic-sync

# Force index rebuild
curl -X POST https://myorg.hematite.io/api/v1/search/rebuild \
  -H "Authorization: Bearer $TOKEN"

# Verify search is working
curl https://myorg.hematite.io/api/v1/search?q=test \
  -H "Authorization: Bearer $TOKEN"

Session Timeout Too Early

Symptom Active sessions terminate before completion
Cause session.timeout set too low, or WebSocket connection dropped by a load balancer
Fix 
# Increase session timeout (seconds)
hematite config set session.timeout=7200

# Check WebSocket proxy settings on load balancer
# Ensure idle_timeout is >= session.timeout value

High Memory Usage

Symptom Kernel crate consuming excessive memory (>8GB)
Cause Too many concurrent sessions, large prompt payloads, or memory leak in a specific agent type
Fix 
# Check active session count per tenant
curl https://myorg.hematite.io/api/v1/sessions/stats \
  -H "Authorization: Bearer $TOKEN"

# Limit concurrent sessions per tenant
hematite config set session.max_concurrent=50

# Restart kernel to reclaim memory (sessions resume from Redis)
systemctl restart noetic-kernel

Diagnostic Commands

Use these commands to gather system state for support:

# Full system health report
hematite diagnostics health-report > health.json

# Agent pool status
hematite diagnostics agent-pool-status

# Redis connection info
hematite diagnostics redis-info

# PostgreSQL replication lag
hematite diagnostics pg-replication

Support: If the issue persists after following these procedures, share the output of hematite diagnostics health-report with your support contact.

Related Pages