OmniSight: Multi-Modal AI Incident Intelligence Fabric

1. The Problem Space: Alert Storms & Blind Spots

Before detailing our software architecture, consider the physical reality of monitored environments:

• Temporal Cascades: A single mechanical bearing failure produces a vibrational spike at t = 0s, thermal elevation at t = 12s, high-frequency acoustic squeal at t = 19s, and visible vapor/smoke at t = 45s. Traditional threshold rules treat these as 4 disconnected events, triggering 4 alerts to different engineering queues.
• Cross-Camera Occlusion: An unauthorized personnel intrusion or forklift violation passes across 6 consecutive camera coverage zones over 90 seconds. Without cross-camera re-identification and spatial topology awareness, operators get 6 independent alarms with no coherent tracking history.
• Unbounded LLM Hallucination Risk: Standard Generative AI agents cannot be directly wired to industrial actuators (e.g. emergency sprinkler systems or production line stop switches) without a mathematically deterministic safety guardrail.

2. System Architecture: Edge-to-Cloud Distributed Fabric

To maintain sub-200ms latency without incurring massive cloud streaming costs, OmniSight employs a hybrid edge-compute topology. Edge nodes (NVIDIA Jetson AGX Orin) handle high-fps video decoding, tensor-accelerated object detection, and acoustic FFT transforms locally. Only structured metadata vectors and triggered incident clips are streamed upstream over gRPC to the central LangGraph orchestration cluster.

OmniSight Multi-Stage Processing Pipeline

• STAGE 01 Edge Ingest RTSP H.264/H.265 + MQTT telemetry at 30 FPS
• STAGE 02 TensorRT CV YOLOv8 + ByteTrack feature extraction
• STAGE 03 Event Windowing DBSCAN temporal cluster & deduplication
• STAGE 04 Vector RAG Qdrant hybrid sparse/dense SOP lookup
• STAGE 05 LangGraph Core 6-agent stateful reasoning loop
• STAGE 06 Dispatch & Act Deterministic policy check + PagerDuty/PLC

3. The 6-Agent LangGraph Orchestration Engine

Unlike monolithic LLM wrappers that dump raw prompts into a single chat completion endpoint, OmniSight delegates investigation steps across six specialized autonomous agents arranged as a directed cyclical graph in LangGraph:

4. Multi-Modal Fusion: Real-Time Sliding Window Correlator

The bedrock of OmniSight's reliability is its temporal correlation layer. The following production FastAPI snippet demonstrates how high-frequency edge events are ingested asynchronously, matched against active incident contexts, and deduplicated using spatial-temporal decay weights:

import time
from typing import Dict, List, Optional
from pydantic import BaseModel, Field
import numpy as np

class AnomalyEvent(BaseModel):
    event_id: str
    source_id: str          # e.g., "CAM_04_SOUTH_BAY" or "IOT_TEMP_VALVE_12"
    modality: str           # "vision" | "acoustic" | "thermal" | "rfid"
    severity_score: float   # 0.0 to 1.0
    detected_class: str     # e.g., "smoke_plume", "overheat_critical"
    timestamp: float = Field(default_factory=time.time)
    embedding: Optional[List[float]] = None

class IncidentCluster:
    def __init__(self, incident_id: str, seed_event: AnomalyEvent):
        self.incident_id = incident_id
        self.events: List[AnomalyEvent] = [seed_event]
        self.created_at = seed_event.timestamp
        self.last_updated = seed_event.timestamp
        self.max_severity = seed_event.severity_score

    def should_merge(self, event: AnomalyEvent, max_delta_s: float = 30.0) -> bool:
        """Determines if a new sensor event belongs to this temporal incident cluster."""
        time_delta = abs(event.timestamp - self.last_updated)
        if time_delta > max_delta_s:
            return False
        
        # Cross-modality temporal boost
        is_same_location = event.source_id.split("_")[1] == self.events[0].source_id.split("_")[1]
        return is_same_location or (time_delta < 5.0 and event.severity_score > 0.85)

    def add_event(self, event: AnomalyEvent):
        self.events.append(event)
        self.last_updated = event.timestamp
        self.max_severity = max(self.max_severity, event.severity_score)

class SpatialTemporalEngine:
    def __init__(self, correlation_window_sec: float = 45.0):
        self.window_sec = correlation_window_sec
        self.active_clusters: Dict[str, IncidentCluster] = {}

    def ingest_event(self, event: AnomalyEvent) -> Optional[IncidentCluster]:
        now = time.time()
        # Clean up expired clusters
        self.active_clusters = {
            cid: cluster for cid, cluster in self.active_clusters.items()
            if (now - cluster.last_updated) <= self.window_sec
        }

        # Check existing active clusters
        for cluster in self.active_clusters.values():
            if cluster.should_merge(event, max_delta_s=self.window_sec):
                cluster.add_event(event)
                return cluster

        # Create new incident cluster if anomaly is actionable
        if event.severity_score >= 0.65:
            new_id = f"INC-{int(now * 1000)}-{event.source_id[:6]}"
            new_cluster = IncidentCluster(incident_id=new_id, seed_event=event)
            self.active_clusters[new_id] = new_cluster
            return new_cluster

        return None

5. Hybrid RAG & SOP Grounding

Industrial safety regulations mandate that autonomous systems never invent containment procedures. OmniSight solves this via a hybrid retrieval engine indexing complete plant safety manuals, chemical datasheets (MSDS), and electrical schematics into Qdrant.

When an incident cluster is formed, the Knowledge Agent synthesizes an incident summary query and executes reciprocal rank fusion (RRF) between dense semantic vectors and sparse BM25 tokens:

from qdrant_client import QdrantClient
from qdrant_client.http import models

def fetch_incident_sop(client: QdrantClient, query_text: str, dense_vec: list, sparse_indices: list, sparse_values: list):
    """
    Executes hybrid dense + sparse retrieval with Reciprocal Rank Fusion (RRF)
    guaranteeing deterministic citation of standard operating procedures.
    """
    results = client.query_points(
        collection_name="industrial_sops",
        prefetch=[
            models.Prefetch(
                query=dense_vec,
                using="dense",
                limit=10
            ),
            models.Prefetch(
                query=models.SparseVector(indices=sparse_indices, values=sparse_values),
                using="sparse",
                limit=10
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=3
    )
    
    citations = []
    for hit in results.points:
        citations.append({
            "sop_id": hit.payload["sop_id"],
            "section": hit.payload["section_title"],
            "mandatory_actions": hit.payload["actions"],
            "safety_level": hit.payload["required_ppe"],
            "score": hit.score
        })
    return citations

6. Real-World Benchmarks & Comparative Analysis

We benchmarked OmniSight across a synthetic simulation reproducing a multi-tier industrial incident: continuous RTSP video streams from 50 cameras, 1,200 IoT temperature sensors, and periodic false-alarm noise injections (sun glare, steam release, vibration artifacts).

System Comparison Matrix

7. Key Engineering Challenges Solved

8. Production Takeaways & Next Horizons

Building OmniSight confirmed that the true challenge in physical-world AI is not model accuracy in isolation—it is context coordination and deterministic safety. A 98% mAP vision model is useless if it drowns operators in unverified alerts or hallucinates remediation steps during a chemical spill.

By decoupling raw perception (YOLOv8 + TensorRT on Edge) from structured reasoning (LangGraph + Hybrid RAG on Cloud) and enforcing a deterministic safety firewall, OmniSight provides an industrial-grade blueprint for autonomous physical operations.

Stack

FastAPI, LangGraph, Python 3.11, TensorRT, YOLOv8, Qdrant, PostgreSQL, DeepStream

Deployment

Edge: NVIDIA Jetson AGX Orin • Cloud: Docker, Kubernetes, Redis Pub/Sub

Repositories

github.com/kulkarnishub377/OmniSight • Interactive Live Demo