Monitoring AI Systems in Production: Observability, Drift, and LLM Telemetry

1. The Two-Dimensional Observability Challenge

Traditional software systems operate under deterministic assumptions: given a specific input state and code path, the system returns an expected output or throws an HTTP exception. Standard application performance monitoring (APM) tools measure CPU utilization, memory pressure, HTTP status codes, and latency distributions (p50, p95, p99).

Production Artificial Intelligence—encompassing both classical predictive models and generative Large Language Models (LLMs)—introduces a complex second dimension: semantic non-determinism. A service can return HTTP 200 OK with 80ms latency while delivering completely inaccurate, toxic, or hallucinated responses that destroy user trust.

Effective AI observability requires monitoring both Service Health and Inference Quality:

+--------------------------------------------------------------------------+
|                     Two-Dimensional AI Observability                     |
+------------------------------------+-------------------------------------+
                                     |
           +-------------------------+-------------------------+
           |                                                   |
           v                                                   v
+-------------------------------+           +-------------------------------+
|    Operational Telemetry      |           |     Cognitive Telemetry       |
|    (Service Reliability)      |           |     (Model Intelligence)      |
+-------------------------------+           +-------------------------------+
| - Request Latency (p95 / p99) |           | - Semantic Drift & Divergence |
| - Time to First Token (TTFT)  |           | - Groundedness & Faithfulness |
| - Inter-Token Latency (ITL)   |           | - Context Precision (RAG)     |
| - Token Consumption & Spend   |           | - Hallucination Indicators    |
| - Provider 429 Rate Limits    |           | - Prompt Injection Attempts   |
| - GPU Utilization / KV Cache  |           | - User Thumbs Up/Down Ratio   |
+-------------------------------+           +-------------------------------+

2. Generative AI Metrics: Streaming Latency and Cost Attribution

Streaming generative systems require specialized operational metrics that standard request/response timers cannot measure:

1. Time to First Token (TTFT): The elapsed duration between the user dispatching a prompt and the first output token streaming to the client. This metric reflects prompt preprocessing latency, queue wait time, and initial KV-cache population.
2. Inter-Token Latency (ITL): The average time required to generate each subsequent token. High ITL causes perceptible typing stutter.
3. Token Usage and Cost Accounting: Tracking prompt tokens, completion tokens, and cache-hit tokens per tenant, route, and model family.

OpenTelemetry Semantic Conventions for Generative AI

To avoid vendor lock-in, modern engineering teams instrument AI pipelines using the official OpenTelemetry GenAI Semantic Conventions:

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('ladestack-ai-service');

export async function executeAiCompletion(prompt: string, tenantId: string) {
  return tracer.startActiveSpan('gen_ai.chat', {
    attributes: {
      'gen_ai.system': 'openai',
      'gen_ai.request.model': 'gpt-4o',
      'gen_ai.request.temperature': 0.2,
      'gen_ai.request.max_tokens': 1024,
      'app.tenant.id': tenantId,
    },
  }, async (span) => {
    const startTime = performance.now();
    try {
      const response = await aiClient.chat.completions.create({
        model: 'gpt-4o',
        messages: [{ role: 'user', content: prompt }],
      });

      const latencyMs = performance.now() - startTime;
      const usage = response.usage;

      // Instrument standard OTel GenAI telemetry metrics
      span.setAttributes({
        'gen_ai.response.id': response.id,
        'gen_ai.response.model': response.model,
        'gen_ai.response.finish_reasons': [response.choices[0].finish_reason],
        'gen_ai.usage.input_tokens': usage?.prompt_tokens ?? 0,
        'gen_ai.usage.output_tokens': usage?.completion_tokens ?? 0,
        'gen_ai.usage.total_tokens': usage?.total_tokens ?? 0,
        'gen_ai.latency.total_ms': latencyMs,
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return response.choices[0].message.content;
    } catch (error: any) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  });
}

3. Retrieval-Augmented Generation (RAG) Quality Evaluation

In RAG pipelines, generation quality is directly constrained by retrieval precision. Monitoring RAG systems requires isolating the retrieval phase from the synthesis phase:

User Query ---> [ Embedding Model ] ---> Vector Database Query
                                                |
                                                v  (Evaluate Retrieval Quality)
                                       Retrieved Context Documents
                                                |
                                                v  (Evaluate Synthesis Quality)
                                        LLM Prompt Generation
                                                |
                                                v
                                         Final Response

The RAG Triad Metrics

1. Context Relevance (Retrieval): Did the vector similarity search return chunks that directly answer the query, or is the prompt filled with irrelevant noise?
2. Groundedness / Faithfulness (Synthesis): Does every factual assertion in the model’s generated answer originate from the retrieved context chunks, or did the model invent external claims?
3. Answer Relevance (Outcome): Does the generated response directly satisfy the user’s intent?

$$\text{Faithfulness Score} = \frac{\text{Number of Claims Supported by Context}}{\text{Total Claims in Answer}}$$

Crucial Engineering Caveat: Automated model-based evaluators (LLM-as-a-judge) provide valuable statistical approximations of drift and regression across thousands of production traces, but cannot prove absolute mathematical correctness for any single answer. Automated evaluations must be paired with human review workflows.

4. Privacy, PII Redaction, and Data Retention

Capturing production AI telemetry creates severe compliance risks under GDPR, HIPAA, and SOC 2. Prompts frequently contain personally identifiable information (PII), proprietary source code, or protected health information.

Telemetry Pipeline Hygiene

1. Client-Side Redaction: Pass raw user text through an in-memory redaction engine (e.g., Microsoft Presidio) to mask social security numbers, credit cards, email addresses, and phone numbers before shipping spans to telemetry backends.
2. Configurable Payload Logging: Production tracing must permit logging token counts, latencies, and metadata without storing raw prompt and response strings in plain text.
3. Retention Policies: Configure short retention windows (e.g., 7–14 days) on raw trace logs, with aggregated metrics retained long-term for trend analysis.

5. Alerting Strategy: Avoiding Alert Fatigue

AI monitoring generates voluminous high-frequency noise. Setting alerts on individual hallucination scores or single-request token spikes guarantees alert fatigue for on-call engineers.

Actionable SLO-Based Alerting

6. Incident Response and Circuit Breaker Topologies

When an AI provider suffers latency spikes, service outages, or model degradation, the system must degrade gracefully:

Inference Gateway
       |
       v
+-------------------------------+
|  Circuit Breaker (Resilience) |
+---------------+---------------+
                |
                +---> Healthy: Primary Provider (e.g., GPT-4o)
                |
                +---> Degraded / 429: Secondary Provider (e.g., Claude 3.5 Sonnet)
                |
                +---> Outage / Trip: Local Fallback (Exact Semantic Cache / Rule Engine)

1. Semantic Caching: High-frequency queries should query an in-memory vector cache (e.g., Redis with similarity search). If a query has an identical or near-identical embedding to a recent request, return the cached answer with zero provider latency.
2. Provider Failover: Decouple business logic through an orchestration gateway capable of instantly rerouting traffic from a failing provider to a healthy secondary provider.

Production AI Monitoring Checklist

• OpenTelemetry GenAI semantic conventions implemented across all inference endpoints.
• Streaming metrics instrumented: Time to First Token (TTFT) and Inter-Token Latency (ITL).
• Token consumption and cost attribution broken down by tenant and model.
• PII redaction active before shipping traces to centralized telemetry collectors.
• RAG pipelines instrumented with Context Relevance and Faithfulness evaluations.
• Provider rate limit (429) alerts configured with exponential backoff and jitter.
• Multi-provider circuit breaker enabled to handle upstream AI service outages.
• Semantic caching enabled for high-frequency identical queries.
• Actionable alerts configured around Service Level Objectives (SLOs), not transient anomalies.