API Reference

All endpoints are served from a single Bloop instance. Ingest routes use HMAC authentication (project key + request body signature), while query and management routes accept either a Bearer token or a browser session cookie.

Error Tracking

MethodPathAuthDescription
GET /health None Health check
POST /v1/ingest HMAC Submit single error event
POST /v1/ingest/batch HMAC Submit batch (max 50)
GET /v1/errors Bearer / Session List errors (paginated, filterable)
GET /v1/errors/:fingerprint Bearer / Session Error detail + samples
GET /v1/errors/:fingerprint/occurrences Bearer / Session Raw events
POST /v1/errors/:fingerprint/resolve Bearer / Session Mark resolved
POST /v1/errors/:fingerprint/ignore Bearer / Session Mark ignored
POST /v1/errors/:fingerprint/mute Bearer / Session Mute (suppress alerts)
POST /v1/errors/:fingerprint/unresolve Bearer / Session Unresolve
GET /v1/errors/:fingerprint/trend Bearer / Session Per-error hourly counts
GET /v1/errors/:fingerprint/history Bearer / Session Status audit trail
GET /v1/releases/:release/errors Bearer / Session Errors for a release
GET /v1/trends Bearer / Session Global hourly counts
GET /v1/stats Bearer / Session Overview stats

Alerting

MethodPathAuthDescription
GET /v1/alerts Bearer / Session List alert rules
POST /v1/alerts Bearer / Session Create alert rule
PUT /v1/alerts/:id Bearer / Session Update alert rule
DELETE /v1/alerts/:id Bearer / Session Delete alert rule
POST /v1/alerts/:id/channels Bearer / Session Add notification channel
POST /v1/alerts/:id/test Bearer / Session Send test notification

Projects & Auth

MethodPathAuthDescription
GET /v1/projects Session List projects
POST /v1/projects Session Create project
GET /v1/projects/:slug Session Project details
POST /v1/projects/:slug/sourcemaps Bearer / Session Upload source map
POST /v1/tokens Session Create API token
GET /v1/tokens Session List tokens
DELETE /v1/tokens/:id Session Revoke token
PUT /v1/admin/users/:id/role Admin Session Promote/demote user
GET /v1/admin/retention Admin Session Get retention config
PUT /v1/admin/retention Admin Session Update retention config
POST /v1/admin/retention/purge Admin Session Trigger immediate purge

Analytics (feature: analytics)

MethodPathAuthDescription
GET /v1/analytics/spikes Bearer / Session Spike detection (z-score)
GET /v1/analytics/movers Bearer / Session Top movers
GET /v1/analytics/correlations Bearer / Session Correlated error pairs
GET /v1/analytics/releases Bearer / Session Release impact scoring
GET /v1/analytics/environments Bearer / Session Environment breakdown

LLM Tracing (feature: llm-tracing)

MethodPathAuthDescription
POST /v1/traces HMAC Submit single LLM trace
POST /v1/traces/batch HMAC Submit batch of traces (max 50)
PUT /v1/traces/:id HMAC Update a running trace
GET /v1/llm/overview Bearer / Session Summary: traces, tokens, cost, errors
GET /v1/llm/usage Bearer / Session Hourly token/cost by model
GET /v1/llm/latency Bearer / Session p50/p90/p99 latency by model
GET /v1/llm/models Bearer / Session Per-model breakdown
GET /v1/llm/traces Bearer / Session Paginated trace list
GET /v1/llm/traces/:id Bearer / Session Full trace + span hierarchy
GET /v1/llm/settings Bearer / Session Content storage policy
PUT /v1/llm/settings Bearer / Session Update content storage policy
GET /v1/llm/search Bearer / Session Full-text search over traces
POST /v1/llm/traces/:id/scores Bearer / Session Submit a score for a trace
GET /v1/llm/traces/:id/scores Bearer / Session Get scores for a trace
GET /v1/llm/scores/summary Bearer / Session Score summary across traces
GET /v1/llm/prompts Bearer / Session List prompt versions with metrics

Query Parameters for GET /v1/errors

ParamTypeDescription
releasestringFilter by release
environmentstringFilter by environment
sourcestringFilter by source (ios/android/api)
routestringFilter by route
statusstringFilter by status (unresolved/resolved/ignored)
sincei64Epoch ms lower bound on last_seen
untili64Epoch ms upper bound on last_seen
sortstringSort: last_seen (default), total_count, first_seen
limiti64Page size (default 50, max 200)
offseti64Pagination offset

Event Contract

The IngestEvent schema defines the shape of every error event submitted to Bloop. Required fields are marked; all others are optional.

json
{
  "timestamp": 1700000000,
  "source": "ios",
  "environment": "production",
  "release": "1.2.0",
  "app_version": "1.2.0",
  "build_number": "42",
  "route_or_procedure": "/api/checkout",
  "screen": "PaymentScreen",
  "error_type": "RuntimeError",
  "message": "Connection refused at port 5432",
  "stack": "RuntimeError: Connection refused\n  at connect (db.swift:42)",
  "http_status": 500,
  "request_id": "req-abc-123",
  "user_id_hash": "sha256:a1b2c3...",
  "device_id_hash": "sha256:d4e5f6...",
  "fingerprint": "custom-fp-override",
  "metadata": {
    "os_version": "17.2",
    "device_model": "iPhone 15"
  }
}

Hard caps on payload sizes:

  • Total payload: 32 KB
  • stack: 8 KB
  • metadata: 4 KB
  • message: 2 KB

Events exceeding these limits are rejected with 400 Bad Request.

API Tokens

API tokens provide scoped, long-lived bearer credentials for CI pipelines, AI agents, and scripts. They are always scoped to a single project and grant fine-grained access based on the scopes assigned at creation time.

Creating Tokens

Create tokens from the dashboard (Settings → API Tokens) or via the API. The plaintext token is SHA-256 hashed before storage and is only shown once at creation time — copy it immediately.

bash
# Create a token (requires session auth)
curl -X POST http://localhost:5332/v1/tokens \
  -H "Cookie: session=..." \
  -H "Content-Type: application/json" \
  -d '{"name":"CI deploy","project_id":"my-project","scopes":["errors:read","sourcemaps:upload"]}'

# Use the token
curl http://localhost:5332/v1/errors \
  -H "Authorization: Bearer bloop_abc123..."

Available Scopes

ScopeGrants access to
errors:readList/get errors, occurrences, trends, stats
errors:writeResolve, ignore, mute, unresolve errors
sourcemaps:uploadUpload source maps
alerts:readList alert rules and channels
alerts:writeCreate, update, delete rules and channels
analytics:readQuery analytics endpoints (spikes, movers, correlations)
llm:readQuery LLM tracing endpoints (overview, usage, traces, search, scores, prompts)
llm:writeSubmit scores for LLM traces

Bearer tokens are project-scoped — you don't need to pass a project_id query parameter. The project is determined automatically from the token. If a token lacks the required scope, the request returns 403 Forbidden.

Alerting

Bloop supports configurable alert rules that evaluate incoming events and notify your team via webhooks, Slack, or email when conditions are met.

Rule Types

TypeFires when
New IssueA never-seen fingerprint appears for the first time
ThresholdAn error exceeds N occurrences within a configurable time window
SpikeError rate exceeds a z-score threshold compared to the rolling baseline
LLM Cost SpikeHourly LLM cost exceeds a dollar threshold
LLM Error RateLLM error rate exceeds a percentage threshold
LLM LatencyLLM p90 latency exceeds a millisecond threshold (optionally per model)

LLM Alert Rule Examples

The LLM-specific alert rules use the same /v1/alerts CRUD endpoints with new rule_type values.

bash
# Alert when hourly LLM cost exceeds $10
curl -X POST http://localhost:5332/v1/alerts \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "LLM cost spike",
    "rule_type": "llm_cost_spike",
    "threshold_dollars": 10.0,
    "window_hours": 1,
    "project_id": "my-project"
  }'

# Alert when LLM error rate exceeds 5%
curl -X POST http://localhost:5332/v1/alerts \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "LLM error rate",
    "rule_type": "llm_error_rate",
    "threshold_percent": 5.0,
    "window_hours": 1,
    "project_id": "my-project"
  }'

# Alert when gpt-4o p90 latency exceeds 5 seconds
curl -X POST http://localhost:5332/v1/alerts \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPT-4o slow responses",
    "rule_type": "llm_latency",
    "threshold_ms": 5000,
    "window_hours": 1,
    "model": "gpt-4o",
    "project_id": "my-project"
  }'

LLM Alert Rule Configuration

Rule TypeRequired FieldsOptional Fields
llm_cost_spike threshold_dollars (float), window_hours (int) project_id
llm_error_rate threshold_percent (float), window_hours (int) project_id
llm_latency threshold_ms (int), window_hours (int) model (string), project_id

Notification Channels

Each alert rule can deliver notifications to multiple channels:

Creating a Rule

bash
curl -X POST http://localhost:5332/v1/alerts \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High error rate",
    "rule_type": "threshold",
    "threshold": 100,
    "window_secs": 3600,
    "project_id": "my-project"
  }'

Cooldown

After an alert fires, it enters a cooldown period to prevent alert fatigue during error storms. The default cooldown is 900 seconds (15 minutes) between re-fires for the same rule and fingerprint. This is configurable via alerting.cooldown_secs in config.toml.

Email Alerts (SMTP)

Bloop can send alert notifications via email. This requires SMTP configuration in config.toml or via environment variables.

toml
[smtp]
enabled = true
host = "smtp.example.com"
port = 587
username = "[email protected]"
password = "your-password"
from = "[email protected]"
starttls = true

Once SMTP is configured, add an email channel to any alert rule via the dashboard (Settings → Alerts → Add Channel → Email) or via the API:

bash
curl -X POST http://localhost:5332/v1/alerts/1/channels \
  -H "Content-Type: application/json" \
  -H "Cookie: session=YOUR_SESSION_TOKEN" \
  -d '{"config": {"type": "email", "to": "[email protected]"}}'

Email alerts require smtp.enabled = true. If SMTP is not configured, creating an email channel will succeed but emails will not be sent. Use the Test button on the alert rule to verify email delivery.

Projects

Bloop supports multiple projects, each with its own isolated namespace. Every project gets its own API key (used for HMAC-signed ingest), alerts, source maps, and retention settings. Projects are identified by a URL-safe slug and a human-readable name.

bash
# Create project
curl -X POST http://localhost:5332/v1/projects \
  -H "Cookie: session=..." \
  -H "Content-Type: application/json" \
  -d '{"name":"My Mobile App","slug":"my-mobile-app"}'

# List projects
curl http://localhost:5332/v1/projects -H "Cookie: session=..."

The response includes the project's auto-generated HMAC key. Pass this key as X-Project-Key in your SDK configuration for ingestion authentication.

A default project is created automatically on first startup using the legacy hmac_secret from config. SDKs that don't send X-Project-Key are routed to this default project for backward compatibility.

User Management

Bloop has two user roles: Admin and User. The first user to register is automatically promoted to Admin. Admins can manage projects, alert rules, API tokens, source maps, users, invites, and retention settings.

Promote / Demote Users

bash
# Promote user to admin
curl -X PUT http://localhost:5332/v1/admin/users/2/role \
  -H "Cookie: session=..." \
  -H "Content-Type: application/json" \
  -d '{"role":"admin"}'

Self-demotion is not allowed. You cannot remove your own admin role. This ensures there is always at least one admin who can manage the system.

Data Retention

Bloop automatically prunes old data based on configurable retention settings. By default, raw events are pruned after 7 days while aggregates are kept indefinitely. Retention is configurable per-project and managed via admin-only endpoints.

bash
# Get current retention settings
curl http://localhost:5332/v1/admin/retention -H "Cookie: session=..."

# Update retention
curl -X PUT http://localhost:5332/v1/admin/retention \
  -H "Cookie: session=..." \
  -H "Content-Type: application/json" \
  -d '{"raw_events_days": 14}'

# Trigger immediate purge
curl -X POST http://localhost:5332/v1/admin/retention/purge -H "Cookie: session=..."

Automatic pruning runs every hour by default. The interval is configurable via retention.prune_interval_secs in config.toml.

Source Maps

Upload source maps for JavaScript and TypeScript stack trace deobfuscation. Source maps are uploaded per project per release. Once uploaded, stack traces for that release are automatically deobfuscated in the dashboard and API responses.

bash
# Upload a source map
curl -X POST http://localhost:5332/v1/projects/my-app/sourcemaps \
  -H "Authorization: Bearer bloop_..." \
  -F "release=1.2.3" \
  -F "file=@dist/app.js.map"

The dashboard provides a toggle to switch between the original (deobfuscated) and minified (raw) stack trace views when source maps are available for the matching project and release.

Source maps are stored compressed (gzip) and cached in memory for fast lookups. The release field in the upload must exactly match the release field sent with error events for matching to work.

Analytics (DuckDB)

Bloop includes an optional analytics engine powered by DuckDB for advanced error analysis. DuckDB reads the existing SQLite database directly via its sqlite_scanner extension — there is no data migration, no schema changes, and no write contention with the main SQLite writer.

Feature-gated: Analytics requires building with cargo build --features analytics. It is not included in the default build.

Analysis Types

Five analysis endpoints are available:

  1. Spike Detection — z-score based anomaly detection on hourly error rates. Returns fingerprints where the hourly count deviates significantly from the rolling mean.
  2. Top Movers — Errors with the biggest absolute rate change between the current and previous time windows. Surfaces rapidly increasing or decreasing errors.
  3. Correlations — Pairs of errors that tend to co-occur in the same hourly windows (Pearson correlation ≥0.7, requiring ≥6 data points). Useful for root cause discovery.
  4. Release Impact — Scores each release by error introduction and reduction. Combines new fingerprints and error count deltas into a composite impact score.
  5. Environment Breakdown — Error distribution and percentiles (P50/P90/P99 hourly rates) broken down by environment.
bash
# Get spike detection results
curl "http://localhost:5332/v1/analytics/spikes?hours=24&threshold=2.5" \
  -H "Authorization: Bearer bloop_..."

Analytics data is read-only from DuckDB. There is no write contention with the main SQLite writer. Results are cached in memory (configurable via analytics.cache_ttl_secs, default 60 seconds).

LLM Tracing

Monitor AI and LLM API calls across your application. Track token usage, costs, latency, and errors per model and provider. Hierarchical traces group related LLM calls together with nested spans for generation, tool use, retrieval, and custom operations.

Feature-gated: LLM tracing requires building with cargo build --features llm-tracing AND setting llm_tracing.enabled = true at runtime in config.toml.

Ingest API

Submit traces with nested spans. All ingest endpoints use HMAC authentication (same as error ingest).

MethodPathDescription
POST /v1/traces Submit single trace with nested spans
POST /v1/traces/batch Submit up to 50 traces per batch
PUT /v1/traces/:id Update a running trace (status, output, end time)

Trace JSON Contract

json
{
  "id": "trace-001",
  "session_id": "session-abc",
  "user_id": "user-123",
  "name": "chat-completion",
  "prompt_name": "summarize-v2",
  "prompt_version": "1.3.0",
  "status": "completed",
  "input": "User prompt text",
  "output": "AI response text",
  "metadata": {"key": "value"},
  "started_at": 1700000000000,
  "ended_at": 1700000001200,
  "spans": [
    {
      "id": "span-001",
      "parent_span_id": null,
      "span_type": "generation",
      "name": "gpt-4o call",
      "model": "gpt-4o",
      "provider": "openai",
      "input_tokens": 100,
      "output_tokens": 50,
      "cost": 0.0025,
      "latency_ms": 1200,
      "time_to_first_token_ms": 300,
      "status": "ok",
      "error_message": null,
      "input": "prompt text",
      "output": "completion text",
      "metadata": {}
    }
  ]
}

Span types: generation, tool, retrieval, custom.

Validation limits:

  • Trace ID: max 128 characters
  • Max 100 spans per trace
  • Max 50 traces per batch

Cost format: Specify costs in dollars (float). They are stored internally as microdollars (integer) to avoid floating-point precision issues. $0.0025 becomes 2500 micros.

Query Endpoints

EndpointDescription
GET /v1/llm/overviewSummary: total traces, spans, tokens, cost, error rate
GET /v1/llm/usageHourly token/cost breakdown by model
GET /v1/llm/latencyp50/p90/p99 latency by model, avg TTFT
GET /v1/llm/modelsPer-model: calls, tokens, cost, error rate
GET /v1/llm/tracesPaginated trace list with filters and sorting
GET /v1/llm/traces/:idFull trace + span hierarchy
GET /v1/llm/searchFull-text search over trace names, sessions, users
POST /v1/llm/traces/:id/scoresSubmit a score for a trace
GET /v1/llm/traces/:id/scoresGet all scores for a trace
GET /v1/llm/scores/summaryAggregated score summary across traces
GET /v1/llm/promptsList prompt versions with performance metrics

Shared Query Parameters

ParamTypeDefaultDescription
project_idstringfrom tokenFilter by project
hoursi6424Time window (max 720)
limiti6450Page size (max 200)
modelstringFilter by model name
providerstringFilter by provider
statusstringFilter by trace status (completed, error, running)
session_idstringFilter by session
sortstringnewestSort order: newest, oldest, cost_desc, latency_desc, tokens_desc
offseti640Pagination offset
bash
# Get LLM usage overview
curl "http://localhost:5332/v1/llm/overview?hours=24" \
  -H "Authorization: Bearer bloop_..."

# Get traces for a specific model
curl "http://localhost:5332/v1/llm/traces?model=gpt-4o&limit=20" \
  -H "Authorization: Bearer bloop_..."

LLM Proxy API

Zero-instrumentation tracing by acting as a reverse proxy to LLM providers. Simply point your LLM client at bloop's proxy endpoint instead of the provider directly. All calls are automatically traced with prompts, completions, token usage, cost, and latency captured server-side.

MethodPathAuthDescription
POST /v1/proxy/openai/{*path} Provider API Key Proxy OpenAI API calls with automatic tracing

How It Works

  1. Your app sends request to /v1/proxy/openai/chat/completions with your OpenAI API key
  2. bloop forwards request to OpenAI, capturing prompt content
  3. OpenAI returns response (streaming or non-streaming)
  4. bloop captures completion, token usage, calculates cost
  5. Trace is created and sent to pipeline
  6. Original response returned to your app (unchanged)

Setup

bash
# Point your OpenAI client at bloop
# Python
import openai
client = openai.OpenAI(
    api_key="sk-your-openai-key",
    base_url="http://localhost:5332/v1/proxy/openai"
)

# TypeScript
import OpenAI from "openai";
const client = new OpenAI({
    apiKey: "sk-your-openai-key",
    baseURL: "http://localhost:5332/v1/proxy/openai"
});

# Or via environment variable (most SDKs)
export OPENAI_BASE_URL=http://localhost:5332/v1/proxy/openai

Configuration

Proxy behavior is controlled via config.toml:

toml
[llm_tracing.proxy]
enabled = true
providers = ["openai"]
openai_base_url = "https://api.openai.com/v1"
capture_prompts = true
capture_completions = true
capture_streaming = true

Content Storage

LLM prompts and completions can contain sensitive data. Content storage is configurable per-project with three policies:

PolicyWhat's storedUse case
none (default)Prompts and completions stripped at ingest, never hit diskProduction with sensitive data
metadata_onlyOnly metadata JSON keptStructured tags without prompt content
fullEverything stored (prompts, completions, metadata)Development, debugging, audit trails
bash
# Check current policy
curl "http://localhost:5332/v1/llm/settings?project_id=my-app" \
  -H "Authorization: Bearer bloop_..."

# Update policy
curl -X PUT "http://localhost:5332/v1/llm/settings?project_id=my-app" \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{"content_storage": "full"}'

Content storage policy is enforced at ingest time. Changing from none to full only affects future traces — previously ingested traces with stripped content cannot be recovered.

LLM Experiments (A/B Testing)

Create and manage A/B tests for prompt versions. Compare metrics like latency, cost, success rate, and user feedback scores across prompt variants to determine the best performing version.

Experiment Endpoints

MethodPathDescription
GET/v1/llm/experimentsList all experiments for a project
POST/v1/llm/experimentsCreate a new experiment with variants
GET/v1/llm/experiments/:idGet experiment details with metrics
GET/v1/llm/experiments/:id/compareCompare two variants with statistical analysis

Creating an Experiment

bash
# Create an A/B test for prompt variants
curl -X POST "http://localhost:5332/v1/llm/experiments" \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Summarizer Prompt Test",
    "prompt_name": "summarizer",
    "description": "Testing detailed vs concise prompts",
    "variants": [
      {
        "name": "Baseline",
        "version": "v1",
        "prompt_template": "Summarize: {text}",
        "traffic_percentage": 50
      },
      {
        "name": "Detailed",
        "version": "v2",
        "prompt_template": "Provide a detailed summary in 3-5 sentences: {text}",
        "traffic_percentage": 50
      }
    ]
  }'

Comparing Variants

bash
# Compare baseline vs treatment variant
curl "http://localhost:5332/v1/llm/experiments/experiment-123/compare?baseline=v1&treatment=v2" \
  -H "Authorization: Bearer bloop_..."

# Response includes:
# - Latency improvement %
# - Cost improvement %
# - Success rate improvement %
# - Statistical significance (p-value, confidence level)
# - Recommendation

Experiment Metrics

Each variant tracks the following metrics automatically:

Full-text search over LLM traces. Searches across trace names, session IDs, and user IDs to quickly find specific traces or groups of related traces.

Search Endpoint

MethodPathAuthDescription
GET /v1/llm/search Bearer / Session Full-text search over traces

Query Parameters

ParamTypeDefaultDescription
qstringrequiredSearch query (matches trace name, session_id, user_id)
project_idstringfrom tokenFilter by project
hoursi6424Time window to search within
limiti6450Max results to return
bash
# Search for traces by name
curl "http://localhost:5332/v1/llm/search?q=chat-completion&hours=48" \
  -H "Authorization: Bearer bloop_..."

# Search by session ID
curl "http://localhost:5332/v1/llm/search?q=session-abc&limit=20" \
  -H "Authorization: Bearer bloop_..."

Response

json
{
  "traces": [
    {
      "id": "trace-001",
      "name": "chat-completion",
      "session_id": "session-abc",
      "user_id": "user-123",
      "status": "completed",
      "total_tokens": 150,
      "total_cost_micros": 2500,
      "latency_ms": 1200,
      "started_at": 1700000000000
    }
  ],
  "total": 1,
  "query": "chat-completion"
}

LLM Scoring

Attach quality scores to LLM traces for evaluation and monitoring. Scores are named (e.g. "quality", "relevance", "safety") and normalized to a 0.0–1.0 range. Use scores to track LLM output quality over time, compare prompt versions, and build evaluation datasets.

Scoring Endpoints

MethodPathAuthDescription
POST /v1/llm/traces/:id/scores Bearer / Session Submit a score for a trace
GET /v1/llm/traces/:id/scores Bearer / Session Get all scores for a trace
GET /v1/llm/scores/summary Bearer / Session Aggregated score summary across traces

Submit a Score

bash
curl -X POST http://localhost:5332/v1/llm/traces/trace-001/scores \
  -H "Authorization: Bearer bloop_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "quality",
    "value": 0.85,
    "comment": "Good response, accurate and well-structured"
  }'

Score Body

FieldTypeRequiredDescription
namestringYesScore name (e.g. "quality", "relevance"). Max 128 characters.
valuefloatYesScore value, must be between 0.0 and 1.0 inclusive.
commentstringNoOptional comment explaining the score.

Get Scores for a Trace

bash
curl "http://localhost:5332/v1/llm/traces/trace-001/scores" \
  -H "Authorization: Bearer bloop_..."

Response

json
{
  "scores": [
    {
      "id": 1,
      "trace_id": "trace-001",
      "name": "quality",
      "value": 0.85,
      "comment": "Good response, accurate and well-structured",
      "created_at": "2024-01-15T10:30:00Z"
    },
    {
      "id": 2,
      "trace_id": "trace-001",
      "name": "relevance",
      "value": 0.92,
      "comment": null,
      "created_at": "2024-01-15T10:31:00Z"
    }
  ]
}

Score Summary

Get aggregated statistics for all score names across traces in a time window.

bash
curl "http://localhost:5332/v1/llm/scores/summary?hours=48" \
  -H "Authorization: Bearer bloop_..."

Summary Query Parameters

ParamTypeDefaultDescription
project_idstringfrom tokenFilter by project
hoursi6424Time window

Response

json
{
  "scores": [
    {
      "name": "quality",
      "count": 142,
      "avg_value": 0.78,
      "min_value": 0.12,
      "max_value": 0.99
    },
    {
      "name": "relevance",
      "count": 89,
      "avg_value": 0.85,
      "min_value": 0.45,
      "max_value": 1.0
    }
  ],
  "window_hours": 48
}

Multiple scores with the same name can be submitted for a single trace (e.g. from different evaluators). The summary endpoint aggregates across all scores with a given name.

Prompt Version Logging

Track which prompt templates and versions are used across your LLM traces. By tagging traces with prompt_name and prompt_version, you can monitor performance regressions across prompt iterations and compare metrics between versions.

Tagging Traces with Prompt Versions

Include the optional prompt_name and prompt_version fields when submitting traces via POST /v1/traces or POST /v1/traces/batch.

json
{
  "id": "trace-042",
  "name": "summarize-article",
  "prompt_name": "summarize-v2",
  "prompt_version": "1.3.0",
  "session_id": "session-xyz",
  "status": "completed",
  "started_at": 1700000000000,
  "ended_at": 1700000001500,
  "spans": [...]
}

Prompt Metrics Endpoint

MethodPathAuthDescription
GET /v1/llm/prompts Bearer / Session List prompt versions with aggregated performance metrics

Query Parameters

ParamTypeDefaultDescription
project_idstringfrom tokenFilter by project
hoursi6424Time window
bash
curl "http://localhost:5332/v1/llm/prompts?hours=72" \
  -H "Authorization: Bearer bloop_..."

Response

json
{
  "prompts": [
    {
      "prompt_name": "summarize-v2",
      "prompt_version": "1.3.0",
      "trace_count": 482,
      "avg_latency_ms": 1350.5,
      "avg_tokens": 175.2,
      "total_cost_micros": 1205000,
      "error_rate": 0.02,
      "last_seen": "2024-01-15T14:22:00Z"
    },
    {
      "prompt_name": "summarize-v2",
      "prompt_version": "1.2.0",
      "trace_count": 1204,
      "avg_latency_ms": 1580.3,
      "avg_tokens": 210.8,
      "total_cost_micros": 3612000,
      "error_rate": 0.05,
      "last_seen": "2024-01-14T09:15:00Z"
    }
  ]
}

Both prompt_name and prompt_version are optional. Traces without these fields are excluded from the /v1/llm/prompts aggregation. Use consistent naming (e.g. semver for versions) to get meaningful comparisons across prompt iterations.