AI observability tier: Small platforms population sweep (Lunary, OpenLIT, Pezzo)

NuClide Research · 2026-05-10

Summary

Phase 1 finishing pass. Three smaller AI-observability platforms surveyed in a single batch:

DCWF KSAT coverage

Auto-derived from DCWF AI work-role rule files (ksat-tag).

• 672 (AI Test & Evaluation Specialist): K7003, K7044, S7068, S7070, S7075, T5904
• 733 (AI Risk & Ethics Specialist): K7040, K7051, S7067, T5854, T5868, T5893
• overlap (Common AI KSATs (all 5 roles)): K1158, K1159, K22, K6311, K6935, K7003

None of these platforms have a single unauthenticated instance in the public-internet population. The cross-platform synthesis is now decisive: Phoenix is the sole observability platform shipping with default-no-auth.

Reproduce with VisorBishop: visorbishop -i hosts.txt -ip-shadow See VisorBishop or visorplus bishop.

Lunary

lunary.ai. Open-source LLM observability + prompt management. YC-backed.

Population: 6 Shodan hits, 1 confirmed Lunary instance.

Auth posture on the one confirmed instance: all sensitive endpoints return 401 {"message":"Invalid access token"}. The publicly-exposed instance at genesysappliedresearch.com enforces auth on every protected route.

Default secret in source? Looked at the lunary-ai/lunary repo .env.example. No static literal secrets in committed examples; operators get JWT_SECRET=changeme placeholder values that won’t function until rotated. Better than Helicone’s literal BETTER_AUTH_SECRET="MKUcaeqyMD7UBkGeFYY5hwxKS1aB6Vsi" pattern.

OpenLIT

openlit.io. Open-source LLM/GenAI observability with built-in eval/playground/prompt-management.

Population: 23 Shodan hits, 23 confirmed OpenLIT instances. 100% auth-fronted.

All 23 hosts return identical behavior:

NextAuth.js cookie-based session middleware. Every API endpoint is wrapped in the auth middleware that redirects unauthenticated requests to the login flow. 0 unauthenticated finds.

Hosting distribution: 11 AWS / 4 Azure / 2 GCP / 2 Oracle / 1 Huawei / 1 OVH / 1 DigitalOcean / 1 BT.

IP-shadow find: 124.71.61.247 (Huawei Cloud China)

The Huawei Cloud China OpenLIT instance also exposes node_exporter on port 9100 unauthenticated. Standard Prometheus exporter, dumps full host metrics:

• CPU/memory/disk/network usage
• Process list (with process_* metrics)
• Go runtime stats (GC, goroutines)
• File handles, network connections

Not a credential leak directly but enables follow-on targeting: attacker learns the host’s load patterns, disk usage trends, and process count without authenticating. Combined with the public-facing OpenLIT (which itself is auth-protected), this gives an attacker the operator’s deployment fingerprint at no cost.

This is the same IP-direct-shadow class pattern as the Phoenix population’s reputacion.digital host: the platform itself is properly configured, but co-located services on the same IP escape the platform’s auth layer because they’re separate services bound to the public interface.

Pezzo

pezzo.ai. Open-source LLMOps platform (prompt management, observability, dataset versioning). Originally TypeScript/Nest.js.

Population: 3 hits via http.title:"Pezzo", 1 confirmed Pezzo instance.

The 65-hit http.html:"pezzo" dork is mostly noise (people named Pezzo, products with “Pezzo” as a substring, etc.). Only 3 hits with a literal <title>Pezzo</title>:

The single Pezzo instance has the same Next.js SPA-shadowing pattern as Phoenix and Helicone. Every endpoint returns the SPA HTML regardless of path. Probing the actual backend at port 3000 returned a 405 (Method Not Allowed) on GraphQL, indicating the API is reachable but requires the right method (POST). Without an API key the requests will fail.

Not exploitable from outside. Auth is enforced at the GraphQL layer; the SPA-shadow is a hardening problem (it makes it harder for legitimate users to debug their connection, not easier for attackers).

Traceloop / OpenLLMetry and HoneyHive

Both returned 0 Shodan hits with the natural dorks. These platforms either:

• Run primarily as exporter libraries (the Traceloop OpenLLMetry SDK sends to their cloud)
• Use proprietary protocols that don’t surface in HTTP fingerprints
• Have very small self-hosted populations

Not enough signal to survey at population scale.

IP-direct-shadow sweep (30 unique IPs across the 3 platforms)

Same 11-port nmap sweep as the Phoenix/Langfuse/Helicone/LangSmith surveys.

Findings: 1 critical, 0 noteworthy elsewhere.

Compare to prior platforms:

The pattern: Phoenix operators have a 27% rate of co-located unauth services. Every other observability platform’s operator population is at single-digit percentages.

Cross-platform synthesis (Phase 1 complete)

After 7 platforms surveyed (Phoenix + Langfuse + Helicone + LangSmith + Lunary + OpenLIT + Pezzo), the picture stabilizes:

The class-level finding is decisive: Phoenix is the only observability platform in this cohort that ships with default-no-auth. The 25% Phoenix unauth rate at population scale is not a “this is hard to deploy securely” problem. It’s a “Phoenix specifically ships with auth off” problem. Every other vendor in the same product category has made the opposite design choice.

The cross-platform synthesis will document this as Methodology Insight #13: Shipping defaults are load-bearing for population-scale security posture. Phoenix’s PHOENIX_ENABLE_AUTH=False default produces 94 publicly-readable trace stores including patient health data (Lillia), biodefense MCM agent prompts, and ~5.5B tokens of customer LLM traffic. The same product class shipped with auth-required defaults produces 0 unauthenticated instances at 4-5× the population size.

Next steps

1. Lunary survey ✓
2. OpenLIT survey ✓
3. Pezzo survey ✓
4. Traceloop / HoneyHive scoping ✓ (insufficient Shodan signal)
5. Cross-platform SYNTHESIS document, pull all seven platform surveys into one cross-cuts analysis
6. Phase 2. Depth+breadth deep-dives per the phase plan
7. Phase 3. Meta-fingerprinter tool

Evidence pack

~/recon/2026-05-10-llm-sweep/{lunary,openlit,pezzo}/

• Per-platform host lists, probe results, IP-shadow nmap output
• small-platforms-ip-shadow.{nmap,gnmap,xml}: combined IP-shadow sweep across 30 unique IPs

Cross-references:

• phoenix-llm-observability-survey-2026-05-10.md: the outlier
• langfuse-llm-observability-survey-2026-05-10.md
• helicone-llm-observability-survey-2026-05-10.md
• langsmith-llm-observability-survey-2026-05-10.md
• Methodology Insight #12