Helicone LLM-observability population survey (21-host self-hosted population)

NuClide Research · 2026-05-10

Summary

Third platform in the AI-observability cross-platform sweep, after Phoenix (25% unauth) and Langfuse (0% unauth). Helicone is a YC-backed LLM observability + AI gateway product, SOC 2 + GDPR compliant, primarily SaaS but offers a Docker-based self-host path.

DCWF KSAT coverage

Auto-derived from DCWF AI work-role rule files (ksat-tag).

• 672 (AI Test & Evaluation Specialist): K7003, K7004, S7068, S7070, S7075, T5904, T5919
• 733 (AI Risk & Ethics Specialist): K7040, K7051, T5854, T5868
• overlap (Common AI KSATs (all 5 roles)): K1157, K1158, K1159, K22, K6311, K6935, K7003

Population finding: 21 self-hosted Helicone instances total. Almost an order of magnitude smaller than Phoenix or Langfuse. A SaaS-first product. Of those:

• 5 are Helicone’s own AWS infrastructure (api.helicone.ai, jawn.helicone.ai, eu.api.helicone.ai)
• ~16 are self-hosted operator instances
• 0 unauthenticated (auth model is mandatory via BetterAuth or Supabase; no master “auth off” toggle)
• 2 secondary-surface IP-shadow finds (1 MailHog empty, 1 Cockpit login-required)

The cross-platform pattern continues: Helicone is closer to Langfuse than Phoenix. Mandatory-auth by design. The 25% Phoenix rate remains a Phoenix-specific phenomenon.

Reproduce with VisorBishop: visorbishop -i helicone-hosts.txt -ip-shadow See VisorBishop or visorplus bishop.

Discovery dorks

Combining dorks 2+3 and deduplicating: 21 unique hosts.

The order-of-magnitude smaller population than Phoenix (377) or Langfuse (1,333) reflects Helicone’s product strategy: they push customers to the cloud SaaS by default. The Docker self-host path exists (documented at docs/getting-started/self-host/docker.mdx) but is rarely exercised at population scale.

Population

Self-hosted operator identifications visible in hostnames:

Source-level auth audit

Helicone supports two auth backends, both mandatory:

1. Supabase (default for the cloud product; available for self-hosters via web/.env.example). Uses Postgres + Supabase Auth’s row-level security. JWT-based session tokens issued by Supabase.
2. BetterAuth (introduced for the Docker self-host path; web/.env.hosted.example and web/.env.example.better-auth). Next.js-native auth using cookie-based sessions signed with BETTER_AUTH_SECRET.

The .env.example files include a literal default secret:

BETTER_AUTH_SECRET="MKUcaeqyMD7UBkGeFYY5hwxKS1aB6Vsi"

This identical string appears in three .env.example files (web/, valhalla/jawn/, docker/). Operators who copy any of those verbatim and don’t rotate the secret have predictable session-cookie signatures. A forgery primitive for anyone with the value. The official docker.mdx documentation does explicitly call out using openssl rand -base64 32 for production deployments, but the Quick Start docker-run command in the same doc omits BETTER_AUTH_SECRET entirely, defaulting to change-me-in-production per the documented env-table.

The MinIO bundled with Helicone’s helicone-all-in-one Docker image defaults to:

S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin

If an operator exposes the bundled S3 to the public internet (port 9080 in the documented setup), these defaults stand. The MinIO API would accept these credentials and grant full read/write to the request-response-storage bucket. Which is where Helicone stores the bodies of every captured LLM request and response.

Latent-primitive summary:

Probing for these primitives against the live operator population would constitute credential testing on third-party infrastructure. Outside the read-only scope of this survey. Documented for the threat-model writeup.

Auth-posture probe across the 21 hosts

All 21 hosts tested with POST /v1/oai/chat/completions (Helicone’s OpenAI-compatible proxy endpoint) without auth headers. Results:

The SPA-HTML-on-POST pattern is the same shadowing behavior we identified on Phoenix port 6006. The Next.js router returns the marketing page for any route the React app doesn’t explicitly handle, which makes external probing harder than a clean “404 from FastAPI” response would. False-positive 200s during initial probing.

IP-direct-shadow sweep

Applying Methodology Insight #12. 11-port nmap sweep against the 19 unique IPs in the Helicone population (excluding two AWS LB-only addresses):

Finding: MailHog at 188.34.196.197:8025 (api-helicone.dev.bridge-mt.net likely)

The Hetzner DE host running Helicone also has MailHog running on port 8025. The same pattern we found on the Teetsh operator in the Phoenix sweep. Currently GET /api/v2/messages?limit=0 returns {"total":0}. The store is empty. The exposure is latent: any time the application sends mail (password resets, account verification, notifications), captured messages become publicly readable until the operator clears the store.

Finding: Cockpit at 62.171.190.103:9090 (vmi3071145.contaboserver.net, Contabo DE)

Cockpit is the Red Hat web-based Linux server admin UI. Login-required at /login. Not unauth-exploitable directly, but binding Cockpit to 0.0.0.0:9090 instead of 127.0.0.1:9090 (or putting it behind a VPN) is a hardening miss. Default Cockpit auth uses system credentials; weak SSH passwords on the host become Cockpit login attempts at scale.

Neither finding is a Helicone vulnerability. Both are co-located operator services exposed alongside Helicone. The same IP-direct-shadow class pattern we documented in the Phoenix case study.

Cross-platform synthesis (preliminary, 3 platforms in)

The pattern across three platforms:

• Phoenix is the outlier. Its default-no-auth shipping behavior is unique among the three observed so far.
• Helicone has the smallest population by far: SaaS-first product strategy.
• Both Langfuse and Helicone enforce auth by design. Operators cannot turn auth off, only configure which credential providers are accepted.
• The IP-shadow methodology surfaces operator-side misconfigurations regardless of the platform’s auth strength. Operators who deploy ANY of these platforms on a Linux host tend to expose other services on that same host.

Latent primitives in Helicone are different in kind from Phoenix’s:

Helicone’s latent primitives require either (a) credential testing or (b) cryptographic forgery. Both materially harder than Phoenix’s “the GraphQL schema is reachable unauthenticated” finding. Different threat models.

Next steps (research, not disclosure-yet)

1. Shodan harvest ✓ 21 hosts total
2. Source-level auth audit ✓ BetterAuth + Supabase, mandatory; default-secret pattern documented
3. Auth-posture probe across 21 hosts ✓ 0 unauth findings
4. IP-direct-shadow sweep ✓ 2 minor finds (MailHog empty, Cockpit login-required)
5. Default-secret prevalence probe (deferred). Would require active forgery testing
6. LangSmith population survey, next platform in the cross-platform sweep
7. Cross-platform synthesis document, after LangSmith + 2-3 more land

Evidence pack

~/recon/2026-05-10-llm-sweep/helicone/

• helicone-hosts.tsv: deduplicated Shodan hit list (21 unique URL/hostname pairs)
• helicone-ips.txt: 19 unique IPs
• helicone-probe-results.tsv: per-host auth-posture probe results
• helicone-ip-shadow.{nmap,gnmap,xml}: IP-shadow port sweep

Cross-references:

• phoenix-llm-observability-survey-2026-05-10.md (25% unauth)
• langfuse-llm-observability-survey-2026-05-10.md (0% unauth)
• Methodology Insight #12