LLM observability engineering notes · TripleCloud

TripleCloud Engineering

Notes from the team building IceGate — observability on an open lakehouse.

A kitten in a winter parka inspects a stack of ice blocks through a magnifying glass - the clear blocks labelled prompt and tokens have their contents frozen visibly inside, while the plain white blocks hold nothing

LLM Observability

What is LLM observability? Tracing where the payload is the signal

LLM observability is the practice of recording what an LLM application actually did on every request - the prompt, the response, the tool calls, the tokens - so any single run can be reconstructed afterwards. Technically it's distributed tracing. Practically it's a different job, because the payload rather than the timing is the signal. Here's what it records, how it differs from monitoring and evals, and what it changes about your data and your bill.

A kitten in a winter parka building an igloo out of glowing ice blocks labeled OTEL and TRACES, hauling OpenAI API blocks on a sled

OpenTelemetry GenAI semantic conventions: how to instrument an LLM agent

The OpenTelemetry GenAI semantic conventions let you trace an LLM agent the same way you trace the rest of your stack. A hands-on guide: zero-code auto-instrumentation, manual agent and tool spans, token usage and cost, opt-in prompt capture, grouping multi-turn conversations across traces, and where to send the data. None of it ties you to a vendor, so the data stays yours.

One agentic trace of ten spans - five LLM spans priced at the AI rate, five ordinary infrastructure spans priced as plain telemetry

LLM Observability

LLM monitoring and observability: the AI tax on ordinary telemetry

LLM monitoring and observability is billed at one premium AI rate, but agentic apps emit mixed traces: heavy LLM spans interleaved with just as many ordinary ones - HTTP, DB, queues, infra. So your observability cost grows with the number of agents and tool calls - with your success, not with the value of the data. Here's the fix: price AI data as AI data, and everything else as plain telemetry.

Observability data - logs, metrics, traces, events - ingested directly into Iceberg tables on Amazon S3

Can you build observability ingestion on S3 alone - no Kafka, no disks, no coordination layer?

Yes - you can run observability ingestion on S3 alone, with no stateful layer to operate: no Kafka, no local disks, no coordination service. We replaced an Apache Kafka + Flink + OTel pipeline (~$700–800/month at 10 MB/s) with one engine where the data, the WAL, and the Apache Iceberg catalog all live in S3, with durability backed by S3. Here's how it works.

Sign in to save this post