Grafana Tempo and OpenTelemetry: distributed tracing on VPS

OpenTelemetry is a graduated CNCF project that unifies the three pillars of observability — traces, metrics, and logs — under a vendor-neutral standard. Unlike Datadog or New Relic agents that lock you into a proprietary contract and format, the OTel Collector receives, transforms, and exports telemetry to any compatible backend. You can switch storage without changing a single line of application code.

SDKs cover Node.js, Python, Go, Java, PHP, Ruby, and most modern languages. W3C TraceContext and B3 context propagators are included: they automatically forward the trace-id between your microservices via HTTP headers, with no extra configuration. The OTel Collector can run as a local agent on each VM or as a centralized gateway — you choose the topology that fits your infrastructure.

Your VPS needs at least 2 GB of RAM (4 GB recommended for production) and root or sudo access. Docker Engine ≥ 24 and Docker Compose v2 must be installed.

Plan a dedicated subdomain — for example grafana.your-domain.com — to expose the Grafana interface behind a TLS reverse proxy (nginx or Caddy). Ports 4317 (OTLP/gRPC) and 4318 (OTLP/HTTP) must be open in your firewall on the private side to receive telemetry from your applications; do not expose them on the public interface. Tempo port 3200 and Grafana port 3000 remain internal to the Docker network.

For storage, plan for at least 10 GB of free space for trace blocks. The default mount at /tmp/tempo does not survive restarts: replace it with a named Docker volume or a dedicated disk mount before going to production.

The data flow follows four steps:

1. Your application instruments its requests with an OTel SDK and exports spans to the OTel Collector (OTLP gRPC port 4317 or HTTP port 4318).
2. The OTel Collector receives, batches, then exports to Tempo via internal OTLP gRPC. It can simultaneously send metrics to Prometheus and logs to Loki.
3. Grafana Tempo stores compressed trace blocks on disk (or S3). It exposes a TraceQL API on port 3200 for Grafana queries.
4. Grafana queries Tempo with TraceQL, displays flamegraphs, service graphs, and span metrics. If Tempo is linked to Loki, clicking a span opens the logs for that service on the same time window.

TraceQL is Tempo's native query language. A typical query looks like: {service.name="api" && status=error && duration>500ms}. You filter by resource attribute, span attribute, status, or duration — and immediately get traces that cross a latency threshold.

Span Metrics is a Tempo pipeline that automatically derives RED metrics (Rate, Error, Duration) from incoming traces, without adding a single counter to your code. These metrics can be scraped by Prometheus and displayed in standard Grafana dashboards.

By default, Tempo stores blocks locally in /tmp/tempo. For a durable installation, two options are available.

Local storage on a Docker volume: this is the recommended configuration for a standalone VPS. The tempo-data volume defined in docker-compose.yml persists data across restarts. Adjust block_retention in tempo.yaml according to your needs: 72 h for a staging environment, 168 h (7 days) to 720 h (30 days) for production. Each hour of compressed traces occupies approximately 100–300 MB depending on span volume.

S3-compatible object storage: for a multi-node stack or uncapped storage, replace the local backend with an s3 backend in tempo.yaml:

storage:
  trace:
    backend: s3
    s3:
      bucket: my-traces-bucket
      endpoint: s3.amazonaws.com
      region: eu-west-1
      access_key: ${S3_ACCESS_KEY}
      secret_key: ${S3_SECRET_KEY}
    wal:
      path: /tmp/tempo/wal

Tempo is compatible with AWS S3, MinIO, Scaleway Object Storage, and any S3 API-compatible backend. The WAL (Write-Ahead Log) always stays on local disk to ensure durability of in-flight traces.

Tempo's compactor merges small blocks and enforces the retention policy. It runs in the background and requires no manual intervention. Monitor the blocks/ folder size with docker exec tempo du -sh /tmp/tempo/blocks to anticipate storage needs.

OTel auto-instrumentation for Node.js automatically captures outgoing HTTP calls, Express/Fastify requests, database queries, and many other libraries without modifying your business logic.

Install the required packages:

npm install @opentelemetry/sdk-node \
             @opentelemetry/auto-instrumentations-node \
             @opentelemetry/exporter-trace-otlp-grpc

Create a tracing.js file to load first:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'grpc://localhost:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Start your application with this file loaded first:

export OTEL_SERVICE_NAME=my-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://your-vps:4317
node --require ./tracing.js app.js

For Docker environments, pass the OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT variables in your docker-compose.yml or Kubernetes manifest. All incoming and outgoing HTTP calls will be automatically traced and correlated by the W3C TraceContext propagator.

The opentelemetry-distro package provides a bootstrap mechanism that automatically installs all instrumentation packages for libraries detected in your environment.

Install and bootstrap:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

Launch your application prefixed with opentelemetry-instrument:

export OTEL_SERVICE_NAME=my-python-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://your-vps:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
opentelemetry-instrument python app.py

Django, Flask, FastAPI, SQLAlchemy, and HTTP clients (requests, httpx, aiohttp) are instrumented automatically. For containerized deployments, define the OTEL_* variables in your environment file and ensure port 4317 of the OTel Collector is accessible from your containers' network.

For manual instrumentation with finer control:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process-order") as span:
    span.set_attribute("order.id", order_id)
    # your business logic

Grafana 10+ lets you define alerts directly on metrics derived from traces. Tempo's Span Metrics automatically generate three metrics per service: traces_spanmetrics_calls_total (request rate), traces_spanmetrics_latency_bucket (latency histogram), and the associated error counter.

To create an SLO on your API error rate:
1. In Grafana, go to Alerting → Alert rules → New alert rule.
2. Choose the Prometheus datasource (or Tempo via Span Metrics) and enter a query like rate(traces_spanmetrics_calls_total{status_code="STATUS_CODE_ERROR", service="my-api"}[5m]) / rate(traces_spanmetrics_calls_total{service="my-api"}[5m]).
3. Set the alert threshold (for example > 0.05 for an error rate above 5%) and the evaluation window.
4. Configure a contact point: Alertmanager, PagerDuty, Slack, or webhook — all integrate from Alerting → Contact points.

For P99 latency, query histogram_quantile(0.99, rate(traces_spanmetrics_latency_bucket{service="my-api"}[5m])) and trigger an alert if the value exceeds your error budget (for example 2 seconds). These trace-based alerts are far more precise than synthetic metric alerts, as they reflect the real experience of each individual request.

No spans arriving in Tempo. First check that the OTel Collector is reachable from your application (telnet your-vps 4317). Check the Collector logs (docker compose logs otel-collector): a connection refused error on tempo:4317 indicates Tempo is not yet started or its internal port is not exposed on the Docker network. Ensure both services share the same network (observability).

Connection refused from application to Collector. On a UFW or firewalld firewall, allow ports 4317 and 4318 only for your application IPs, not for the entire internet. In production, applications and Collector should be on the same private network; never expose the Collector on the public interface without authentication.

Truncated traces or missing spans. The Collector's batch processor groups spans before sending. If your application sends few spans, traces can take up to timeout seconds (5 s by default) to appear in Grafana. For development, reduce timeout: 1s in the batch processor. If spans are systematically absent, verify the context propagator is activated in your SDK (W3CTraceContextPropagator) — without it, calls between services generate orphaned traces.

Tempo consuming too much disk space. Reduce block_retention in tempo.yaml and restart Tempo. The compactor will apply the new policy on the next cycle (every 5 minutes by default). In an emergency, manually delete old blocks in /tmp/tempo/blocks while leaving the WAL intact.

Grafana can't find traces from Loki logs. Verify that your applications inject trace_id into their logs (structured JSON format recommended) and that the Derived fields setting in the Loki datasource points to the correct pattern ("traceId": "(\w+)"). The link URL must point to the Tempo datasource with the correct Grafana datasource identifier.