Telemetry architecture

Monozu Cloud services export request, dependency, exception, log, and custom metric telemetry through a central OpenTelemetry Collector on Azure Container Apps. Apps speak OTLP/gRPC to the collector; the collector routes to four Application Insights components by service.name.

Data path

flowchart LR
  subgraph apps [ACA apps]
    Cloud[cloud-mz-env]
    Worker[worker-mz-env]
    Ingest[ingest-mz-env]
    Manage[manage-mz-env]
  end
  Collector["otel-mz-env internal HTTPS/443"]
  subgraph ai [App Insights]
    AIc[appi-cloud]
    AIw[appi-worker]
    AIi[appi-ingest]
    AIm[appi-manage]
  end
  Cloud -->|OTLP gRPC| Collector
  Worker --> Collector
  Ingest --> Collector
  Manage --> Collector
  Collector -->|service.name| AIc
  Collector --> AIw
  Collector --> AIi
  Collector --> AIm
  AIc --> LAW[Log Analytics log-mz-env]
  AIw --> LAW
  AIi --> LAW
  AIm --> LAW

Stdout from containers is logfmt for live tail (az containerapp logs show). WARN+ logs are also forwarded via OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set.

Service names (`service.name` / `AppRoleName`)

Container App	`SERVICE_NAME` / `OTEL_SERVICE_NAME`
Cloud API	`monozu-cloud-backend`
Worker	`monozu-cloud-worker`
Ingestor	`monozu-ingestor`
Management	`monozu-management-backend`

Per-service instrumentation

Each Go service has its own internal/telemetry package (no shared Go module). Common behaviour:

OTLP/gRPC exporters when OTEL_EXPORTER_OTLP_ENDPOINT is set; no-op locally when unset (fail-open)
Fiber: github.com/gofiber/contrib/v3/otel + request ID / access log / tenant_id (no user_id)
SQL: otelsql; outbound HTTP: otelhttp; Azure SDK where used
Worker: job spans + queue depth + DB pool metrics as OTel metrics

Environment variables (apps)

Variable	Purpose
`OTEL_EXPORTER_OTLP_ENDPOINT`	`https://otel-mz-{env}.internal.{cae-domain}`
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`
`OTEL_RESOURCE_ATTRIBUTES`	`service.namespace=monozu,deployment.environment.name={env}`
`OTEL_TRACE_SAMPLE_RATIO`	Head sampling at app (dev `1`, prod `0.2`)
`SERVICE_NAME`	`service.name` resource attribute

Apps do not receive APPLICATIONINSIGHTS_CONNECTION_STRING. Connection strings are collector-only (Bicep deploy secrets + kv-mz-shared-{env} copies).

Collector (dev)

Setting	Value
Image	`otel/opentelemetry-collector-contrib` (pinned tag in Bicep)
Ingress	Internal HTTPS, HTTP/2 (gRPC)
Scale	`minReplicas: 0`, `maxReplicas: 1` (dev)
Routing	`routing` processor on `service.name` → four `azuremonitor` exporters
Health	`:13133` (`extensions.health_check`)

Seed collector App Insights secrets: Infrastructure/scripts/seed-otel-collector-secrets.sh dev.

LAW tables

Table	Source
`AppRequests`	HTTP (`operation_kind=http`) and worker jobs (`operation_kind=job`)
`AppDependencies`	SQL, outbound HTTP, Azure SDK calls
`AppExceptions`	5xx and panic only (not 4xx / validation)
`AppTraces`	Structured logs (WARN+ by default via OTLP)
`AppMetrics`	DB pool gauges, jobs queue depth

Correlation and dimensions

OperationId — W3C trace ID; links requests, dependencies, exceptions, and logs.
tenant_id — UUID in Properties after authentication (never before).
operation_kind — http or job.
request_id — per-request UUID in logs and span attributes.

Sampling

Layer	Dev	Production
App (`OTEL_TRACE_SAMPLE_RATIO`)	`1`	`0.2`
Collector tail sampling (prod later)	passthrough 100%	100% errors + >2s + 20% remainder
App Insights resource (`samplingPercentage`)	100%	100%

Privacy

Never exported: SQL text/arguments, URL query strings, request/response bodies, authorization headers, secrets, raw DSNs, user_id.

Internal M2M URLs

Set in Bicep (apps.bicep):

Cloud → Management: MANAGEMENT_API_URL=https://manage-mz-{env}.internal.{cae-domain}
Management → Cloud: CLOUD_API_URL=https://cloud-mz-{env}.internal.{cae-domain}

Origin only (no /api/v1 suffix). Auth tokens remain in Key Vault.

Edge distinction

Edge appliances send device payload telemetry through the ingestor path. That is separate from this Application Insights pipeline.