Observability troubleshooting

Endpoint names show only GET /api/v1

Symptom: AppRequests.Name (and panels like Slowest endpoints or legacy Slowest requests) list only GET /api/v1 / POST /api/v1 instead of route templates such as GET /api/v1/incidents/:id.

Cause: HTTP telemetry middleware runs on the Fiber group (/api/v1). Before the fix, the span name was taken from c.Route().Path before c.Next(), when Fiber still points at the group route.

Fix (code): Middleware now calls setResolvedHTTPRoute after c.Next() in backend, ingestor, and management. Deploy the updated service images.

Verify after deploy:

AppRequests
| where TimeGenerated > ago(1h)
| where AppRoleName == "monozu-cloud-backend"
| summarize count() by Name
| order by count_ desc

Expect distinct names per route template, not a single GET /api/v1 bucket.

Note: Legacy dash-monozu-obs-{env} and dash-mz-app-{env} both group by Name — no dashboard change required once telemetry is fixed.

Infrastructure dashboard — all panels “No data”

dash-mz-infra-{env} uses Azure Monitor Metrics (queryType: Azure Monitor), not LAW. App dashboard (dash-mz-app-{env}) can work while infra shows empty panels.

1. Redeploy infra dashboard

Infra dashboard is deployed from main.bicep (after the apps module), not shared.bicep. Run the infrastructure pipeline after changing monozu-infra.dashboard.json.

Confirm placeholders were substituted — in Portal, open dash-mz-infra-{env} → definition JSON and check azureMonitor.resources[] entries are objects with real subscription, resourceGroup, resourceName, and region (not __ACA_CLOUD_NAME__ etc.). Each Metrics target must list exactly one resource object (apps live in different RGs; multi-resource batch queries return InvalidSubscriptionId: 'providers').

1b. Empty Resource picker in panel editor

If the Azure Monitor Resource field is empty when editing a panel:

1. Confirm each azureMonitor.resources[] entry is a structured object (not a string like $app or an ARM ID).
2. Confirm all five object fields are present and substituted after deploy.
3. Confirm region matches the ACA/SQL region (__METRICS_REGION__ → location in Bicep, typically polandcentral).

1c. InvalidSubscriptionId: 'providers'

Cause: one Metrics target listed multiple Container Apps from different resource groups. Fix: one target per resource (refs A–E). Redeploy after regenerating from _build_dashboards.py.

1d. Received invalid query parameter with empty interval

Cause: Metrics targets omit timeGrain, so Portal Grafana sends interval= empty and Azure returns 400. Panels may start working after editing because the UI fills Time grain to auto. Fix: set "timeGrain": "auto" on every azureMonitor target.

2. Confirm metrics exist in Azure Portal

For the same time range as the dashboard:

1. Container App → Metrics → CpuPercentage or Replicas
2. SQL database → Metrics → DTU percentage

If Portal metrics are empty, the Grafana panels will be empty too (scale-to-zero on dev is common for ACA).

3. resultFormat on Azure Monitor targets

Infra panels require resultFormat: "time_series" (charts) or "table" (stat panels) on each azureMonitor target. Regenerate from _build_dashboards.py and redeploy if panels were created before this fix.

4. Dev scale-to-zero

containerAppMinReplicas = 0 on dev — ACA CPU/memory/replicas may be empty when idle. SQL DTU should still show data. If SQL is also empty, suspect deploy/placeholder issues (steps 1–2), not replica count.

Post-deploy checklist

Check	Expected
dash-mz-infra-{env} metric resources	Structured objects with subscription/RG/name/region
ACA/SQL panel targets	resources: [{...}] objects + top-level region
ACA Metrics blade (when app running)	Data in last 24h
SQL Metrics blade	DTU data always
Queue depth panel	Value when worker ≥1 replica emits monozu.jobs.queue.depth

Empty LAW tables

1. Confirm OTEL_EXPORTER_OTLP_ENDPOINT points at the internal collector FQDN on each Container App.
2. Confirm the OTel collector (otel-mz-{env}) is provisioned and healthy (Infrastructure/scripts/verify-dev-otel-health.sh dev).
3. Wait 2–5 minutes after deploy for first ingestion into LAW.
4. Check the service is receiving traffic (HTTP) or the worker is running (jobs).
5. Verify you are querying the correct workspace: log-mz-{env}.

Datasource / panel errors

• Datasource UID must be exactly azure-monitor.
• Monitoring Reader is required on rg-monozu-shared-{env} and the app resource groups — not subscription-wide Reader.
• KQL panels need resultFormat: "table" for tables and "time_series" for charts.

KQL casts and filters

• Compare ResultCode numerically: toint(ResultCode) < 500, not string comparison.
• Multi-select filters use $__contains(column, $variable) with allValue: "all".
• HTTP panels filter tostring(Properties.operation_kind) == "http"; jobs use "job".

Missing tenant_id

tenant_id appears only after authentication. Unauthenticated routes and /health will not have it. /health is excluded from request telemetry.

Sampling expectations

Environment	OTEL_TRACE_SAMPLE_RATIO	Expected trace volume
dev	1	~100% of requests/dependencies
prd	0.2	~20% of successful paths

5xx, panic, exceptions, and custom metrics are always exported. Low prd volume on success paths is normal.

Dropped telemetry

When the collector is scaled to zero (dev) or unreachable, apps fail open — requests continue; OTLP export is dropped. Scale the collector to ≥1 replica and confirm OTEL_EXPORTER_OTLP_ENDPOINT on apps.

Missing ACA preview metrics

CpuPercentage / MemoryPercentage are preview metrics. The infrastructure dashboard includes UsageNanoCores and WorkingSetBytes fallbacks. If all ACA panels are empty, verify Monitoring Reader on rg-monozu-cloud-{env} (and ingest/manage for those apps).

Queue depth stale or zero

monozu.jobs.queue.depth is emitted every minute by the worker only. Scale worker to ≥1 replica. Native Azure QueueMessageCount is hourly — do not use it for operational backlog.

Raw container logs

az containerapp logs show -n cloud-mz-dev -g rg-monozu-cloud-dev --follow

Replace app name and resource group for worker/ingest/manage.