Docs / Monitoring

Monitoring and health

Choose the signal that matches your responsibility: public uptime, your tenant's delivery state, or full self-hosted telemetry.

Need	Use	Authentication
Is the process reachable?	`GET /api/health`	None
Can it serve real traffic?	`GET /api/ready`	None
Are my subscriptions delivering?	`GET /api/monitoring/status`	Entra management token
Full traces, metrics, logs	Azure Monitor and optional OTLP	Self-hosted operator configuration

Hosted customers

Public uptime probes

curl -fsS https://tdckbprod-flex-prod.azurewebsites.net/api/health
curl -fsS https://tdckbprod-flex-prod.azurewebsites.net/api/ready

/health is liveness only. Use /ready for alerting because it also checks required configuration and Cosmos. Both responses are payload-free, sanitized, and marked Cache-Control: no-store.

Tenant delivery status

curl -fsS https://tdckbprod-flex-prod.azurewebsites.net/api/monitoring/status \
  -H "Authorization: Bearer $ACCESS_TOKEN"

The response aggregates healthy, failing, unknown, and disabled subscriptions and includes each subscription's last delivery. unknown means no retained delivery result yet—not a failure. See the response contract.

Prometheus blackbox exporter

There is intentionally no shared public /metrics endpoint: global labels could reveal other tenants' volume, platforms, failures, or capacity. Probe readiness safely instead:

scrape_configs:
  - job_name: permylastwebhook-ready
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: [https://tdckbprod-flex-prod.azurewebsites.net/api/ready]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Alert when probe_success == 0 for three consecutive probes. Never place an ingest key in a Prometheus target URL.

Self-hosted OpenTelemetry

Application Insights v3 provides the primary OpenTelemetry pipeline. Add an outbound OTLP/HTTP exporter without disabling Azure Monitor:

OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example:4318
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>

The base endpoint expands to /v1/traces, /v1/metrics, and /v1/logs. Signal-specific variables override it:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example:4318/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://collector.example:4318/v1/metrics
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://collector.example:4318/v1/logs
OTEL_EXPORTER_OTLP_METRICS_HEADERS=Authorization=Bearer%20metrics-token

Credential rules

Remote collectors must use HTTPS. HTTP is accepted only for loopback collectors. Put credentials in percent-encoded headers stored in Key Vault—not URL userinfo, query strings, source, or logs.

OTel Collector to Prometheus

receivers:
  otlp:
    protocols:
      http:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Point the app at the collector's HTTPS OTLP receiver, then configure Prometheus to scrape otel-collector:8889. Discover metric names from the collector because SDK versions can change them.

Recommended alerts

Condition	Window	Action
Readiness probe fails	3 consecutive probes	Check Azure status, deployment, and Cosmos.
Tenant status is degraded	2 polls	Inspect failing subscriptions and Last delivery.
HTTP 5xx > 1%	15 minutes	Correlate requests, exceptions, and deployment.
Teams dependency failures > 5%	15 minutes	Check Microsoft service health and destination validity.
Cosmos throttling > 0	5 minutes	Inspect RU per request and burst shape.

Telemetry troubleshooting

Azure Monitor works but collector is empty: confirm OTLP/HTTP, TLS trust, header encoding, and all /v1/* paths.
Only one signal is missing: check that signal's endpoint and header override.
An invalid exporter endpoint disables that exporter without taking down the app or Application Insights.
Do not use org IDs, destination URLs, keys, approval tokens, Redis URLs, or payload fields as labels or span attributes.

Delivery problem?

Monitoring says degraded? Follow the end-to-end troubleshooting decision tree.