Skip to main content
If you use Prometheus for monitoring and the Grafana stack for dashboards, you can scrape Checkly’s metrics for every check type from a dedicated, secured endpoint. Any other scraper that sends a bearer token, such as an OpenTelemetry Collector, Grafana Alloy or the Datadog Agent, reads the same endpoint.
A plan that includes the Prometheus integration, otherwise the endpoint returns 403.
Looking for the v1 endpoint? See Legacy v1 endpoint.

Set up

  1. Go to Account settings > Integrations and find the Prometheus card. It shows your endpoint URL, your bearer tokens and a scrape config sample. If the card shows a Create Prometheus endpoint button instead, click it first; the URL and tokens appear once the endpoint exists. Prometheus integration card with the Create Prometheus endpoint button Prometheus integration card in account settings
  2. Under Bearer tokens, click Add token and copy the value. The full value is shown only once, right after you create it. An account can have at most two tokens, so you can rotate: add a token, point your scrape config at it, then revoke the old one.
  3. Add the checkly job under scrape_configs in your prometheus.yml and replace the placeholders with your account ID and token. It matches the sample on the card, plus scrape_timeout: 30s.
  1. Restart Prometheus. Metrics appear after the first scrape.

Options

Add these query parameters to the endpoint URL, or under params in the scrape config. The sample above turns on includeRetryAttempts and locationLabelEnabled, like the sample on the card. The endpoint rejects any other query parameter, and values other than true or false, with 400. Each combination of parameters is cached separately.

Check metrics

Together these metrics cover every check type. Each one also carries the labels listed under Labels. The Labels column below lists the extra ones. For example, a passing check reads:
checkly_check_status can be useful for viewing the current status of a check, whereas checkly_check_result_total can be useful for calculating overall statistics.
checkly_check_result_total counts check results for statistics and alerting. It is not a billing measure and does not match the usage shown on the Billing page or by checkly account usage summary.

Labels

Heartbeat metrics

Heartbeat monitors are passive checks that wait for your scheduled jobs and automated processes to send a ping. The Prometheus exporter includes Heartbeat monitors in the standard checkly_check_status and checkly_check_result_total metrics, plus two heartbeat-specific gauges designed for dead man’s switch alerting.

How heartbeats map to checkly_check_result_total

Heartbeat monitors count in checkly_check_result_total over the same hourly window as other check types. The status label values map to heartbeat event states as follows:

Dead man’s switch PromQL examples

Alert when a heartbeat ping is overdue

Compare the current time against the last success timestamp. If the difference exceeds the expected interval, the job has missed its window:

Alert with a safety margin

Multiply the interval by a margin such as 1.5 to avoid false alarms from minor delays:

Grafana table of heartbeat health

Show how long ago each heartbeat was last seen:

Private location metrics

The Prometheus exporter also contains metrics for monitoring private locations. Use them to make sure your private locations have enough Checkly Agent instances running to execute all of your checks, and to drive agent autoscaling. Every private location metric carries the labels private_location_name, private_location_slug_name and private_location_id. Check runs flow through two queues: main carries Browser checks, Multistep checks and Playwright Check Suites, and uptime carries API checks, URL monitors and the TCP, DNS, ICMP, SSL, gRPC and Traceroute monitors. For metrics scraped from the agent container itself, including an opt-in per-check set with the checkly_agent_ prefix, see Agent metrics.

Behavior

  • Caching and limits. Responses are cached for 60 seconds, so a scrape interval below 60 seconds fetches the same data. Responses are gzip compressed when the scraper asks for it, which Prometheus does by default. The endpoint allows 50 requests per minute per account. Above that it returns 429 with Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset headers. If you run multiple Prometheus instances, each instance scrapes independently. Align their scrape intervals where possible.
  • Errors. An unknown token or a lapsed account returns 401. A revoked token can keep working for up to 10 minutes. Tokens do not expire. A plan without the Prometheus integration returns 403.
  • Timeouts. A scrape fails as a whole if any part of the backing query fails. The query times out after 25 seconds, and a scrape that arrives while another one is refreshing the cache waits for that refresh. For accounts with many checks, keep scrape_timeout at 30s or higher.
  • Hourly window. Counters and histograms reset every hour at HH:01:30, for example 14:01:30. Handle the resets with the rate or increase functions. Gauges built from runs reset too. After the reset and until a check’s next run:
    • checkly_ssl_monitor_days_until_expiry, checkly_icmp_monitor_packet_loss_ratio and checkly_traceroute_monitor_hops read 0.
    • checkly_check_http_status_code, checkly_grpc_monitor_status_code, checkly_traceroute_monitor_destination_reached, checkly_ssl_monitor_info, checkly_ssl_monitor_hostname_verified, checkly_ssl_monitor_chain_trusted, checkly_ssl_monitor_ocsp_stapled and checkly_ssl_monitor_key_size_bits disappear.
    • With locationLabelEnabled, each location reads 0 or is absent until it runs, so checks that run round-robin read 0 on most locations.
    • A monitor that runs every hour reads 0 from the reset until its run. One that runs less often reads 0 in every hour without a run.
    An alert on the raw checkly_ssl_monitor_days_until_expiry gauge below a threshold therefore fires after every reset until the monitor runs again. Alert on its highest value over a range longer than the monitor’s interval instead, for example with max_over_time. checkly_time_to_ssl_expiry_seconds and checkly_check_status read the current status and do not reset.
  • Histograms. Histograms expose only the +Inf bucket. Use _sum / _count averages as in Histogram averages, not histogram_quantile(). A missing web vital counts as 0 and pulls averages down.
  • Tag parsing. Checkly does not clean up tag keys. The label name is the run of letters, digits and underscores right before the first colon, and the value is everything after that colon: my-env:prod becomes env="prod", and a:b:c becomes a="b:c". A key that starts with a digit produces an invalid label name, and a tag whose key matches a built-in label name is ignored. See the Prometheus data model for valid label names.
  • Retries. Retry attempts are left out unless you set includeRetryAttempts=true. checkly_check_status, checkly_time_to_ssl_expiry_seconds and all heartbeat metrics ignore the parameter.

PromQL examples

Currently failing checks

To graph whether checks are passing or failing, use the query:
Passing checks will have the value 1 while failing and degraded checks will have the value 0. This can be used to build a Grafana table of currently failing checks.

Failure percentage

To calculate the share of check runs that failed in the last 24 hours, as a ratio from 0 to 1, use:
With locationLabelEnabled, a location with no runs in the window returns NaN for this query.

Histogram averages

The different histogram metrics can all be used to compute averages. For example, query the average web vitals times for a check using:

Scraping with other tools

The prometheus receiver of the OpenTelemetry Collector, included in the otelcol-contrib distribution, scrapes the endpoint with the same settings. Replace debug with the exporter for your backend.
The token file holds only the bearer token. Exporters for every common backend are listed in the Collector configuration docs. If you scrape with the Datadog Agent OpenMetrics check or Grafana Alloy prometheus.scrape, use the same URL, bearer token and interval.

Legacy v1 endpoint

The v1 endpoint is deprecated. Use the v2 endpoint described above for new setups. The v1 endpoint keeps working for existing scrape jobs.
The v1 endpoint is /accounts/<your account ID>/prometheus/metrics. It takes the same bearer tokens, allows 300 requests per minute, caches responses for 30 seconds and accepts only the disableTagParsing parameter.

Legacy v1 metrics and labels

Checks with no run in the last hour are absent from the three check metrics. Labels:
  • checkly_check_status and checkly_check_degraded_status: check_name, check_type, muted, activated, degraded, group for checks in a check group, tags and tag labels.
  • checkly_check_result: check_name, check_type, region, group for checks in a check group, http_status_code on API checks only, tags and tag labels.
  • check_type is any check type in lowercase. muted and activated are the check’s own settings, without its check group.
  • tags holds only the tags that are not key:value. Each key:value tag becomes a label: the key is the text before the first colon, with any characters, so team-x:y produces the invalid label name team-x. The value stops at the second colon. Label values are not escaped.
  • The private location metrics carry private_location_name, private_location_slug_name and private_location_id.

Legacy v1 example output

Notice that:
  • checkly_check_status and checkly_check_degraded_status carry muted, activated and degraded, reflecting whether a check sends alerts, is running and is degraded. “Customer API” is in a check group, so it also carries group.
  • checkly_check_result has one row per region that ran the check in the last hour, with a region label. A run from a private location carries the private location’s slug as its region. Rows for API checks also carry http_status_code.
  • The private location “ACME data center” has one Checkly Agent connected. checkly_private_location_queue_size and checkly_private_location_oldest_scheduled_check_run show no backlog of check run jobs.

Legacy v1 scrape config

The v1 endpoint answers with the content type text/html. Prometheus 3 and later reject that unless the job sets fallback_scrape_protocol, as above. Prometheus 2 does not know that setting, so remove the line there. For a worked example with Grafana, see the blog post on monitoring website performance with Checkly, Prometheus and Grafana.