rollfuse
OBSERVABILITY

Automatic Rollback From a Metric Your Grafana Already Reads

How it works

A connector reads a PromQL query through Grafana's data source proxy, and a guardrail aborts the rollout when the window's average crosses the threshold.

The number that tells you a rollout is going wrong is probably already on a Grafana dashboard: an error ratio, a p95 latency, a queue depth. Grafana shows it to a person. A rollfuse guardrail can act on it: a connector reads the same query through Grafana's HTTP API, and when the value crosses a threshold during a rollout, rollfuse aborts the rollout automatically.

This page shows that path for a Prometheus-compatible data source in Grafana, and the alternative of sending the values yourself.

How rollfuse reads a metric through Grafana

A rollfuse connector makes a GET request to an HTTPS URL, sends its pull secret as Authorization: Bearer <secret>, and turns the JSON response into observations. It does this periodically and whenever you trigger a sync.

Grafana's data source proxy forwards a request to one of its data sources, identified by UID, at /api/datasources/proxy/uid/<datasource-uid>/.... Grafana authenticates API requests with a service account token sent as a bearer token. Put the two together and the connector's URL is the Prometheus instant-query endpoint behind your Grafana:

https://grafana.example.com/api/datasources/proxy/uid/<datasource-uid>/api/v1/query?query=<your PromQL, URL-encoded>

and the connector's pull secret is the Grafana service account token.

We tested this against a Grafana Cloud stack on 11 October 2026. With the service account token, the proxy answered 200 with the standard Prometheus API response ({"status":"success","data":{"resultType":"vector","result":[...]}}). Without the token, it answered 401.

1. Define the metric in rollfuse

A metric definition names the metric and its unit. It holds no query; the PromQL stays in the connector's URL.

POST /v1/projects/{project_id}/metric-definitions
{
  "name": "Checkout 5xx ratio",
  "key": "checkout_5xx_ratio",
  "aggregation_type": "average",
  "unit": "ratio"
}

2. Test the mapping against your Grafana

The Prometheus response keeps each series under data.result, with its point as a [timestamp, "value"] pair. You can try the mapping against the live URL before saving anything; the preview fetches once and stores nothing:

POST /v1/projects/{project_id}/connectors/preview
{
  "source_url": "https://grafana.example.com/api/datasources/proxy/uid/<datasource-uid>/api/v1/query?query=<URL-encoded PromQL>",
  "secret": "<Grafana service account token>",
  "root_path": "data.result",
  "value_field": "value.1",
  "observed_at_field": "value.0",
  "observed_at_format": "unix_seconds"
}

The response shows the raw body Grafana returned and the observations the mapping produced, or which field failed to resolve.

3. Create the connector

The same four mapping fields go into a contract template version in your project, which has to be submitted and approved before a connector can use it. Then create the connector with the Grafana URL, the service account token as secret, the metric definition and the environment the rollout runs in. The connectors tutorial walks through preview, creation and sync.

4. Attach a guardrail to the rollout

A guardrail references the metric definition, a comparison operator, a threshold and an evaluation window. You can create it in the console or through the API:

POST /v1/rollouts/{rollout_id}/guardrails
{
  "metric_definition_id": "<metric_definition_id>",
  "comparison_operator": "gt",
  "threshold": 0.01,
  "evaluation_window_seconds": 900
}

The automatic rollback tutorial covers this call and how to read the guardrail's evaluation history.

Write a query that returns one number

Each series in data.result becomes one observation, and the guardrail compares the average of all observations in its window with the threshold. A query that returns one series per pod or per route would mix them into a single average. Aggregate down to one series instead, for example:

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

If your service labels its metrics by flag variation, the query can also measure only the variation being rolled out, instead of the whole service.

Or send the values yourself

If your Grafana isn't reachable from the internet, or the data source isn't Prometheus-compatible, a job of yours can run the query and send the result to rollfuse with a service credential that has the metrics:write scope:

POST /v1/environments/{environment_id}/metric-definitions/{metric_definition_id}/observations
{
  "observations": [
    { "value": 0.004, "occurred_at": "2026-10-11T14:00:00Z" }
  ]
}

The guardrail treats these observations the same way as the ones a connector records.

What happens on a breach

When the window's average crosses the threshold, rollfuse aborts the rollout: the flag goes back to serving its default variation to 100% of that environment, and the rollout stops advancing. Your SDKs pick up the new configuration on their next update. Nothing is redeployed; the new code path just stops being served while someone investigates.

Automatic rollback is available on every plan within a per-plan guardrail limit (Free: 1, Starter: 5, Growth and Enterprise: unlimited). See pricing for the plans.

Seeing each variation on your dashboards

Grafana can show how each variation behaves if your service exports metrics labeled by variation. The public rollfuse/examples-go-prometheus repository does this with the Go SDK: it labels checkout_requests_total and checkout_request_duration_seconds with variation_key, counts rollfuse_flag_evaluations_total by flag, variation and reason, and ships a Grafana dashboard for them. That instrumentation lives in your service's code; rollfuse doesn't send metrics to Grafana.

Limits worth knowing

  • Grafana must be reachable over public HTTPS. The connector only accepts https URLs and refuses to connect to private, loopback or link-local addresses, so a Grafana that only answers inside your network can't be the source. Send the values yourself in that case.
  • The token is sent on every sync. Use a service account that only needs to query that data source, and rotate the connector's secret when you rotate the token.
  • No data means no rollback. If Grafana or the data source stops answering for longer than the evaluation window, the window ends up with no observations; rollfuse records that as insufficient data, not as a breach, and the rollout is not aborted.
  • Guardrail evaluations stay in rollfuse. A signed-in member with the guardrails:manage permission reads them in the console or through the API with their session; service credentials can't read them.
  • The mapping above is for Prometheus-compatible data sources. Other data source types answer in their own format and need their own mapping, if they fit the connector's model at all.
  • A project can have up to 20 connectors.

Sources