# VictoriaMetrics Integration ### Overview [VictoriaMetrics](https://victoriametrics.com/) is a fast, cost-effective, and scalable time-series database. Alert rules are evaluated by **vmalert**, which fires notifications to a configured **Alertmanager** instance. Alertmanager then groups the alerts and delivers a webhook payload to the platform. This integration supports automatic alert creation on firing events and automatic resolution when Alertmanager sends a `resolved` notification. ### Integration Flow 1. VictoriaMetrics ingests and stores metrics from your scrapers (vmagent, Prometheus, etc.). 2. **vmalert** continuously evaluates your alert rules against VictoriaMetrics. When a rule's condition is met, it pushes the alert to Alertmanager. 3. Alertmanager groups the alerts and delivers a webhook POST request to the platform endpoint. 4. When the alert condition clears, Alertmanager sends a `resolved` notification and the platform automatically closes the alert. ### Webhook Payload Schema The payload delivered to the platform follows the standard Alertmanager webhook format (version 4), since vmalert uses the Alertmanager protocol natively. ```json { "receiver": "string", "status": "firing | resolved", "alerts": [ { "status": "firing | resolved", "labels": { "alertname": "string", "severity": "string", "cluster": "string" }, "annotations": { "summary": "string", "description": "string" }, "startsAt": "ISO8601 timestamp", "endsAt": "ISO8601 timestamp", "generatorURL": "string", "fingerprint": "string" } ], "groupLabels": {}, "commonLabels": { "alertname": "string", "severity": "string" }, "commonAnnotations": { "summary": "string", "description": "string" }, "externalURL": "string", "version": "4", "groupKey": "string", "truncatedAlerts": 0 } ``` ### Setup #### Step 1 — Create an Alert Source on the Platform 1. Navigate to **Sources → Add Source**. 2. Search for **VictoriaMetrics** and select it. 3. Give the source a name and click **Save**. 4. Copy the **ITOC360 URL** and **Token** — you will need them in the next step. #### Step 2 — Install and Configure Alertmanager vmalert does not deliver alerts directly to external systems. You need an Alertmanager instance to receive alerts from vmalert and route them to the platform. Install Alertmanager using your preferred method (binary, Docker, Helm). Then configure it to forward alerts to the platform: **alertmanager.yml** ```yaml global: resolve_timeout: 5m route: receiver: itoc360-webhook group_wait: 10s group_interval: 1m repeat_interval: 4h receivers: - name: itoc360-webhook webhook_configs: - url: "https://api.itoc360.app/functions/v1/events?token=" send_resolved: true ``` > `send_resolved: true` is required for automatic alert resolution on the platform. #### Step 3 — Run vmalert vmalert needs three things to operate: a datasource (VictoriaMetrics), a notifier (Alertmanager), and one or more rule files. **Docker example:** ```bash docker run -d --name vmalert \ -p 8880:8880 \ -v /path/to/rules:/etc/rules \ victoriametrics/vmalert:latest \ -rule=/etc/rules/*.yaml \ -datasource.url=http://:8428 \ -remoteWrite.url=http://:8428 \ -notifier.url=http://:9093 \ -evaluationInterval=15s ``` Flag reference: * `-datasource.url` — VictoriaMetrics endpoint used to evaluate rule expressions. * `-remoteWrite.url` — where vmalert persists alert state and recording rule results. * `-notifier.url` — Alertmanager address that will receive the alerts. * `-rule` — path to your rule files (supports wildcards). #### Step 4 — Create Alert Rules vmalert uses the same rule format as Prometheus. Create rule files in the directory you mounted in Step 3. **Example: rules/production.yaml** ```yaml groups: - name: production-critical interval: 1m rules: - alert: HighCPUUsage expr: | 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90 for: 5m labels: severity: critical annotations: summary: "High CPU usage detected" description: "Instance {{ $labels.instance }} CPU usage is above 90% for more than 5 minutes." ``` The `severity` label is used by the platform for priority mapping (see table below). #### Step 5 — Verify the Integration After starting vmalert and Alertmanager: 1. Open the vmalert UI at `http://:8880/vmalert/groups` — your rule groups should be listed and active. 2. Open the Alertmanager UI at `http://:9093` — confirm that firing alerts are routed to the `itoc360-webhook`receiver. 3. Confirm the alert appears on the platform under the source you created in Step 1. ### Sample Payload The following is a real payload captured during integration testing. **ALERT (firing):** ```json { "receiver": "itoc360-webhook", "status": "firing", "alerts": [ { "status": "firing", "labels": { "alertname": "HighCPUUsage", "instance": "vm-server-01:9100", "severity": "critical", "cluster": "east-1" }, "annotations": { "summary": "VictoriaMetrics: High CPU usage detected", "description": "Instance vm-server-01:9100 CPU usage is above 90% for more than 5 minutes." }, "startsAt": "2026-04-17T11:30:00.000Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://vmalert:8880/vmalert/alert?group_id=1234567890&alert_id=9876543210", "fingerprint": "vmtest34e164e9af873ac1" } ], "groupLabels": {}, "commonLabels": { "alertname": "HighCPUUsage", "severity": "critical", "cluster": "east-1" }, "commonAnnotations": { "summary": "VictoriaMetrics: High CPU usage detected" }, "externalURL": "http://alertmanager:9093", "version": "4", "groupKey": "{}:{alertname=\"HighCPUUsage\"}", "truncatedAlerts": 0 } ``` **RESOLVE (resolved):** ```json { "receiver": "itoc360-webhook", "status": "resolved", "alerts": [ { "status": "resolved", "labels": { "alertname": "HighCPUUsage", "instance": "vm-server-01:9100", "severity": "critical", "cluster": "east-1" }, "annotations": { "summary": "VictoriaMetrics: High CPU usage detected", "description": "Instance vm-server-01:9100 CPU usage is above 90% for more than 5 minutes." }, "startsAt": "2026-04-17T11:30:00.000Z", "endsAt": "2026-04-17T11:45:00.000Z", "generatorURL": "http://vmalert:8880/vmalert/alert?group_id=1234567890&alert_id=9876543210", "fingerprint": "vmtest34e164e9af873ac1" } ], "version": "4" } ``` ### Field Mapping Reference | Payload Field | Description | | ----------------------------------- | ------------------------------------------------------------------------------------ | | `status` | Top-level event type: `firing` → ALERT, `resolved` → RESOLVE | | `alerts[0].fingerprint` | Unique identifier per alert label set — used for fingerprint matching | | `alerts[0].labels.alertname` | Name of the alert rule that fired | | `alerts[0].labels.severity` | Severity label from the rule definition — used for priority mapping | | `alerts[0].labels.cluster` | Optional external label set via vmalert `-external.label` flag (useful in HA setups) | | `alerts[0].annotations.summary` | Short human-readable alert title | | `alerts[0].annotations.description` | Detailed description of the alert condition | | `alerts[0].startsAt` | ISO 8601 timestamp when the alert started firing | | `alerts[0].endsAt` | ISO 8601 timestamp when resolved (`0001-...` means still active) | | `alerts[0].generatorURL` | Link back to the originating rule in the vmalert UI | | `commonLabels` | Labels shared across all alerts in this group | | `commonAnnotations` | Annotations shared across all alerts in this group | | `groupKey` | Alertmanager group key used for deduplication | ### Priority Mapping The platform maps the `severity` label from your alert rule to an internal priority level. | VictoriaMetrics `severity` Label | Platform Priority | | -------------------------------- | ----------------- | | `critical` | CRITICAL | | `error` | HIGH | | `warning` | MEDIUM | | `info` | LOW | | (not set) | MEDIUM (default) | You control the `severity` label in your alert rule definitions. Use consistent values across your rule files for predictable priority routing. ### RESOLVE Detection The platform automatically resolves an alert when Alertmanager sends a payload with `"status": "resolved"`. This requires `send_resolved: true` in your Alertmanager webhook configuration (set in Step 2). The resolved event is matched to the original alert using the `fingerprint` field, which Alertmanager generates deterministically from the alert's label set. As long as the labels do not change between firing and resolution, the fingerprint will match and the alert will be closed. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.itoc360.com/integrations/inbound-integrations/observability-and-apm/victoriametrics-integration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.