VictoriaMetrics Integration

Overview

VictoriaMetrics is a fast, cost-effective, and scalable time-series database. Alert rules are evaluated by vmalert, which fires notifications to a configured Alertmanager instance. Alertmanager then groups the alerts and delivers a webhook payload to the platform.

This integration supports automatic alert creation on firing events and automatic resolution when Alertmanager sends a resolved notification.

Integration Flow

1. VictoriaMetrics ingests and stores metrics from your scrapers (vmagent, Prometheus, etc.).

2. vmalert continuously evaluates your alert rules against VictoriaMetrics. When a rule's condition is met, it pushes the alert to Alertmanager.

3. Alertmanager groups the alerts and delivers a webhook POST request to the platform endpoint.

4. When the alert condition clears, Alertmanager sends a resolved notification and the platform automatically closes the alert.

Webhook Payload Schema

The payload delivered to the platform follows the standard Alertmanager webhook format (version 4), since vmalert uses the Alertmanager protocol natively.

{
  "receiver": "string",
  "status": "firing | resolved",
  "alerts": [
    {
      "status": "firing | resolved",
      "labels": {
        "alertname": "string",
        "severity": "string",
        "cluster": "string"
      },
      "annotations": {
        "summary": "string",
        "description": "string"
      },
      "startsAt": "ISO8601 timestamp",
      "endsAt": "ISO8601 timestamp",
      "generatorURL": "string",
      "fingerprint": "string"
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "alertname": "string",
    "severity": "string"
  },
  "commonAnnotations": {
    "summary": "string",
    "description": "string"
  },
  "externalURL": "string",
  "version": "4",
  "groupKey": "string",
  "truncatedAlerts": 0
}

Setup

Step 1 — Create an Alert Source on the Platform

1. Navigate to Sources → Add Source.

2. Search for VictoriaMetrics and select it.

3. Give the source a name and click Save.

4. Copy the ITOC360 URL and Token — you will need them in the next step.

Step 2 — Install and Configure Alertmanager

vmalert does not deliver alerts directly to external systems. You need an Alertmanager instance to receive alerts from vmalert and route them to the platform.

Install Alertmanager using your preferred method (binary, Docker, Helm). Then configure it to forward alerts to the platform:

alertmanager.yml

global:
  resolve_timeout: 5m

route:
  receiver: itoc360-webhook
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 4h

receivers:
  - name: itoc360-webhook
    webhook_configs:
      - url: "https://api.itoc360.app/functions/v1/events?token=<x-itoc360-token>"
        send_resolved: true

send_resolved: true is required for automatic alert resolution on the platform.

Step 3 — Run vmalert

vmalert needs three things to operate: a datasource (VictoriaMetrics), a notifier (Alertmanager), and one or more rule files.

Docker example:

docker run -d --name vmalert \
  -p 8880:8880 \
  -v /path/to/rules:/etc/rules \
  victoriametrics/vmalert:latest \
  -rule=/etc/rules/*.yaml \
  -datasource.url=http://<victoriametrics-host>:8428 \
  -remoteWrite.url=http://<victoriametrics-host>:8428 \
  -notifier.url=http://<alertmanager-host>:9093 \
  -evaluationInterval=15s

Flag reference:

• -datasource.url — VictoriaMetrics endpoint used to evaluate rule expressions.

• -remoteWrite.url — where vmalert persists alert state and recording rule results.

• -notifier.url — Alertmanager address that will receive the alerts.

• -rule — path to your rule files (supports wildcards).

Step 4 — Create Alert Rules

vmalert uses the same rule format as Prometheus. Create rule files in the directory you mounted in Step 3.

Example: rules/production.yaml

groups:
  - name: production-critical
    interval: 1m
    rules:
      - alert: HighCPUUsage
        expr: |
          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High CPU usage detected"
          description: "Instance {{ $labels.instance }} CPU usage is above 90% for more than 5 minutes."

The severity label is used by the platform for priority mapping (see table below).

Step 5 — Verify the Integration

After starting vmalert and Alertmanager:

1. Open the vmalert UI at http://<vmalert-host>:8880/vmalert/groups — your rule groups should be listed and active.

2. Open the Alertmanager UI at http://<alertmanager-host>:9093 — confirm that firing alerts are routed to the itoc360-webhookreceiver.

3. Confirm the alert appears on the platform under the source you created in Step 1.

Sample Payload

The following is a real payload captured during integration testing.

ALERT (firing):

{
  "receiver": "itoc360-webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighCPUUsage",
        "instance": "vm-server-01:9100",
        "severity": "critical",
        "cluster": "east-1"
      },
      "annotations": {
        "summary": "VictoriaMetrics: High CPU usage detected",
        "description": "Instance vm-server-01:9100 CPU usage is above 90% for more than 5 minutes."
      },
      "startsAt": "2026-04-17T11:30:00.000Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://vmalert:8880/vmalert/alert?group_id=1234567890&alert_id=9876543210",
      "fingerprint": "vmtest34e164e9af873ac1"
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "alertname": "HighCPUUsage",
    "severity": "critical",
    "cluster": "east-1"
  },
  "commonAnnotations": {
    "summary": "VictoriaMetrics: High CPU usage detected"
  },
  "externalURL": "http://alertmanager:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"HighCPUUsage\"}",
  "truncatedAlerts": 0
}

RESOLVE (resolved):

{
  "receiver": "itoc360-webhook",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "alertname": "HighCPUUsage",
        "instance": "vm-server-01:9100",
        "severity": "critical",
        "cluster": "east-1"
      },
      "annotations": {
        "summary": "VictoriaMetrics: High CPU usage detected",
        "description": "Instance vm-server-01:9100 CPU usage is above 90% for more than 5 minutes."
      },
      "startsAt": "2026-04-17T11:30:00.000Z",
      "endsAt": "2026-04-17T11:45:00.000Z",
      "generatorURL": "http://vmalert:8880/vmalert/alert?group_id=1234567890&alert_id=9876543210",
      "fingerprint": "vmtest34e164e9af873ac1"
    }
  ],
  "version": "4"
}

Field Mapping Reference

Payload Field	Description
status	Top-level event type: firing → ALERT, resolved → RESOLVE
alerts[0].fingerprint	Unique identifier per alert label set — used for fingerprint matching
alerts[0].labels.alertname	Name of the alert rule that fired
alerts[0].labels.severity	Severity label from the rule definition — used for priority mapping
alerts[0].labels.cluster	Optional external label set via vmalert -external.label flag (useful in HA setups)
alerts[0].annotations.summary	Short human-readable alert title
alerts[0].annotations.description	Detailed description of the alert condition
alerts[0].startsAt	ISO 8601 timestamp when the alert started firing
alerts[0].endsAt	ISO 8601 timestamp when resolved (0001-... means still active)
alerts[0].generatorURL	Link back to the originating rule in the vmalert UI
commonLabels	Labels shared across all alerts in this group
commonAnnotations	Annotations shared across all alerts in this group
groupKey	Alertmanager group key used for deduplication

Priority Mapping

The platform maps the severity label from your alert rule to an internal priority level.

VictoriaMetrics severity Label	Platform Priority
critical	CRITICAL
error	HIGH
warning	MEDIUM
info	LOW
(not set)	MEDIUM (default)

You control the severity label in your alert rule definitions. Use consistent values across your rule files for predictable priority routing.

RESOLVE Detection

The platform automatically resolves an alert when Alertmanager sends a payload with "status": "resolved". This requires send_resolved: true in your Alertmanager webhook configuration (set in Step 2).

The resolved event is matched to the original alert using the fingerprint field, which Alertmanager generates deterministically from the alert's label set. As long as the labels do not change between firing and resolution, the fingerprint will match and the alert will be closed.