Netdata Integration

Overview

Netdata is an open-source, real-time infrastructure monitoring tool that tracks thousands of metrics per second with per-second resolution across servers, containers, databases, and applications.

This integration listens for alerts dispatched by the Netdata Agent via its custom webhook notification method and turns them into actionable alerts on the platform. Alerts are automatically resolved when Netdata sends a CLEAR notification for the same alarm.

Integration Flow

1. The Netdata Agent continuously evaluates health rules against the metrics it collects.

2. When an alarm's state transitions (for example CLEAR → WARNING, WARNING → CRITICAL, or CRITICAL → CLEAR), the Agent invokes alarm-notify.sh.

3. alarm-notify.sh calls the custom_sender() function that you configure in health_alarm_notify.conf.

4. That function builds a JSON payload from the alarm context and posts it to the platform endpoint.

5. The platform classifies the event by status — any value other than CLEAR opens or updates an alert, and CLEARresolves it.

Webhook Payload Schema

{
  "name": "string",
  "chart": "string",
  "family": "string",
  "status": "WARNING | CRITICAL | CLEAR | UNDEFINED | UNINITIALIZED",
  "old_status": "string",
  "value": "string",
  "old_value": "string",
  "value_string": "string",
  "old_value_string": "string",
  "units": "string",
  "info": "string",
  "when": "unix timestamp",
  "host": "string",
  "unique_id": "string",
  "alarm_id": "string",
  "event_id": "string",
  "duration": "string",
  "non_clear_duration": "string",
  "classification": "string",
  "component": "string",
  "type": "string",
  "severity": "string",
  "calc_expression": "string",
  "total_warnings": "string",
  "total_critical": "string",
  "src": "string",
  "goto_url": "string",
  "image_url": "string"
}

Setup

Step 1 — Create an Alert Source on the Platform

1. Navigate to Sources → Add Source.

2. Search for Netdata and select it.

3. Give the source a name and click Save.

4. Copy the ITOC360 URL and Token — you will need them in Step 3. The full webhook URL has the form:

https://api.itoc360.app/functions/v1/events?token=<x-itoc360-token>

Step 2 — Access the Netdata Configuration

Netdata's notification settings live in health_alarm_notify.conf. How you reach this file depends on how you run Netdata.

If Netdata runs in Docker:

Copy the default configuration out of the container so you can edit it on your host:

docker exec netdata cat /usr/lib/netdata/conf.d/health_alarm_notify.conf > ./health_alarm_notify.conf

Open the file in your preferred editor. You will push the edited file back into the container in Step 4.

If Netdata runs on a host (apt/yum install):

Use the edit-config helper so the file ends up in the right location:

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf

Step 3 — Add the Custom Sender Block

Append the following block to the end of health_alarm_notify.conf. Replace the webhook URL with the one you copied in Step 1.

#------------------------------------------------------------------------------
# itoc360 custom webhook notifications
#------------------------------------------------------------------------------

SEND_CUSTOM="YES"
DEFAULT_RECIPIENT_CUSTOM="sysadmin"

custom_sender() {
    local webhook_url="https://api.itoc360.app/functions/v1/events?token=<x-itoc360-token>"

    local payload=$(cat <<EOF
{
  "name": "${name}",
  "chart": "${chart}",
  "family": "${family}",
  "status": "${status}",
  "old_status": "${old_status}",
  "value": "${value}",
  "old_value": "${old_value}",
  "src": "${src}",
  "duration": "${duration}",
  "non_clear_duration": "${non_clear_duration}",
  "units": "${units}",
  "info": "${info}",
  "when": "${when}",
  "host": "${host}",
  "unique_id": "${unique_id}",
  "alarm_id": "${alarm_id}",
  "event_id": "${event_id}",
  "calc_expression": "${calc_expression}",
  "total_warnings": "${total_warnings}",
  "total_critical": "${total_critical}",
  "classification": "${classification}",
  "component": "${component}",
  "type": "${type}",
  "severity": "${severity}",
  "value_string": "${value_string}",
  "old_value_string": "${old_value_string}",
  "image_url": "${image_url}",
  "goto_url": "${goto_url}"
}
EOF
)

    httpcode=$(docurl \
        --request POST \
        --header "Content-Type: application/json" \
        --data "${payload}" \
        "${webhook_url}")

    if [ "${httpcode}" = "200" ] || [ "${httpcode}" = "201" ] || [ "${httpcode}" = "204" ]; then
        info "sent custom webhook notification for: ${host} ${chart}.${name} is ${status}"
        sent=$((sent + 1))
    else
        error "failed to send custom webhook notification for: ${host} ${chart}.${name} is ${status}, http code ${httpcode}"
    fi

    return 0
}

SEND_CUSTOM="YES" turns on the custom notification method, and DEFAULT_RECIPIENT_CUSTOM="sysadmin" ensures alerts routed to the sysadmin role reach this integration. Adjust the recipient role if your setup routes alerts elsewhere.

Save the file.

Step 4 — Apply the Configuration

If Netdata runs in Docker:

docker cp ./health_alarm_notify.conf netdata:/etc/netdata/health_alarm_notify.conf
docker exec netdata chown netdata:netdata /etc/netdata/health_alarm_notify.conf
docker restart netdata

If Netdata runs on a host:

Changes made via edit-config take effect on the next alarm evaluation, but a restart forces a clean reload:

sudo systemctl restart netdata

Step 5 — Verify the Integration

Netdata ships with a built-in test command that dispatches three sample alarms in sequence (WARNING, CRITICAL, and CLEAR):

# Docker
docker exec -u netdata netdata bash -c \
  'export NETDATA_ALARM_NOTIFY_DEBUG=1 && /usr/libexec/netdata/plugins.d/alarm-notify.sh test'

# Host install
sudo su -s /bin/bash netdata
export NETDATA_ALARM_NOTIFY_DEBUG=1
/usr/libexec/netdata/plugins.d/alarm-notify.sh test

You should see three # OK lines in the output, one per alarm. On the platform, under the source you created in Step 1, a single alert should open on the WARNING event, stay open through the CRITICAL event, and automatically resolve when the CLEAR event arrives.

Sample Payloads

The following payloads were captured during integration testing.

ALERT — WARNING state:

{
  "name": "test_alarm",
  "chart": "test.chart",
  "family": "",
  "status": "WARNING",
  "old_status": "CLEAR",
  "value": "100",
  "old_value": "90",
  "src": "/usr/libexec/netdata/plugins.d/alarm-notify.sh",
  "duration": "1",
  "non_clear_duration": "1",
  "units": "units",
  "info": "this is a test alarm to verify notifications work",
  "when": "1776694259",
  "host": "3b60a2cac47d",
  "unique_id": "1",
  "alarm_id": "1",
  "event_id": "1",
  "severity": "WARNING",
  "classification": "Test",
  "value_string": "new value",
  "old_value_string": "old value",
  "goto_url": "https://registry.my-netdata.io/registry-alert-redirect.html?host=3b60a2cac47d&chart=test.chart&alarm=test_alarm&alarm_status=WARNING"
}

RESOLVE — CLEAR state:

{
  "name": "test_alarm",
  "chart": "test.chart",
  "family": "",
  "status": "CLEAR",
  "old_status": "CRITICAL",
  "value": "100",
  "old_value": "90",
  "src": "/usr/libexec/netdata/plugins.d/alarm-notify.sh",
  "duration": "1",
  "non_clear_duration": "3",
  "units": "units",
  "info": "this is a test alarm to verify notifications work",
  "when": "1776694259",
  "host": "3b60a2cac47d",
  "unique_id": "1",
  "alarm_id": "1",
  "event_id": "3",
  "severity": "Recovered from CRITICAL",
  "classification": "Test",
  "value_string": "new value",
  "old_value_string": "old value",
  "goto_url": "https://registry.my-netdata.io/registry-alert-redirect.html?host=3b60a2cac47d&chart=test.chart&alarm=test_alarm&alarm_status=CLEAR"
}

Field Mapping Reference

Payload Field	Description
status	Alarm state — any value other than CLEAR opens or updates an alert, CLEAR resolves it
host	Hostname of the Netdata Agent — combined with alarm_id and name to fingerprint the alert
alarm_id	Stable identifier for the alarm across state transitions
name	Name of the alarm rule — shown in the alert title
chart	Chart the alarm is attached to — shown in the alert title
info	Human-readable description of why the alarm fired
value / units	Current metric value and its unit of measurement
old_value / old_status	Previous metric value and alarm state
severity	Descriptive severity string, which may include escalation context (e.g. Escalated to CRITICAL)
classification / component / type	Alarm taxonomy fields set in the alarm definition
when	Unix timestamp of the state change
duration / non_clear_duration	How long the alarm has been in its current state, and since it was last clear
calc_expression	Expression that was evaluated to produce the current value
total_warnings / total_critical	Total warning and critical alarms currently active on the host
goto_url	Direct link to the alarm in the Netdata dashboard
unique_id / event_id	Identifiers used internally by Netdata for this specific transition

Priority Mapping

The platform maps Netdata's status field to an internal priority level.

Netdata status	Platform Priority
CRITICAL	CRITICAL
WARNING	MEDIUM
UNDEFINED	LOW
UNINITIALIZED	LOW
CLEAR	LOW (this event resolves the alert rather than opening one)

The severity field in the payload (for example Escalated to CRITICAL or Recovered from CRITICAL) is a human-readable description that Netdata composes at notification time. The platform does not use it for priority routing — status is the authoritative field.

RESOLVE Detection

The platform resolves an alert when the Netdata Agent sends a payload with "status": "CLEAR". This is the standard notification that alarm-notify.sh emits when an alarm's condition no longer holds.

The resolve event is matched to the original alert using a fingerprint derived from host, alarm_id, and name. As long as those three fields stay the same across the firing and resolving notifications — which is the default Netdata behavior — the fingerprint will match and the alert will be closed.

Troubleshooting

No payload arrives on the platform after a test:

Check the Netdata error log for delivery failures:

# Docker
docker exec netdata tail -50 /var/log/netdata/error.log

# Host
sudo tail -50 /var/log/netdata/error.log

Look for failed to send custom webhook notification lines — they include the HTTP status code returned by the platform endpoint.

Test runs but no # OK for the custom sender:

Confirm the configuration was reloaded. For Docker, make sure you ran docker restart netdata after copying the file in. For host installs, confirm SEND_CUSTOM="YES" appears in the active file and that the file belongs to the netdata user:

sudo ls -la /etc/netdata/health_alarm_notify.conf

Alerts are created but never resolve:

The most common cause is a custom alarm that does not transition back to CLEAR cleanly — for example, if the alarm rule is deleted before the condition recovers. Check the Netdata dashboard's alerts view to confirm the alarm itself is actually clearing.