Escalations

Escalations are the decision engines of ITOC360. While "Sources" bring the data in, Escalations decide what to do with it.

An Escalation Policy allows you to filter incoming alerts based on specific rules (e.g., "Only alerts with severity 'Disaster'") and define a step-by-step notification path (e.g., "Notify the On-Call Engineer first; if they don't answer in 5 minutes, notify the Manager").

Creating an Escalation Policy

To set up your routing logic:

1. Navigate to Management > Escalations.

2. Click the Create Escalation button.

3. Name: Give your policy a clear name (e.g., "Database Critical Incidents").

4. Description: Optional. Add a short summary of what this policy routes and why (e.g., “Escalate Zabbix DB DISASTER alerts to DBA on-call”).

5. Source: Select the Source you created in the previous step (e.g., "Zabbix Prod"). This links the policy to a specific data stream.

Rule Engine

This is where you define which alerts trigger this policy. ITOC360 provides a flexible Rule Builder with three modes:

• Visual: A point-and-click interface to build logic without code. You can group conditions using AND / OR logic.

  • Example: IF $.status EQUALS PROBLEM AND $.severity EQUALS DISASTER.

• JSON: For advanced users who prefer writing raw query logic.

• AI (Coming Soon): An intelligent assistant that helps you generate rules using natural language.

Once your conditions are set, toggle the Active switch to ON and click Create.

Pro Tip: To streamline your setup, you don't have to start from scratch. You can quickly apply a pre-configured Rule Template or Import an existing JSON rule structure.

Defining Escalation Levels

After creating the policy, you must define the notification steps. This is done via Levels.

1. Click on the name of the Escalation Policy you just created.

2. Switch to the Levels tab.

3. Click Create Level.

Level Configuration

A "Level" represents a step in the chain of command.

• Level Number: The order of execution (e.g., Level 1 runs first).

• Timeout: The duration (in seconds) the system waits before escalating to the next level. (e.g., 300 for 5 minutes).

• Stop on Acknowledge: If checked, the escalation process stops immediately if a user acknowledges the alert. (Highly Recommended).

  • For more information -> Acknowledge System

• Stop on Recovery: If checked, the escalation stops if the monitoring tool sends a "Resolved/OK" signal.

Assigning Recipients to Levels

Once a level is created, you define who gets notified at that step. You can assign:

• A Specific User: For direct notification.

• A Team: To notify all members or use the team's internal logic.

• A Schedule: To automatically route the alert to whoever is currently on-call according to the shift calendar.

Example Scenario:

• Level 1: Notify "Database Schedule" (On-Call Engineer). Wait 5 minutes.

• Level 2: Notify "Team Lead". Wait 10 minutes.

• Level 3: Notify "CTO".