{
  "id": "hATJ4H49s5vvlVRQ",
  "name": "AlertAssistant",
  "tags": [],
  "nodes": [
    {
      "id": "dbac538b-b899-4b55-8b99-ce62a87ef11d",
      "name": "AI Agent",
      "type": "@n8n/n8n-nodes-langchain.agent",
      "position": [
        928,
        112
      ],
      "parameters": {
        "options": {
          "systemMessage": "=You are an senior SRE assistant for a game studio infrastructure. Your main goal is to assist in the investigation of the incident indicated by the Alert.\n\n## Kubernetes Tools\nUse these to inspect cluster state: pods, deployments, ingress, services, logs, events, etc\n- Read container logs for error investigation\n- List resources across namespaces\n- Main workload is in namespace: kiss\n\n## Grafana Tools  \nUse these to query metrics, logs, dashboards, and alerts:\n- **Prometheus**: query_prometheus for PromQL queries (CPU, memory, request rates, error rates). Datasource uid: {{ $('SetVars').first().json.prometheus_uid }}\n- **Loki**: query_loki_logs for LogQL log searches. Datasource uid: {{ $('SetVars').first().json.loki_uid }}\n- **Dashboards**: search_dashboards, get_dashboard_summary to find relevant dashboards\n\n## Vector store tools\n- Read alerts rules to understand the conditions for their activation\n\n## Digitalocean tools:\n- Use if you need information about App Platform, Droplets, Kubernetes(DOKS), Networking\n\n## Github tools\n- Use this to find out if releases have been executed.\n- Search only within the organization playneta.\n\n## Critical Rules for Tool Usage\n1. **Never retry a failed tool call more than once.** If a Kubernetes resource (pod, deployment, node) is not found on the first attempt, it likely no longer exists. Do NOT retry the same query.\n2. **If any tool returns an error or empty result**, acknowledge it and move on.\n3.  Don't make any changes to anything through mcp.\n\n\n## Common Important rules\n- Don't ask anything, analyze it yourself\n\n## Investigation flow recommendations\n- Pay attention to the alert labels. They will help you understand what the problem is and where it is.\n- Study the alert description and its labels.\n- Try executing the request that triggered the alert.\n\n## Output format\n### What happened\nDescription of what happened and what happened to what\n\n### Event timeline\nDescription of events preceding the alert for more than 10 minutes\n\n### Root cause\nNo more than two root cause suggestions\n\n### Troubleshooting tips\nStep-by-step description of actions for each root cause\n\nWhen formatting your text, make sure it's easy to read in Slack."
        }
      },
      "typeVersion": 3.1
    },
    {
      "id": "7111ff84-f306-4358-89b2-c40a5f328f7b",
      "name": "OpenAI Chat Model",
      "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
      "position": [
        800,
        240
      ],
      "parameters": {
        "model": {
          "__rl": true,
          "mode": "list",
          "value": "gpt-5.3-codex",
          "cachedResultName": "gpt-5.3-codex"
        },
        "options": {},
        "builtInTools": {}
      },
      "credentials": {
        "openAiApi": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 1.3
    },
    {
      "id": "6ca399a9-7742-4e20-8258-5fa27c3ac305",
      "name": "K8S mcp",
      "type": "@n8n/n8n-nodes-langchain.mcpClientTool",
      "position": [
        1056,
        544
      ],
      "parameters": {
        "include": "selected",
        "options": {},
        "endpointUrl": "http://mcp-kubernetes:8089/mcp",
        "includeTools": [
          "configuration_view",
          "events_list",
          "namespaces_list",
          "nodes_log",
          "nodes_stats_summary",
          "nodes_top",
          "pods_get",
          "pods_list",
          "pods_list_in_namespace",
          "pods_log",
          "pods_top",
          "resources_get",
          "resources_list"
        ]
      },
      "typeVersion": 1.2
    },
    {
      "id": "5e3b7dfa-6f83-4998-918f-04afa764a966",
      "name": "Grafana mcp",
      "type": "@n8n/n8n-nodes-langchain.mcpClientTool",
      "position": [
        1200,
        544
      ],
      "parameters": {
        "include": "selected",
        "options": {
          "timeout": 60000
        },
        "endpointUrl": "http://mcp-grafana:8000/mcp",
        "includeTools": [
          "generate_deeplink",
          "get_alert_rule_by_uid",
          "get_annotations",
          "get_dashboard_by_uid",
          "get_dashboard_panel_queries",
          "get_dashboard_property",
          "get_dashboard_summary",
          "get_datasource_by_name",
          "get_datasource_by_uid",
          "get_panel_image",
          "get_sift_analysis",
          "get_sift_investigation",
          "list_alert_rules",
          "list_loki_label_names",
          "list_loki_label_values",
          "list_prometheus_label_names",
          "list_prometheus_label_values",
          "list_prometheus_metric_metadata",
          "list_prometheus_metric_names",
          "query_loki_logs",
          "query_loki_patterns",
          "query_loki_stats",
          "query_prometheus",
          "query_prometheus_histogram",
          "search_dashboards"
        ]
      },
      "typeVersion": 1.2
    },
    {
      "id": "9ca07b6a-6fb0-4bc8-befa-cf2712ecdf6e",
      "name": "Digitalocean MCP",
      "type": "@n8n/n8n-nodes-langchain.mcpClientTool",
      "position": [
        912,
        544
      ],
      "parameters": {
        "include": "selected",
        "options": {},
        "endpointUrl": "http://mcp-digitalocean:8090/mcp",
        "includeTools": [
          "apps-get-deployment-status",
          "apps-get-info",
          "apps-get-logs",
          "apps-list",
          "byoip-prefix-get",
          "byoip-prefix-list",
          "byoip-prefix-resources-get",
          "certificate-get",
          "certificate-list",
          "doks-get-cluster",
          "doks-get-cluster-upgrades",
          "doks-get-nodepool",
          "doks-list-options",
          "doks-list-nodepools",
          "doks-list-clusters",
          "domain-get",
          "domain-list",
          "domain-record-list",
          "domain-record-get",
          "droplet-get",
          "droplet-list",
          "firewall-get",
          "firewall-list",
          "image-action-get",
          "image-get",
          "image-list",
          "lb-get",
          "lb-list",
          "region-list",
          "reserved-ip-list",
          "reserved-ip-get",
          "size-list",
          "vpc-list",
          "vpc-get",
          "vpc-list-members",
          "vpc-peering-get",
          "vpc-peering-list"
        ]
      },
      "typeVersion": 1.2
    },
    {
      "id": "21f3a33e-693f-451c-a606-35779bc69b69",
      "name": "Receive alerts",
      "type": "n8n-nodes-base.webhook",
      "position": [
        80,
        112
      ],
      "parameters": {
        "path": "alertmanager",
        "options": {},
        "httpMethod": "POST",
        "authentication": "basicAuth"
      },
      "credentials": {
        "httpBasicAuth": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 2.1
    },
    {
      "id": "7809686c-7023-42b7-8e36-c44dbe01f858",
      "name": "PreProcessAlerts",
      "type": "n8n-nodes-base.code",
      "position": [
        464,
        112
      ],
      "parameters": {
        "jsCode": "const webhookData = $input.first().json.body || $input.first().json;\n\nconst SKIP_LABELS = new Set([\n  \"alertname\", \"endpoint\", \"instance\", \"job\",\n  \"metrics_path\", \"prometheus\", \"project\", \"env\",\n]);\n\nconst LABEL_RENAME = {\n  \"grafana_board\": \"Grafana Dashboard\",\n  \"persistentvolumeclaim\": \"PVC\",\n};\n\nconst alerts = webhookData.alerts || [];\nconst results = [];\n\nfor (const alert of alerts) {\n  const status = alert.status || \"unknown\";\n  const labels = alert.labels || {};\n  const annotations = alert.annotations || {};\n  const alertname = labels.alertname || \"UnknownAlert\";\n  const fingerprint = alert.fingerprint || \"\";\n  const startsAt = alert.startsAt || \"\";\n  const generatorURL = alert.generatorURL || \"\";\n\n  const lines = [\n    `Analyze this ${status.toUpperCase()} alert.`,\n    `Alertname: ${alertname}`,\n    `Severity: ${labels.severity || \"unknown\"}`,\n  ];\n\n  for (const key of Object.keys(labels).sort()) {\n    if (SKIP_LABELS.has(key) || key === \"severity\") continue;\n    const displayKey = LABEL_RENAME[key] || key;\n    lines.push(`${displayKey}: ${labels[key]}`);\n  }\n\n  if (startsAt) lines.push(`Started At: ${startsAt}`);\n  if (generatorURL) lines.push(`Alert Source: ${generatorURL}`);\n\n  for (const key of Object.keys(annotations).sort()) {\n    const displayKey = key.replace(/_/g, \" \").replace(/\\b\\w/g, c => c.toUpperCase());\n    lines.push(`${displayKey}: ${annotations[key]}`);\n  }\n\n  const chatInput = lines.join(\"\\n\");\n  const sessionId = fingerprint\n    ? `alert-${fingerprint}`\n    : `alert-${alertname}-${labels.namespace || \"\"}-${labels.service || \"\"}`;\n\n  results.push({\n    json: {\n      action: \"sendMessage\",\n      chatInput,\n      sessionId,\n      alert_id: fingerprint,\n    },\n  });\n}\n\nreturn results;"
      },
      "typeVersion": 2
    },
    {
      "id": "00f1a936-5093-4095-b433-0b213ddf6bac",
      "name": "Qdrant Vector Store",
      "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
      "position": [
        1360,
        544
      ],
      "parameters": {
        "mode": "retrieve-as-tool",
        "options": {},
        "toolDescription": "Use this tool to fetch knowledge from the database. ",
        "qdrantCollection": {
          "__rl": true,
          "mode": "list",
          "value": "observability",
          "cachedResultName": "observability"
        }
      },
      "credentials": {
        "qdrantApi": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 1.3
    },
    {
      "id": "6a32e809-cf33-4825-8540-de91b2281fd6",
      "name": "Embeddings Google Gemini",
      "type": "@n8n/n8n-nodes-langchain.embeddingsGoogleGemini",
      "position": [
        1360,
        736
      ],
      "parameters": {
        "modelName": "models/gemini-embedding-001"
      },
      "credentials": {
        "googlePalmApi": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 1
    },
    {
      "id": "5a710de1-2e38-4618-9d27-ac0795489e6f",
      "name": "Gitlab MCP",
      "type": "@n8n/n8n-nodes-langchain.mcpClientTool",
      "position": [
        768,
        544
      ],
      "parameters": {
        "include": "selected",
        "options": {},
        "endpointUrl": "=http://mcp-gitlab:8091/mcp",
        "includeTools": [
          "search_repositories",
          "get_file_contents",
          "get_merge_request",
          "get_merge_request_diffs",
          "get_branch_diffs",
          "get_merge_request_version",
          "list_merge_request_versions",
          "list_namespaces",
          "get_namespace",
          "get_project",
          "list_projects",
          "list_group_projects",
          "get_repository_tree",
          "list_merge_requests",
          "list_commits",
          "get_commit",
          "get_commit_diff",
          "list_pipelines",
          "get_pipeline",
          "list_pipeline_jobs",
          "list_pipeline_trigger_jobs",
          "get_pipeline_job",
          "get_pipeline_job_output"
        ]
      },
      "typeVersion": 1.2
    },
    {
      "id": "50222e20-ae04-4588-b1ed-ad98de338ddb",
      "name": "GetAlertMessages",
      "type": "n8n-nodes-base.slack",
      "position": [
        1408,
        112
      ],
      "parameters": {
        "limit": 10,
        "filters": {
          "latest": ""
        },
        "resource": "channel",
        "channelId": {
          "__rl": true,
          "mode": "id",
          "value": "C053DG9S2EL"
        },
        "operation": "history",
        "authentication": "oAuth2"
      },
      "credentials": {
        "slackOAuth2Api": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 2.4
    },
    {
      "id": "11347d4b-0fb5-4bfc-974c-9a9bd714043b",
      "name": "SendReport",
      "type": "n8n-nodes-base.slack",
      "position": [
        1792,
        112
      ],
      "parameters": {
        "text": "={{ $json.agentOutput }}",
        "select": "channel",
        "channelId": {
          "__rl": true,
          "mode": "list",
          "value": "C053DG9S2EL",
          "cachedResultName": "alerts"
        },
        "otherOptions": {
          "thread_ts": {
            "replyValues": {
              "thread_ts": "={{ $json.thread_ts }}"
            }
          }
        },
        "authentication": "oAuth2"
      },
      "credentials": {
        "slackOAuth2Api": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 2.4
    },
    {
      "id": "8c081f53-f49c-4f32-ae6d-bd1f7b2520f5",
      "name": "FindAlert",
      "type": "n8n-nodes-base.code",
      "position": [
        1600,
        112
      ],
      "parameters": {
        "jsCode": "const alertId = $('PreProcessAlerts').first().json.alert_id || '';\nlet agentOutput = \"\";\ntry {\n  const agentData = $('AI Agent').first().json;\n  agentOutput = agentData.output || \"No data\";\n} catch (e) {}\n\nconst items = $input.all();\n\nlet threadTs = '';\nlet matchedTs = '';\nlet matchedTitle = '';\n\nfor (const item of items) {\n  const msg = item.json;\n  const msgText = msg.text || '';\n  const attachments = msg.attachments || [];\n\n  if (alertId && msgText.includes(alertId)) {\n    threadTs = msg.thread_ts || msg.ts;\n    matchedTs = msg.ts;\n    matchedTitle = msgText.substring(0, 100);\n    break;\n  }\n\n  for (const att of attachments) {\n    const title = att.title || '';\n    const attText = att.text || '';\n\n    if (alertId && (title.includes(alertId) || attText.includes(alertId))) {\n      threadTs = msg.thread_ts || msg.ts;\n      matchedTs = msg.ts;\n      matchedTitle = title;\n      break;\n    }\n  }\n\n  if (threadTs) break;\n}\n\nreturn [{\n  json: {\n    thread_ts: threadTs,\n    matched_ts: matchedTs,\n    matched_title: matchedTitle,\n    alert_id: alertId,\n    agentOutput: agentOutput,\n  }\n}];"
      },
      "typeVersion": 2
    },
    {
      "id": "2afd4eb8-9fa9-46a9-8f31-036ff8abefde",
      "name": "SetVars",
      "type": "n8n-nodes-base.set",
      "position": [
        272,
        112
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "72ef64c3-edd7-4ab4-b859-38f28fab3505",
              "name": "prometheus_uid",
              "type": "string",
              "value": "cb0069a2-5028-423c-b8a6-95bf3c9d39e7"
            },
            {
              "id": "994e2e14-d614-4c91-8897-aa234c9df98b",
              "name": "loki_uid",
              "type": "string",
              "value": "grafanacloud-logs"
            }
          ]
        },
        "includeOtherFields": true
      },
      "typeVersion": 3.4
    },
    {
      "id": "51180c57-ba7d-495a-8381-21aebe08ae22",
      "name": "Sticky Note",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        0,
        0
      ],
      "parameters": {
        "width": 640,
        "height": 352,
        "content": "# Input chain\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nReceive and format alerts from Alertmanager"
      },
      "typeVersion": 1
    },
    {
      "id": "e08cf46f-f9b8-4d26-8a14-1d05d3e644a3",
      "name": "Sticky Note1",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        752,
        0
      ],
      "parameters": {
        "color": 6,
        "width": 528,
        "height": 352,
        "content": "# Alert analysis"
      },
      "typeVersion": 1
    },
    {
      "id": "458f8d22-49df-4c78-a215-65550b4f4d7d",
      "name": "Sticky Note2",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        736,
        448
      ],
      "parameters": {
        "color": 4,
        "width": 896,
        "height": 464,
        "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# MCP servers\nAdd correct URLs for remote MCPs\nUse following mcp:\n* [grafana-mcp](https://github.com/grafana/mcp-grafana)\n* [kubernetes-mcp-server](https://github.com/containers/kubernetes-mcp-server)\n* [digitalocean-mcp](https://github.com/digitalocean/digitalocean-mcp)\n* [gitlab-mcp](https://github.com/zereight/gitlab-mcp)\nCreate QdrantAPI credentials"
      },
      "typeVersion": 1
    },
    {
      "id": "44309ad0-cfea-4085-87c4-73d423c27674",
      "name": "Sticky Note3",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        1360,
        0
      ],
      "parameters": {
        "width": 640,
        "height": 352,
        "content": "# Output chain\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPost generated report to Slack"
      },
      "typeVersion": 1
    },
    {
      "id": "f8f27172-e960-41f9-b639-d7f02b4b2773",
      "name": "Sticky Note4",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        0,
        416
      ],
      "parameters": {
        "color": 5,
        "width": 656,
        "height": 704,
        "content": "# Overview\nThis workflow helps automatically analyze alerts occurring in the infrastructure and suggest solutions even before the on-duty engineer sees the alert.\n## How It Works\nThe workflow receives an alert from Alertmanager via Webhook.\nThe variables required for operation are set.\nA prompt is prepared for the agent containing only the data necessary for analysis.\nThe agent performs diagnostics as described in the system prompt. During operation, it can access various systems via MCP to obtain additional information.\nA message in a Slack channel corresponding to the processed alert is found.\nA report is sent to the Slack thread.\n## How to Use\nGenerate webhook credentials and use them in Alertmanager.\nAdd the alert fingerprint to the Slack message template.\nSet variables in the SetVars node.\nAdd your own rules and recommendations to the system prompt.\nRun MCP servers.\nChoose the Slack channel with alerts.\n## Requirements:\n- OpenAI  or Anthropic API key\n- Slack API key\n- Google Gemini API key\n- Webhook receiver in Alertmanager\n- Webhook token\n- Qdrant with token\n- Gitlab Access key\n"
      },
      "typeVersion": 1
    }
  ],
  "active": false,
  "settings": {
    "binaryMode": "separate",
    "availableInMCP": false,
    "executionOrder": "v1"
  },
  "versionId": "81480c98-aaa6-4f7a-a447-7f3473461613",
  "connections": {
    "K8S mcp": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "SetVars": {
      "main": [
        [
          {
            "node": "PreProcessAlerts",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "AI Agent": {
      "main": [
        [
          {
            "node": "GetAlertMessages",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "FindAlert": {
      "main": [
        [
          {
            "node": "SendReport",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Gitlab MCP": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "Grafana mcp": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "Receive alerts": {
      "main": [
        [
          {
            "node": "SetVars",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Digitalocean MCP": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "GetAlertMessages": {
      "main": [
        [
          {
            "node": "FindAlert",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "PreProcessAlerts": {
      "main": [
        [
          {
            "node": "AI Agent",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "OpenAI Chat Model": {
      "ai_languageModel": [
        [
          {
            "node": "AI Agent",
            "type": "ai_languageModel",
            "index": 0
          }
        ]
      ]
    },
    "Qdrant Vector Store": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "Embeddings Google Gemini": {
      "ai_embedding": [
        [
          {
            "node": "Qdrant Vector Store",
            "type": "ai_embedding",
            "index": 0
          }
        ]
      ]
    }
  }
}