Detect LLM Prompt Regressions with Data Tables, Openai Apis, and Telegram

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

{
  "name": "Detect LLM prompt regressions on a schedule with Data Tables and Telegram",
  "nodes": [
    {
      "id": "32b434ec-8e8c-48fd-b47a-f3040d9bf6d0",
      "name": "Overview",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        0,
        64
      ],
      "parameters": {
        "width": 480,
        "height": 948,
        "content": "## Detect prompt regressions on a schedule\n\nCI for your LLM prompts. Keep a golden test set in n8n Data Tables, run it on a schedule, and get a Telegram alert only when quality drops below a blessed baseline - not on every provider hiccup.\n\n**Who it's for:** teams running production prompts in n8n (support classifiers, extractors, reply generators) who want a regression signal without an external eval platform.\n\n### How it works\n* A schedule starts each run; Execute workflow runs it on demand.\n* Profiles (prompt + model + temperature) and cases (input + expected) load from Data Tables; on an empty instance the workflow seeds demo rows so the first run works out of the box.\n* Each case calls its own model through a generic OpenAI-compatible HTTP request.\n* Deterministic checks (exact, contains, regex, numeric) grade answers for free; an LLM judge scores only cases marked `llm-judge`.\n* Each run is snapshotted and compared to the blessed baseline; a flipped case must fail two runs in a row before it counts, while a broad pass-rate drop alerts at once.\n* A Telegram digest goes out on a baseline capture, regression, recovery or ops failure, and stays silent when quality holds.\n\n### Setup\n1. Pick the cadence in the Schedule Trigger.\n2. Work through the three grey setup notes: credential, Data Tables, Telegram chat.\n3. Tune thresholds, alert mode and judge model in the Config node.\n4. Run once with Execute workflow to seed demo data and capture the first baseline.\n5. Add your own prompts and cases as table rows; no canvas edits needed.\n\n### Requirements\n* Any OpenAI-compatible chat API (OpenRouter, OpenAI, Groq, Ollama); the demo uses free OpenRouter models\n* [Data Tables](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.datatable/) available on your instance (n8n 1.113+)\n* [Telegram bot credential](https://docs.n8n.io/integrations/builtin/credentials/telegram/)\n\n### Need help?\nAsk in the [n8n Forum](https://community.n8n.io/)!"
      },
      "typeVersion": 1
    },
    {
      "id": "55f26e62-b32a-4ed1-90b7-678e23bc5920",
      "name": "Note 1",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        528,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 416,
        "height": 622,
        "content": "## 1. Trigger and config\n[Read more about the Schedule Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.scheduletrigger/)\n\nRuns on the schedule you set, or on demand with Execute workflow. The Config node holds every tunable in one place: thresholds, alert mode, judge model and the rebaseline flag."
      },
      "typeVersion": 1
    },
    {
      "id": "a4199bbb-dd9e-4cb5-8bea-8550cf68f930",
      "name": "Note 2",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        976,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 1508,
        "height": 622,
        "content": "## 2. Load or seed the test set\n[Read more about the Data Table node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.datatable/)\n\nProfiles and cases load from Data Tables. On an empty instance the workflow seeds demo rows, so the first run works with no data entry."
      },
      "typeVersion": 1
    },
    {
      "id": "8d0c56ab-453d-494f-91b9-9a5527f6d8b5",
      "name": "Note 3",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2512,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 424,
        "height": 622,
        "content": "## 3. Resolve and validate\n[Read more about the Code node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code/)\n\nEach case is joined to its profile's model and temperature. A case with a missing profile or empty fields is flagged invalid and reported, never silently dropped."
      },
      "typeVersion": 1
    },
    {
      "id": "fcdcc8cb-b3b8-485a-be7b-c52a6d7be2e8",
      "name": "Note 4",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2960,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 1087,
        "height": 622,
        "content": "## 4. Call the model and check\n[Read more about the HTTP Request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nEach prompt runs on its own model via a generic OpenAI-compatible request. Deterministic checks grade most cases; the LLM judge scores only `llm-judge` cases."
      },
      "typeVersion": 1
    },
    {
      "id": "5dde6d60-4bde-4be8-beaa-53874417c6b5",
      "name": "Note 5",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        4080,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 1876,
        "height": 622,
        "content": "## 5. Aggregate, snapshot and compare\n[Read more about the Merge node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.merge/)\n\nPer-profile pass rate and judge score are rolled up, compared against the blessed baseline (not just the last run), and saved as a new snapshot. On rebaseline the old baseline flag is cleared first."
      },
      "typeVersion": 1
    },
    {
      "id": "2b196f9e-36dd-42fc-88e1-197a47c64601",
      "name": "Note 6",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        5984,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 870,
        "height": 622,
        "content": "## 6. Report and alert\n[Read more about the Telegram node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.telegram/)\n\nThe figures are prepared, the message is composed in a Set node you can edit, and Telegram fires on baseline capture, regression, recovery or ops failure, staying silent when quality holds."
      },
      "typeVersion": 1
    },
    {
      "id": "233ed33d-1720-42c0-8f0c-21e21fd66056",
      "name": "Setup - tables",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        976,
        768
      ],
      "parameters": {
        "color": 7,
        "width": 1000,
        "height": 250,
        "content": "**Setup - Data Tables**\nCreate three [Data Tables](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.datatable/) and select each in its nodes:\n- `eval_profiles` - profileName, systemPrompt, model (string), temperature (number)\n- `eval_cases` - profile, label, input, expected, checkType (string)\n- `eval_history` - runAt (string), isBaseline (boolean), summary (string), results (string)\n\neval_history grows every run; prune old non-baseline rows periodically."
      },
      "typeVersion": 1
    },
    {
      "id": "36834436-3b5f-4351-b9e4-b0c8705b9b2f",
      "name": "Setup - provider",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2960,
        768
      ],
      "parameters": {
        "color": 7,
        "width": 900,
        "height": 250,
        "content": "**Setup - model provider**\n1. Create an [HTTP Header Auth](https://docs.n8n.io/integrations/builtin/credentials/httprequest/) credential: name `Authorization`, value `Bearer YOUR_API_KEY`.\n2. Select it in both HTTP Request nodes.\n3. Point `apiBaseUrl` in Config at your provider.\n\nFree-tier models come and go. If one stops routing, swap `judgeModel` or the table `model` column."
      },
      "typeVersion": 1
    },
    {
      "id": "0101b7a0-d845-4ab7-b8fb-75bf5d409939",
      "name": "Setup - telegram",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        5984,
        768
      ],
      "parameters": {
        "color": 7,
        "width": 844,
        "height": 250,
        "content": "**Setup - Telegram**\nSet your [Telegram credential](https://docs.n8n.io/integrations/builtin/credentials/telegram/) and chat ID on the last node. The report is in `{{ $json.text }}` with a subject in `{{ $json.subject }}`, so this node swaps for Slack, email or a webhook."
      },
      "typeVersion": 1
    },
    {
      "id": "c1ed2463-64a5-42db-a415-fec7ec6840ed",
      "name": "Final output",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        6912,
        112
      ],
      "parameters": {
        "color": 7,
        "width": 520,
        "height": 640,
        "content": "## Final output\n\nExample of the digest this workflow sends to Telegram on a regression:\n\n**Prompt regression report** - 2026-07-13T11:55:15Z\n5 cases, 5 evaluated\n\n\ud83d\udd34 REGRESSION vs baseline (2026-07-13T11:50:35Z)\n\nPass rate: 60% (-40.0 pp vs baseline). Judge avg: 5.00.\n\n- support-writer: 100% (2/2), judge 5.00\n- support-classifier: 33% (1/3)\n\nConfirmed regressions:\n\n- angry late order\n- neutral hours question\n\nFailing cases:\n\n- angry late order - expected negative | got neutral\n- neutral hours question - expected neutral | got positive"
      },
      "typeVersion": 1
    },
    {
      "id": "81caba8e-cda6-4510-b032-abbf6ec2c116",
      "name": "Schedule",
      "type": "n8n-nodes-base.scheduleTrigger",
      "position": [
        576,
        432
      ],
      "parameters": {
        "rule": {
          "interval": [
            {
              "triggerAtHour": 8
            }
          ]
        }
      },
      "typeVersion": 1.3
    },
    {
      "id": "c45846cf-7139-4777-8583-3d77a01598d4",
      "name": "Config",
      "type": "n8n-nodes-base.set",
      "position": [
        800,
        432
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "35e595e4-12cb-46b9-8d20-1800226e72c4",
              "name": "apiBaseUrl",
              "type": "string",
              "value": "https://openrouter.ai/api/v1"
            },
            {
              "id": "83e75da9-f5d3-4bf4-bb11-a483b18937c4",
              "name": "judgeModel",
              "type": "string",
              "value": "openai/gpt-oss-20b:free"
            },
            {
              "id": "771aa8cb-0615-4e11-9200-00f6ce115a7f",
              "name": "maxCases",
              "type": "number",
              "value": 50
            },
            {
              "id": "de8d84be-5967-427e-b679-ad5f2354cde6",
              "name": "judgeScoreThreshold",
              "type": "number",
              "value": 4
            },
            {
              "id": "45868df6-ff71-4577-8130-650bc14869ab",
              "name": "regressionPassRatePp",
              "type": "number",
              "value": 5
            },
            {
              "id": "9eaf5d04-09ce-487e-acf2-01025faad4e5",
              "name": "regressionJudgeScoreDrop",
              "type": "number",
              "value": 0.5
            },
            {
              "id": "d9a142a0-8634-4200-b9cb-86ec47fa972c",
              "name": "alertMode",
              "type": "string",
              "value": "on-regression"
            },
            {
              "id": "82027b40-b229-4a12-9dc0-bb7b4814f9fb",
              "name": "rebaseline",
              "type": "boolean",
              "value": false
            }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "70e3f91e-ee78-493c-a875-4c2a2ba922f9",
      "name": "Load profiles",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        1024,
        432
      ],
      "parameters": {
        "operation": "get",
        "returnAll": true,
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_profiles"
        }
      },
      "typeVersion": 1.1,
      "alwaysOutputData": true
    },
    {
      "id": "830396fb-3d55-40ff-bfce-2374784a4fae",
      "name": "Load test cases",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        1248,
        432
      ],
      "parameters": {
        "limit": "={{ $('Config').first().json.maxCases }}",
        "operation": "get",
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_cases"
        }
      },
      "executeOnce": true,
      "typeVersion": 1.1,
      "alwaysOutputData": true
    },
    {
      "id": "848efdb2-b26e-4bf7-8e2e-f3bd79e0e18a",
      "name": "Any data?",
      "type": "n8n-nodes-base.if",
      "position": [
        1456,
        432
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "loose"
          },
          "combinator": "or",
          "conditions": [
            {
              "id": "54f07e24-c472-4058-b768-1b4f93a68939",
              "operator": {
                "type": "string",
                "operation": "exists",
                "singleValue": true
              },
              "leftValue": "={{ $json.id }}",
              "rightValue": ""
            },
            {
              "id": "fca02cbf-a95b-44ca-9d3e-a846456e5b2c",
              "operator": {
                "type": "string",
                "operation": "exists",
                "singleValue": true
              },
              "leftValue": "={{ $('Load profiles').first().json.id }}",
              "rightValue": ""
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "a3bb840a-36bb-4a26-a2a1-0596b8ca35fa",
      "name": "Build demo profiles",
      "type": "n8n-nodes-base.code",
      "position": [
        1456,
        576
      ],
      "parameters": {
        "jsCode": "const demoProfiles = [\n  {\n    profileName: 'support-classifier',\n    systemPrompt: 'Classify the sentiment of the customer message as exactly one lowercase word: positive, negative, or neutral. Reply with only that single word and nothing else.',\n    model: 'openai/gpt-oss-20b:free',\n    temperature: 0,\n  },\n  {\n    profileName: 'support-writer',\n    systemPrompt: 'You are a customer support agent. Reply to the customer in one short, polite sentence.',\n    model: 'google/gemma-4-31b-it:free',\n    temperature: 0,\n  },\n];\n\nreturn demoProfiles.map((profile) => ({ json: profile }));"
      },
      "typeVersion": 2
    },
    {
      "id": "2f82dd3d-fcf0-41a3-8678-c1b749d09259",
      "name": "Seed profiles",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        1680,
        576
      ],
      "parameters": {
        "columns": {
          "value": {
            "model": "={{ $json.model }}",
            "profileName": "={{ $json.profileName }}",
            "temperature": "={{ $json.temperature }}",
            "systemPrompt": "={{ $json.systemPrompt }}"
          },
          "schema": [
            {
              "id": "profileName",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "profileName",
              "defaultMatch": false
            },
            {
              "id": "systemPrompt",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "systemPrompt",
              "defaultMatch": false
            },
            {
              "id": "model",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "model",
              "defaultMatch": false
            },
            {
              "id": "temperature",
              "type": "number",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "temperature",
              "defaultMatch": false
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [],
          "attemptToConvertTypes": true,
          "convertFieldsToString": false
        },
        "options": {},
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_profiles"
        }
      },
      "typeVersion": 1.1
    },
    {
      "id": "80d7a07c-23f0-4ccc-8d6f-dfc36fa10cc9",
      "name": "Build demo cases",
      "type": "n8n-nodes-base.code",
      "position": [
        1904,
        576
      ],
      "parameters": {
        "jsCode": "const demoCases = [\n  { profile: 'support-classifier', label: 'angry late order', input: 'This is the third time my order is late. I want a refund now.', expected: 'negative', checkType: 'exact' },\n  { profile: 'support-classifier', label: 'happy fast delivery', input: 'Thank you so much, the delivery was incredibly fast!', expected: 'positive', checkType: 'exact' },\n  { profile: 'support-classifier', label: 'neutral hours question', input: 'What are your opening hours on Sunday?', expected: 'neutral', checkType: 'exact' },\n  { profile: 'support-writer', label: 'broken item apology', input: 'My package arrived broken.', expected: 'Reply apologizes for the problem and offers a concrete next step such as a replacement or refund.', checkType: 'llm-judge' },\n  { profile: 'support-writer', label: 'sunday opening', input: 'Are you open on Sunday?', expected: 'Sunday', checkType: 'contains' },\n];\n\nreturn demoCases.map((testCase) => ({ json: testCase }));"
      },
      "typeVersion": 2
    },
    {
      "id": "e4c08c11-961f-4854-ae6d-ca4826cf7e5c",
      "name": "Seed cases",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        2128,
        576
      ],
      "parameters": {
        "columns": {
          "value": {
            "input": "={{ $json.input }}",
            "label": "={{ $json.label }}",
            "profile": "={{ $json.profile }}",
            "expected": "={{ $json.expected }}",
            "checkType": "={{ $json.checkType }}"
          },
          "schema": [
            {
              "id": "profile",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "profile",
              "defaultMatch": false
            },
            {
              "id": "label",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "label",
              "defaultMatch": false
            },
            {
              "id": "input",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "input",
              "defaultMatch": false
            },
            {
              "id": "expected",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "expected",
              "defaultMatch": false
            },
            {
              "id": "checkType",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "checkType",
              "defaultMatch": false
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [],
          "attemptToConvertTypes": true,
          "convertFieldsToString": false
        },
        "options": {},
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_cases"
        }
      },
      "typeVersion": 1.1
    },
    {
      "id": "f86598cd-5731-4cee-9ab8-ec9b045ac727",
      "name": "Cases",
      "type": "n8n-nodes-base.noOp",
      "position": [
        2336,
        432
      ],
      "parameters": {},
      "typeVersion": 1
    },
    {
      "id": "9bb54236-8aa3-40ab-87af-b29245817824",
      "name": "Resolve profile",
      "type": "n8n-nodes-base.code",
      "position": [
        2560,
        432
      ],
      "parameters": {
        "mode": "runOnceForEachItem",
        "jsCode": "const validCheckTypes = ['exact', 'contains', 'regex', 'numeric', 'llm-judge'];\n\nconst rowsOf = (nodeName) => {\n  try {\n    return $(nodeName).all().map((item) => item.json).filter((row) => row && row.id != null);\n  } catch {\n    return [];\n  }\n};\n\nconst loadProfiles = () => {\n  const loadedRows = rowsOf('Load profiles');\n  return loadedRows.length > 0 ? loadedRows : rowsOf('Seed profiles');\n};\n\nconst isBlank = (value) => !value || String(value).trim() === '';\n\nconst findInvalidReason = (testCase, checkType, profile) => {\n  if (isBlank(testCase.input)) return 'empty input';\n  if (isBlank(testCase.expected)) return 'empty expected';\n  if (!validCheckTypes.includes(checkType)) return 'unknown checkType: ' + checkType;\n  if (!profile) return 'profile not found: ' + (testCase.profile || '(none)');\n  return null;\n};\n\nconst testCase = $input.item.json;\nconst checkType = (testCase.checkType && String(testCase.checkType).trim()) || 'exact';\nconst profile = loadProfiles().find((candidate) => candidate.profileName === testCase.profile);\nconst invalidReason = findInvalidReason(testCase, checkType, profile);\n\n$input.item.json = {\n  caseId: testCase.id ?? null,\n  label: testCase.label ?? '(unlabeled)',\n  profile: testCase.profile ?? null,\n  input: testCase.input ?? '',\n  expected: testCase.expected ?? '',\n  checkType,\n  valid: invalidReason == null,\n  reason: invalidReason ?? '',\n  systemPrompt: profile ? profile.systemPrompt : '',\n  model: profile ? profile.model : '',\n  temperature: profile ? (profile.temperature ?? 0) : 0,\n};\nreturn $input.item;"
      },
      "typeVersion": 2
    },
    {
      "id": "0c4629c0-3834-4c70-af11-8248365c5934",
      "name": "Case valid?",
      "type": "n8n-nodes-base.if",
      "position": [
        2784,
        432
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "strict"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "8a3811e0-0647-4448-9996-adea6c6c1843",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              },
              "leftValue": "={{ $json.valid }}",
              "rightValue": ""
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "864ba295-2112-41c8-81bc-8400a7c93994",
      "name": "Call model API",
      "type": "n8n-nodes-base.httpRequest",
      "onError": "continueRegularOutput",
      "maxTries": 3,
      "position": [
        3008,
        432
      ],
      "parameters": {
        "url": "={{ $('Config').first().json.apiBaseUrl }}/chat/completions",
        "method": "POST",
        "options": {
          "batching": {
            "batch": {
              "batchSize": 1,
              "batchInterval": 3000
            }
          }
        },
        "jsonBody": "={{ JSON.stringify({ model: $json.model, temperature: $json.temperature, messages: [ { role: 'system', content: $json.systemPrompt }, { role: 'user', content: $json.input } ] }) }}",
        "sendBody": true,
        "specifyBody": "json",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth"
      },
      "retryOnFail": true,
      "typeVersion": 4.2,
      "waitBetweenTries": 5000
    },
    {
      "id": "8c6430ba-d9e9-4a57-80fe-1cbf787ef0a9",
      "name": "Attach case fields",
      "type": "n8n-nodes-base.set",
      "position": [
        3216,
        432
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "edb8dfd7-fafd-46a7-b54c-348d7b7f8229",
              "name": "actualOutput",
              "type": "string",
              "value": "={{ $json.choices && $json.choices[0] ? $json.choices[0].message.content : null }}"
            },
            {
              "id": "501aaedb-2873-4689-a34b-6d74683e38b9",
              "name": "caseId",
              "type": "number",
              "value": "={{ $('Resolve profile').item.json.caseId }}"
            },
            {
              "id": "8b33f53c-ebb1-422e-9bb0-7bbf4379f708",
              "name": "label",
              "type": "string",
              "value": "={{ $('Resolve profile').item.json.label }}"
            },
            {
              "id": "42c73a7d-ceb4-4dae-a82a-ce19b4866622",
              "name": "profile",
              "type": "string",
              "value": "={{ $('Resolve profile').item.json.profile }}"
            },
            {
              "id": "6af7159f-2f0b-44fb-b322-eedac070459c",
              "name": "expected",
              "type": "string",
              "value": "={{ $('Resolve profile').item.json.expected }}"
            },
            {
              "id": "833203d4-ce9e-4be0-a60e-f2d53e645aa7",
              "name": "checkType",
              "type": "string",
              "value": "={{ $('Resolve profile').item.json.checkType }}"
            },
            {
              "id": "c5868fec-7b64-47ce-9cb7-0b34f5e42ad8",
              "name": "error",
              "type": "boolean",
              "value": "={{ $json.error != null }}"
            }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "b8227ea3-eb7a-45a8-9898-f4f3018cce93",
      "name": "Needs judge?",
      "type": "n8n-nodes-base.if",
      "position": [
        3440,
        432
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "loose"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "e52648b1-8ff2-4281-9dda-9db3dfcb9017",
              "operator": {
                "type": "string",
                "operation": "equals"
              },
              "leftValue": "={{ $json.checkType }}",
              "rightValue": "llm-judge"
            },
            {
              "id": "946e82f3-e0fd-47c4-98d0-ab1f6bdb7ac1",
              "operator": {
                "type": "boolean",
                "operation": "false",
                "singleValue": true
              },
              "leftValue": "={{ $json.error }}",
              "rightValue": ""
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "1daa662d-d794-40e6-8fc5-1c1302030aac",
      "name": "Judge output",
      "type": "n8n-nodes-base.httpRequest",
      "onError": "continueRegularOutput",
      "maxTries": 3,
      "position": [
        3664,
        272
      ],
      "parameters": {
        "url": "={{ $('Config').first().json.apiBaseUrl }}/chat/completions",
        "method": "POST",
        "options": {
          "batching": {
            "batch": {
              "batchSize": 1,
              "batchInterval": 3000
            }
          }
        },
        "jsonBody": "={{ JSON.stringify({ model: $('Config').first().json.judgeModel, temperature: 0, messages: [ { role: 'system', content: \"You are a strict evaluator. Score how well the ACTUAL OUTPUT satisfies the CRITERION on an integer 1-5 scale. Reply with ONLY compact JSON, no prose, exactly like {\\\"score\\\": 4, \\\"reason\\\": \\\"short reason\\\"}.\" }, { role: 'user', content: 'INPUT:\\n' + $json.input + '\\n\\nCRITERION:\\n' + $json.expected + '\\n\\nACTUAL OUTPUT:\\n' + $json.actualOutput } ] }) }}",
        "sendBody": true,
        "specifyBody": "json",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth"
      },
      "retryOnFail": true,
      "typeVersion": 4.2,
      "waitBetweenTries": 5000
    },
    {
      "id": "635161ea-0c5a-4fc9-9aaf-4b5dacf5ddad",
      "name": "Normalize judgement",
      "type": "n8n-nodes-base.code",
      "position": [
        3888,
        272
      ],
      "parameters": {
        "mode": "runOnceForEachItem",
        "jsCode": "const scoreRange = Object.freeze({ min: 1, max: 5 });\nconst maxReasonLength = 200;\nconst defaultJudgeScoreThreshold = 4;\n\nconst buildResultBase = (source) => ({\n  caseId: source.caseId,\n  label: source.label,\n  profile: source.profile,\n  checkType: 'llm-judge',\n  scored: true,\n});\n\nconst extractResponseText = (response) => {\n  if (response.choices && response.choices[0] && response.choices[0].message) {\n    return response.choices[0].message.content;\n  }\n  return response.text ?? response.response ?? response.output ?? (typeof response === 'string' ? response : JSON.stringify(response));\n};\n\nconst parseJudgement = (text) => {\n  try {\n    const jsonCandidate = String(text).match(/\\{[\\s\\S]*\\}/);\n    return JSON.parse(jsonCandidate ? jsonCandidate[0] : String(text));\n  } catch {\n    return null;\n  }\n};\n\nconst clampScore = (score) => Math.max(scoreRange.min, Math.min(scoreRange.max, score));\n\nconst finishWith = (result) => {\n  $input.item.json = result;\n  return $input.item;\n};\n\nconst config = $('Config').first().json;\nconst passThreshold = Number(config.judgeScoreThreshold ?? defaultJudgeScoreThreshold);\nconst source = $('Attach case fields').item.json;\nconst base = buildResultBase(source);\n\nif (source.error || source.actualOutput == null) {\n  return finishWith({ ...base, pass: null, error: true, detail: 'model call failed' });\n}\n\nconst judgement = parseJudgement(extractResponseText($input.item.json));\nif (!judgement || typeof judgement.score !== 'number') {\n  return finishWith({ ...base, pass: null, judgeError: true, detail: 'judge returned non-JSON' });\n}\n\nconst score = clampScore(judgement.score);\nreturn finishWith({\n  ...base,\n  score,\n  pass: score >= passThreshold,\n  reason: String(judgement.reason ?? '').slice(0, maxReasonLength),\n});"
      },
      "typeVersion": 2
    },
    {
      "id": "146a1cd8-66c4-480a-b322-2041926ae7b5",
      "name": "Deterministic check",
      "type": "n8n-nodes-base.code",
      "position": [
        3664,
        576
      ],
      "parameters": {
        "jsCode": "const maxDetailLength = 70;\n\nconst normalize = (value) => String(value ?? '').trim();\n\nconst firstNumber = (value) => {\n  const match = String(value).match(/-?\\d+(?:\\.\\d+)?/);\n  return match ? Number(match[0]) : NaN;\n};\n\nconst checkers = new Map([\n  ['exact', (actual, expected) => ({ pass: actual.toLowerCase() === expected.toLowerCase(), detail: '' })],\n  ['contains', (actual, expected) => ({ pass: actual.toLowerCase().includes(expected.toLowerCase()), detail: '' })],\n  ['numeric', (actual, expected) => {\n    const actualNumber = firstNumber(actual);\n    const expectedNumber = firstNumber(expected);\n    return {\n      pass: Number.isFinite(actualNumber) && Number.isFinite(expectedNumber) && actualNumber === expectedNumber,\n      detail: '',\n    };\n  }],\n  ['regex', (actual, expected) => {\n    try {\n      return { pass: new RegExp(expected).test(actual), detail: '' };\n    } catch {\n      return { pass: false, detail: 'invalid regex' };\n    }\n  }],\n]);\n\nconst runCheck = (checkType, actual, expected) => {\n  const checker = checkers.get(checkType);\n  return checker ? checker(actual, expected) : { pass: false, detail: '' };\n};\n\nconst describeFailure = (expected, actual) =>\n  'expected ' + expected.slice(0, maxDetailLength) + ' | got ' + actual.slice(0, maxDetailLength);\n\nconst toCheckResult = (caseResult) => {\n  const base = {\n    caseId: caseResult.caseId,\n    label: caseResult.label,\n    profile: caseResult.profile,\n    checkType: caseResult.checkType,\n    scored: true,\n  };\n  if (caseResult.error || caseResult.actualOutput == null) {\n    return { ...base, pass: null, error: true, detail: 'model call failed' };\n  }\n  const actual = normalize(caseResult.actualOutput);\n  const expected = normalize(caseResult.expected);\n  const { pass, detail } = runCheck(caseResult.checkType || 'exact', actual, expected);\n  return { ...base, pass, detail: detail || (pass ? '' : describeFailure(expected, actual)) };\n};\n\nreturn $input.all().map((item) => ({ json: toCheckResult(item.json) }));"
      },
      "typeVersion": 2
    },
    {
      "id": "c0256a69-0b8f-46d9-aed3-1f1edf0b4d3f",
      "name": "Merge results",
      "type": "n8n-nodes-base.merge",
      "position": [
        4208,
        432
      ],
      "parameters": {
        "numberInputs": 3
      },
      "typeVersion": 3.2
    },
    {
      "id": "d18698b4-555e-4292-af4e-c27787477123",
      "name": "Aggregate run",
      "type": "n8n-nodes-base.code",
      "position": [
        4416,
        432
      ],
      "parameters": {
        "jsCode": "const defaultConfig = Object.freeze({\n  judgeScoreThreshold: 4,\n  regressionPassRatePp: 5,\n  regressionJudgeScoreDrop: 0.5,\n  maxCases: 50,\n});\n\nclass ProfileStats {\n  constructor(profileName) {\n    this.profileName = profileName;\n    this.evaluated = 0;\n    this.passCount = 0;\n    this.judgeScoreSum = 0;\n    this.judgeScoreCount = 0;\n  }\n\n  addResult(pass) {\n    this.evaluated += 1;\n    if (pass) this.passCount += 1;\n  }\n\n  addJudgeScore(score) {\n    this.judgeScoreSum += score;\n    this.judgeScoreCount += 1;\n  }\n\n  toSummary() {\n    return {\n      profile: this.profileName,\n      evaluated: this.evaluated,\n      passCount: this.passCount,\n      passRate: this.evaluated > 0 ? this.passCount / this.evaluated : null,\n      judgeAvgScore: this.judgeScoreCount > 0 ? this.judgeScoreSum / this.judgeScoreCount : null,\n    };\n  }\n}\n\nclass RunAggregator {\n  constructor() {\n    this.total = 0;\n    this.passCount = 0;\n    this.failCount = 0;\n    this.errorCount = 0;\n    this.invalidCount = 0;\n    this.judgeScoreSum = 0;\n    this.judgeScoreCount = 0;\n    this.profiles = new Map();\n    this.perCase = [];\n    this.failing = [];\n  }\n\n  add(result) {\n    this.total += 1;\n    const label = result.label || '(unlabeled)';\n    if (result.valid === false) {\n      this.addInvalid(label, result.reason);\n    } else if (this.isErrored(result)) {\n      this.addError(label, result.judgeError === true);\n    } else {\n      this.addScored(result, label);\n    }\n  }\n\n  isErrored(result) {\n    return result.error === true || result.judgeError === true || result.pass === null || result.pass === undefined;\n  }\n\n  addInvalid(label, reason) {\n    this.invalidCount += 1;\n    this.failing.push({ label, detail: 'invalid: ' + (reason || '') });\n  }\n\n  addError(label, isJudgeError) {\n    this.errorCount += 1;\n    this.failing.push({ label, detail: isJudgeError ? 'judge error' : 'model error' });\n  }\n\n  addScored(result, label) {\n    const stats = this.profileStats(result.profile || '(none)');\n    stats.addResult(result.pass);\n    this.perCase.push({ caseId: result.caseId, pass: result.pass, label });\n    if (result.pass) {\n      this.passCount += 1;\n    } else {\n      this.failCount += 1;\n      this.failing.push({ label, detail: result.detail || String(result.reason || '') });\n    }\n    if (typeof result.score === 'number') {\n      stats.addJudgeScore(result.score);\n      this.judgeScoreSum += result.score;\n      this.judgeScoreCount += 1;\n    }\n  }\n\n  profileStats(profileName) {\n    if (!this.profiles.has(profileName)) {\n      this.profiles.set(profileName, new ProfileStats(profileName));\n    }\n    return this.profiles.get(profileName);\n  }\n\n  buildSummary(config, runAt) {\n    const evaluated = this.passCount + this.failCount;\n    const maxCases = Number(config.maxCases ?? defaultConfig.maxCases);\n    return {\n      runAt,\n      total: this.total,\n      evaluated,\n      passCount: this.passCount,\n      failCount: this.failCount,\n      errorCount: this.errorCount,\n      invalidCount: this.invalidCount,\n      passRate: evaluated > 0 ? this.passCount / evaluated : null,\n      judgeAvgScore: this.judgeScoreCount > 0 ? this.judgeScoreSum / this.judgeScoreCount : null,\n      perProfile: [...this.profiles.values()].map((stats) => stats.toSummary()),\n      truncated: this.total >= maxCases,\n      emptyDataset: this.total === 0,\n      configEcho: {\n        judgeScoreThreshold: Number(config.judgeScoreThreshold ?? defaultConfig.judgeScoreThreshold),\n        regressionPassRatePp: Number(config.regressionPassRatePp ?? defaultConfig.regressionPassRatePp),\n        regressionJudgeScoreDrop: Number(config.regressionJudgeScoreDrop ?? defaultConfig.regressionJudgeScoreDrop),\n      },\n    };\n  }\n}\n\nconst config = $('Config').first().json;\nconst aggregator = new RunAggregator();\nfor (const item of $input.all()) {\n  aggregator.add(item.json);\n}\n\nreturn [{\n  json: {\n    summary: aggregator.buildSummary(config, new Date().toISOString()),\n    perCase: aggregator.perCase,\n    failing: aggregator.failing,\n  },\n}];"
      },
      "typeVersion": 2
    },
    {
      "id": "53b8576d-878f-450b-a591-1e39e110b9ea",
      "name": "Get baseline snapshot",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        4640,
        432
      ],
      "parameters": {
        "limit": 1,
        "filters": {
          "conditions": [
            {
              "keyName": "isBaseline",
              "condition": "isTrue"
            }
          ]
        },
        "orderBy": true,
        "operation": "get",
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_history"
        },
        "orderByColumn": "runAt",
        "orderByDirection": "DESC"
      },
      "typeVersion": 1.1,
      "alwaysOutputData": true
    },
    {
      "id": "1988fb84-3074-466a-9523-90e465cef981",
      "name": "Get previous snapshot",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        4864,
        432
      ],
      "parameters": {
        "limit": 1,
        "orderBy": true,
        "operation": "get",
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_history"
        },
        "orderByColumn": "runAt",
        "orderByDirection": "DESC"
      },
      "typeVersion": 1.1,
      "alwaysOutputData": true
    },
    {
      "id": "e5a75a9f-0ef5-415f-9a4f-3540da34dab5",
      "name": "Compare to baseline",
      "type": "n8n-nodes-base.code",
      "position": [
        5088,
        432
      ],
      "parameters": {
        "jsCode": "const VERDICT = Object.freeze({\n  baseline: 'baseline',\n  rebaseline: 'rebaseline',\n  regression: 'regression',\n  recovered: 'recovered',\n  ops: 'ops',\n  ok: 'ok',\n});\n\nconst defaultThresholds = Object.freeze({\n  regressionPassRatePp: 5,\n  regressionJudgeScoreDrop: 0.5,\n});\n\nconst parseJson = (value, fallback) => {\n  if (typeof value !== 'string') return value ?? fallback;\n  try {\n    return JSON.parse(value);\n  } catch {\n    return fallback;\n  }\n};\n\nconst parseSnapshot = (rows) => {\n  if (!rows || rows.length === 0) return null;\n  const row = rows[0].json ?? rows[0];\n  if (row == null || row.id == null) return null;\n  const summary = parseJson(row.summary, null);\n  if (!summary) return null;\n  return { runAt: row.runAt, summary, results: parseJson(row.results, []) };\n};\n\nconst fetchRows = (nodeName) => {\n  try {\n    return $(nodeName).all();\n  } catch {\n    return [];\n  }\n};\n\nconst passMapOf = (results) => {\n  const map = {};\n  for (const caseResult of (results || [])) map[caseResult.caseId] = caseResult.pass;\n  return map;\n};\n\nconst classifyFlips = (currentCases, baselinePass, previousPass, hasPrevious) => {\n  const regressed = [];\n  const newFailing = [];\n  const recovered = [];\n  for (const currentCase of currentCases) {\n    const passedInBaseline = baselinePass[currentCase.caseId];\n    if (passedInBaseline === true && currentCase.pass === false) {\n      regressed.push({\n        caseId: currentCase.caseId,\n        label: currentCase.label,\n        confirmed: hasPrevious ? previousPass[currentCase.caseId] === false : false,\n      });\n    } else if (passedInBaseline === undefined && currentCase.pass === false) {\n      newFailing.push({ caseId: currentCase.caseId, label: currentCase.label });\n    } else if (passedInBaseline === false && currentCase.pass === true) {\n      recovered.push({ caseId: currentCase.caseId, label: currentCase.label });\n    }\n  }\n  return { regressed, newFailing, recovered };\n};\n\nconst deltaOrNull = (currentValue, baselineValue, scale = 1) =>\n  (currentValue != null && baselineValue != null) ? (currentValue - baselineValue) * scale : null;\n\nconst judgeThresholdChanged = (baselineSummary, currentSummary) =>\n  !!(baselineSummary.configEcho && currentSummary.configEcho\n    && baselineSummary.configEcho.judgeScoreThreshold !== currentSummary.configEcho.judgeScoreThreshold);\n\nconst detectOpsFailure = (summary) => {\n  const brokenCount = (summary.errorCount || 0) + (summary.invalidCount || 0);\n  const brokenRatio = summary.total ? brokenCount / summary.total : 0;\n  return summary.emptyDataset === true || (summary.evaluated || 0) === 0 || brokenRatio > 0.5;\n};\n\nconst captureBaseline = (hasBaseline, opsFailed) => {\n  if (opsFailed) return { verdict: VERDICT.ops, markAsBaseline: false, shouldAlert: true, compare: {} };\n  return {\n    verdict: hasBaseline ? VERDICT.rebaseline : VERDICT.baseline,\n    markAsBaseline: true,\n    shouldAlert: true,\n    compare: {},\n  };\n};\n\nconst compareAgainstBaseline = ({ baseline, previous, currentSummary, currentCases, opsFailed }) => {\n  const thresholds = currentSummary.configEcho || defaultThresholds;\n  const baselineSummary = baseline.summary;\n  const passRateDeltaPp = deltaOrNull(currentSummary.passRate, baselineSummary.passRate, 100);\n  const judgeScoreDelta = deltaOrNull(currentSummary.judgeAvgScore, baselineSummary.judgeAvgScore);\n  const flips = classifyFlips(\n    currentCases,\n    passMapOf(baseline.results),\n    previous ? passMapOf(previous.results) : {},\n    previous != null,\n  );\n  const thresholdChanged = judgeThresholdChanged(baselineSummary, currentSummary);\n  const confirmed = flips.regressed.filter((flip) => flip.confirmed);\n  const passRateDropped = passRateDeltaPp != null\n    && passRateDeltaPp <= -Number(thresholds.regressionPassRatePp ?? defaultThresholds.regressionPassRatePp);\n  const judgeScoreDropped = judgeScoreDelta != null\n    && judgeScoreDelta <= -Number(thresholds.regressionJudgeScoreDrop ?? defaultThresholds.regressionJudgeScoreDrop);\n  const isRegression = passRateDropped || confirmed.length > 0 || (judgeScoreDropped && !thresholdChanged);\n\n  const compare = {\n    passRateDeltaPp,\n    judgeScoreDelta,\n    thresholdChanged,\n    regressed: flips.regressed,\n    confirmed,\n    newFailing: flips.newFailing,\n    recovered: flips.recovered,\n    baselineRunAt: baseline.runAt,\n    prevRunAt: previous ? previous.runAt : null,\n    prevPassRate: previous ? previous.summary.passRate : null,\n  };\n\n  if (isRegression) return { verdict: VERDICT.regression, markAsBaseline: false, shouldAlert: true, compare };\n  if (opsFailed) return { verdict: VERDICT.ops, markAsBaseline: false, shouldAlert: true, compare };\n\n  const previousVerdict = previous ? previous.summary.verdict : null;\n  if (previousVerdict === VERDICT.regression || previousVerdict === VERDICT.ops) {\n    return { verdict: VERDICT.recovered, markAsBaseline: false, shouldAlert: true, compare };\n  }\n  return { verdict: VERDICT.ok, markAsBaseline: false, shouldAlert: false, compare };\n};\n\nconst decideVerdict = ({ baseline, previous, currentSummary, currentCases, config, opsFailed }) => {\n  const rebaselineRequested = config.rebaseline === true || config.rebaseline === 'true';\n  if (!baseline || rebaselineRequested) return captureBaseline(baseline != null, opsFailed);\n  return compareAgainstBaseline({ baseline, previous, currentSummary, currentCases, opsFailed });\n};\n\nconst current = $('Aggregate run').first().json;\nconst currentSummary = current.summary;\nconst currentCases = current.perCase || [];\nconst config = $('Config').first().json;\n\nconst decision = decideVerdict({\n  baseline: parseSnapshot(fetchRows('Get baseline snapshot')),\n  previous: parseSnapshot(fetchRows('Get previous snapshot')),\n  currentSummary,\n  currentCases,\n  config,\n  opsFailed: detectOpsFailure(currentSummary),\n});\n\nconst shouldAlert = decision.shouldAlert || config.alertMode === 'always';\nconst storedVerdict = decision.verdict === VERDICT.recovered ? VERDICT.ok : decision.verdict;\n\nreturn [{\n  json: {\n    verdict: decision.verdict,\n    shouldAlert,\n    markAsBaseline: decision.markAsBaseline,\n    compare: decision.compare,\n    failing: current.failing || [],\n    summary: currentSummary,\n    summaryJson: JSON.stringify({ ...currentSummary, verdict: storedVerdict }),\n    resultsJson: JSON.stringify(currentCases),\n  },\n}];"
      },
      "typeVersion": 2
    },
    {
      "id": "5a763137-f9b2-4677-9401-656159bf6706",
      "name": "Save snapshot",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        5744,
        432
      ],
      "parameters": {
        "columns": {
          "value": {
            "runAt": "={{ $('Compare to baseline').first().json.summary.runAt }}",
            "results": "={{ $('Compare to baseline').first().json.resultsJson }}",
            "summary": "={{ $('Compare to baseline').first().json.summaryJson }}",
            "isBaseline": "={{ $('Compare to baseline').first().json.markAsBaseline }}"
          },
          "schema": [
            {
              "id": "runAt",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "runAt",
              "defaultMatch": false
            },
            {
              "id": "isBaseline",
              "type": "boolean",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "isBaseline",
              "defaultMatch": false
            },
            {
              "id": "summary",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "summary",
              "defaultMatch": false
            },
            {
              "id": "results",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "results",
              "defaultMatch": false
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [],
          "attemptToConvertTypes": true,
          "convertFieldsToString": false
        },
        "options": {},
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_history"
        }
      },
      "executeOnce": true,
      "typeVersion": 1.1
    },
    {
      "id": "22b15caa-45a3-4ec8-b634-c8f481eb10c2",
      "name": "Prepare report fields",
      "type": "n8n-nodes-base.code",
      "position": [
        6064,
        432
      ],
      "parameters": {
        "jsCode": "const maxFailingDetails = 5;\nconst maxLabelsPerSection = 10;\n\nconst escapeHtml = (value) => String(value ?? '')\n  .replace(/&/g, '&amp;')\n  .replace(/</g, '&lt;')\n  .replace(/>/g, '&gt;');\n\nconst formatPercent = (rate) => (rate == null ? 'n/a' : (rate * 100).toFixed(0) + '%');\n\nconst formatSignedDelta = (value) => (value == null ? '' : (value >= 0 ? '+' : '') + value.toFixed(1));\n\nconst capLabels = (entries, limit) => {\n  const labels = entries.map((entry) => escapeHtml(entry.label));\n  if (labels.length <= limit) return labels;\n  return labels.slice(0, limit).concat(['+' + (labels.length - limit) + ' more']);\n};\n\nconst verdictLines = {\n  baseline: () => '\u2705 BASELINE CAPTURED - future runs compare against this run.',\n  rebaseline: () => '\u2705 NEW BASELINE SET - switch rebaseline back to false in Config.',\n  regression: (comparison) => '\ud83d\udd34 REGRESSION vs baseline (' + escapeHtml(comparison.baselineRunAt || '') + ')',\n  ops: () => '\u26a0\ufe0f OPS ISSUE - the eval could not run cleanly (see below).',\n  recovered: () => '\ud83d\udfe2 RECOVERED - quality is back in line with baseline.',\n  ok: () => '\u2705 OK - no regression vs baseline.',\n};\n\nconst buildVerdictLine = (verdict, comparison) => {\n  const build = verdictLines[verdict] ?? verdictLines.ok;\n  return build(comparison);\n};\n\nconst buildHeader = (summary) =>\n  '<b>Prompt regression report</b>\\n' + escapeHtml(summary.runAt) + '\\n'\n  + summary.total + ' cases, ' + summary.evaluated + ' evaluated'\n  + (summary.truncated ? ' \u26a0\ufe0f at maxCases limit' : '');\n\nconst buildMetrics = (summary, comparison) => {\n  const lines = [];\n  lines.push('Pass rate: ' + formatPercent(summary.passRate)\n    + (comparison.passRateDeltaPp != null ? ' (' + formatSignedDelta(comparison.passRateDeltaPp) + ' pp vs baseline)' : ''));\n  if (summary.judgeAvgScore != null) {\n    lines.push('Judge avg: ' + summary.judgeAvgScore.toFixed(2)\n      + (comparison.judgeScoreDelta != null ? ' (' + formatSignedDelta(comparison.judgeScoreDelta) + ')' : ''));\n  }\n  for (const profile of (summary.perProfile || [])) {\n    lines.push('- ' + escapeHtml(profile.profile) + ': ' + formatPercent(profile.passRate)\n      + ' (' + profile.passCount + '/' + profile.evaluated + ')'\n      + (profile.judgeAvgScore != null ? ', judge ' + profile.judgeAvgScore.toFixed(2) : ''));\n  }\n  return '<b>Metrics</b>\\n' + lines.join('\\n');\n};\n\nconst buildFailingSection = (failing) => {\n  const lines = failing.slice(0, maxFailingDetails)\n    .map((entry) => escapeHtml(entry.label) + ' - ' + escapeHtml(entry.detail || ''));\n  if (failing.length > maxFailingDetails) lines.push('+' + (failing.length - maxFailingDetails) + ' more');\n  return '<b>Failing cases</b>\\n' + lines.join('\\n');\n};\n\nconst buildOpsSection = (summary) =>\n  '<b>Ops issues</b>\\n'\n  + (summary.errorCount ? summary.errorCount + ' model/judge errors\\n' : '')\n  + (summary.invalidCount ? summary.invalidCount + ' invalid cases' : '');\n\nconst report = $('Compare to baseline').first().json;\nconst summary = report.summary;\nconst comparison = report.compare || {};\nconst unconfirmed = (comparison.regressed || []).filter((entry) => !entry.confirmed);\n\nconst sections = [];\nsections.push(buildHeader(summary));\nsections.push(buildVerdictLine(report.verdict, comparison));\nsections.push(buildMetrics(summary, comparison));\n\nif (comparison.thresholdChanged) {\n  sections.push('\u2139\ufe0f judgeScoreThreshold changed since baseline - flips are not treated as regressions this run.');\n}\nif ((comparison.confirmed || []).length) {\n  sections.push('<b>Confirmed regressions</b>\\n' + capLabels(comparison.confirmed, maxLabelsPerSection).join('\\n'));\n}\nif (unconfirmed.length) {\n  sections.push('<b>Regressions (unconfirmed, will confirm next run)</b>\\n' + capLabels(unconfirmed, maxLabelsPerSection).join('\\n'));\n}\nif ((comparison.newFailing || []).length) {\n  sections.push('<b>New failures (unconfirmed, possibly flaky)</b>\\n' + capLabels(comparison.newFailing, maxLabelsPerSection).join('\\n'));\n}\nif ((report.failing || []).length) {\n  sections.push(buildFailingSection(report.failing));\n}\nif (summary.errorCount || summary.invalidCount) {\n  sections.push(buildOpsSection(summary));\n}\nif ((comparison.recovered || []).length) {\n  sections.push('<b>Recovered</b>\\n' + capLabels(comparison.recovered, maxLabelsPerSection).join('\\n'));\n}\nif (comparison.prevRunAt) {\n  sections.push('Previous run ' + escapeHtml(comparison.prevRunAt) + ': pass rate ' + formatPercent(comparison.prevPassRate));\n}\n\nconst subject = 'Prompt eval: ' + String(report.verdict).toUpperCase() + ' - pass ' + formatPercent(summary.passRate);\nreturn [{ json: { sections, subject } }];"
      },
      "typeVersion": 2
    },
    {
      "id": "8168aff1-7c7f-423e-87aa-22e802bf38dd",
      "name": "Compose message",
      "type": "n8n-nodes-base.set",
      "position": [
        6256,
        432
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "236d265e-961d-4b1e-8178-9961862970b8",
              "name": "subject",
              "type": "string",
              "value": "={{ $json.subject }}"
            },
            {
              "id": "42002b7c-fa77-4c08-8b81-6fe676457512",
              "name": "text",
              "type": "string",
              "value": "={{ $json.sections.join('\\n\\n') }}"
            }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "8cb5db43-8602-4416-9dc4-a52aa3bd82c0",
      "name": "Alert needed?",
      "type": "n8n-nodes-base.if",
      "position": [
        6448,
        432
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "loose"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "1a4b189d-aebb-4e1a-9325-a942f710ec5c",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              },
              "leftValue": "={{ $('Compare to baseline').first().json.shouldAlert }}",
              "rightValue": ""
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "d57d5d05-e01c-41d4-94a2-c996e2cdefb6",
      "name": "Send report to Telegram",
      "type": "n8n-nodes-base.telegram",
      "position": [
        6640,
        432
      ],
      "parameters": {
        "text": "={{ $json.text }}",
        "chatId": "YOUR_CHAT_ID",
        "additionalFields": {
          "parse_mode": "HTML",
          "appendAttribution": false
        }
      },
      "typeVersion": 1.2
    },
    {
      "id": "ad3593b6-9e67-41ec-bc1a-75088100d996",
      "name": "New baseline?",
      "type": "n8n-nodes-base.if",
      "position": [
        5296,
        432
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "loose"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "381fed13-c987-4473-b0b9-bcc75595e68e",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              },
              "leftValue": "={{ $json.markAsBaseline }}",
              "rightValue": ""
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "7cef2e16-19b4-4e7b-97f4-08fad5254d24",
      "name": "Clear old baseline flag",
      "type": "n8n-nodes-base.dataTable",
      "position": [
        5520,
        272
      ],
      "parameters": {
        "columns": {
          "value": {
            "isBaseline": false
          },
          "schema": [
            {
              "id": "runAt",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "runAt",
              "defaultMatch": false
            },
            {
              "id": "isBaseline",
              "type": "boolean",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "isBaseline",
              "defaultMatch": false
            },
            {
              "id": "summary",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "summary",
              "defaultMatch": false
            },
            {
              "id": "results",
              "type": "string",
              "display": true,
              "removed": false,
              "readOnly": false,
              "required": false,
              "displayName": "results",
              "defaultMatch": false
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [],
          "attemptToConvertTypes": true,
          "convertFieldsToString": false
        },
        "filters": {
          "conditions": [
            {
              "keyName": "isBaseline",
              "condition": "isTrue"
            }
          ]
        },
        "options": {},
        "operation": "update",
        "dataTableId": {
          "__rl": true,
          "mode": "list",
          "value": "",
          "cachedResultName": "eval_history"
        }
      },
      "typeVersion": 1.1,
      "alwaysOutputData": true
    }
  ],
  "settings": {
    "executionOrder": "v1"
  },
  "connections": {
    "Cases": {
      "main": [
        [
          {
            "node": "Resolve profile",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Config": {
      "main": [
        [
          {
            "node": "Load profiles",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Schedule": {
      "main": [
        [
          {
            "node": "Config",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Any data?": {
      "main": [
        [
          {
            "node": "Cases",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Build demo profiles",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Seed cases": {
      "main": [
        [
          {
            "node": "Cases",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Case valid?": {
      "main": [
        [
          {
            "node": "Call model API",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Merge results",
            "type": "main",
            "index": 2
          }
        ]
      ]
    },
    "Judge output": {
      "main": [
        [
          {
            "node": "Normalize judgement"

About this workflow

Continuous quality checks for your production LLM prompts. Keeps a golden test set in n8n Data Tables, runs it on a schedule against any OpenAI-compatible API, compares each run to a blessed baseline, and pings Telegram only when quality actually changes. A Schedule Trigger…

Source: https://n8n.io/workflows/17142/ — original creator credit. Request a take-down →

More AI & RAG workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.