AutomationFlowsWeb Scraping › Research Watch (scheduled)

Research Watch (scheduled)

Research watch (scheduled). Uses executeWorkflowTrigger, httpRequest. Scheduled trigger; 9 nodes.

Cron / scheduled trigger★★★★☆ complexity9 nodesExecute Workflow TriggerHTTP Request
Web Scraping Trigger: Cron / scheduled Nodes: 9 Complexity: ★★★★☆ Added:

This workflow follows the Execute Workflow Trigger → HTTP Request recipe pattern — see all workflows that pair these two integrations.

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

Download .json
{
  "id": "aicp-research-watch",
  "name": "Research watch (scheduled)",
  "settings": {
    "errorWorkflow": "aicp-error-trigger"
  },
  "nodes": [
    {
      "id": "trigger",
      "name": "Weekly (Mon 07:30)",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [
        200,
        300
      ],
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "weeks",
              "triggerAtDay": [
                1
              ],
              "triggerAtHour": 7,
              "triggerAtMinute": 30
            }
          ]
        }
      }
    },
    {
      "id": "manual",
      "name": "Run on demand",
      "type": "n8n-nodes-base.executeWorkflowTrigger",
      "typeVersion": 1.1,
      "position": [
        200,
        460
      ],
      "parameters": {
        "inputSource": "passthrough"
      }
    },
    {
      "id": "fetch",
      "name": "Fetch research",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        420,
        300
      ],
      "parameters": {
        "url": "http://lanes:8081/research?days=7",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "x-lanes-token",
              "value": "={{ $env.LANES_TOKEN }}"
            }
          ]
        },
        "options": {
          "timeout": 120000
        }
      }
    },
    {
      "id": "triage",
      "name": "Triage findings",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        640,
        300
      ],
      "parameters": {
        "jsCode": "// Decide whether this week is worth a MODEL CALL, and build the prompt if so.\n//\n// Cost-awareness: most weeks nothing we depend on moves a decision. Routing a\n// prompt anyway would spend capacity to be told \"no change\" \u2014 so the IF node\n// downstream skips the agent entirely on a quiet week. The lane still posts,\n// because \"we checked and it was quiet\" is a coverage signal worth seeing;\n// silence would be indistinguishable from the lane being broken.\nconst d = $input.first().json || {};\nconst NL = String.fromCharCode(10);\n\n// A failed fetch must never read as a clean bill of health.\nconst partial = d.checked_all_sources === false;\nconst drift = d.version_drift || [];\nconst notable = (d.items || []).filter(i => i.notable);\n\nif (d.releases === undefined) {\n  return [{ json: { needsAgent: false, severity: 'warn',\n    title: 'Research lane: the lane call itself failed',\n    message: ['Response had no shape we recognise.',\n              JSON.stringify(d).slice(0, 400)].join(NL),\n    source: 'research-watch' } }];\n}\n\n// Facts the agent reasons over. Each item already carries WHICH RECORDED\n// DECISION it could move \u2014 the runner is firewalled and cannot look anything\n// up, so everything it needs has to travel in the prompt.\nconst facts = notable.map(i =>\n  `- ${i.repo} ${i.tag} (${i.published}) [${i.running ? 'WE RUN THIS' : 'watched candidate \u2014 NOT deployed'}]` + NL +\n  `  signals=[${i.signals.join(', ')}]` + NL +\n  `  we depend on it because: ${i.why}` + NL +\n  `  it could move: ${i.decision}` + NL +\n  `  release says: ${(i.excerpt || '').slice(0, 300)}`\n).join(NL);\n\n// The OUTCOME, phrased as a headline \u2014 \"n8n is 2 minor versions behind\"\n// rather than \"the watch ran and found 2 things\". Rule 1 of the message\n// contract: a title that only names the job makes the reader open the body\n// to learn whether anything happened.\nconst driftLead = drift.length\n  ? `${drift[0].repo.split('/').pop()} is ${drift[0].behind.replace('0 major, ', '')} version(s) behind` +\n    ` (${drift[0].pinned} \u2192 ${drift[0].latest})` +\n    (drift.length > 1 ? ` +${drift.length - 1} more` : '')\n  : '';\n\nconst driftText = drift.map(r =>\n  `- ${r.repo}: we PIN ${r.pinned}, released ${r.latest} on ${r.published} (${r.behind} behind)`\n).join(NL);\n\nconst prompt = [\n  'You advise the maintainers of an AI agent control plane. Below is a watch',\n  'report generated by a tool, not a human. It covers two different things and',\n  'the difference matters: components WE RUN, where a release is an operational',\n  'event, and WATCHED CANDIDATES we have deliberately NOT deployed, where a',\n  'release is at most an input to a future decision. Each entry is labelled and',\n  'names the recorded decision it might move.',\n  '',\n  'BE SKEPTICAL. Most releases move no decision. Saying \"nothing here warrants a',\n  'change\" is a correct and valuable answer; inventing work is not. Do not',\n  'recommend adopting anything merely because it is new.',\n  '',\n  'RELEASES CARRYING A DECISION-MOVING SIGNAL:',\n  facts || '(none)',\n  '',\n  'VERSION DRIFT (what we pin vs what shipped):',\n  driftText || '(none)',\n  '',\n  partial ? 'WARNING: some sources were UNREACHABLE, so this is a PARTIAL view: ' +\n            (d.unreachable || []).join('; ') : '',\n  '',\n  'Answer in at most 5 short bullets. For each, state: the decision affected, whether',\n  'it actually moves, and the concrete next action (or \"no action\"). Name any claim you',\n  'could not verify from the text above rather than filling the gap.',\n].join(NL);\n\nconst severity = drift.length || partial ? 'warn' : (notable.length ? 'info' : 'info');\nreturn [{ json: {\n  needsAgent: notable.length > 0 || drift.length > 0,\n  severity, prompt,\n  releases: d.releases, notableCount: notable.length,\n  driftCount: drift.length, partial,\n  unreachable: (d.unreachable || []).join('; '),\n  driftText, driftLead, source: 'research-watch',\n} }];"
      }
    },
    {
      "id": "gate",
      "name": "Anything to reason about?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        860,
        300
      ],
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "version": 2,
            "leftValue": "",
            "typeValidation": "strict"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "needsAgent",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              },
              "leftValue": "={{ $json.needsAgent }}",
              "rightValue": ""
            }
          ]
        },
        "options": {}
      }
    },
    {
      "id": "ask",
      "name": "Ask the agent",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        1100,
        200
      ],
      "parameters": {
        "method": "POST",
        "url": "http://router:8080/route",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "x-router-token",
              "value": "={{ $env.ROUTER_TOKEN }}"
            }
          ]
        },
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ prompt: $json.prompt, task_type: 'plan', action: 'advise', trigger: 'schedule' }) }}",
        "options": {
          "timeout": 600000
        }
      }
    },
    {
      "id": "proposal",
      "name": "Format proposal",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1320,
        200
      ],
      "parameters": {
        "jsCode": "// Reshape the router's decision + the agent's advice into a notification.\n//\n// The router's reply is the AUTHORITY on what happened: if it blocked the call\n// (circuit breaker, mode, guardrail) that is the story, not a missing answer.\nconst r = $input.first().json || {};\nconst t = $('Triage findings').first().json || {};\nconst NL = String.fromCharCode(10);\n\nif (r.blocked) {\n  return [{ json: {\n    severity: 'warn',\n    title: 'Research lane: the router BLOCKED the analysis',\n    message: ['blocked: ' + r.blocked, r.message || '',\n              'The findings below were still collected:', t.driftText || '(no drift)'].join(NL),\n    source: 'research-watch',\n  } }];\n}\n\n// claude-runner returns { result: { result: \"<text>\" } }; older shapes nest\n// differently. Pull the text without assuming one, and say so if we cannot.\nconst inner = r.result || {};\nconst advice = (typeof inner === 'string') ? inner\n             : (typeof inner.result === 'string') ? inner.result\n             : (inner.result && inner.result.result) || '';\nconst dec = r.decision || {};\nconst usage = r.usage || {};\n\n// Outcome, not event. No severity emoji here \u2014 Notify prepends one, and two\n// would just be noise.\nconst head = t.driftLead\n  ? t.driftLead\n  : (t.notableCount\n      ? `${t.notableCount} release(s) could move a recorded decision`\n      : 'Nothing moved a recorded decision');\n\nconst body = [\n  t.driftText || '',\n  t.partial ? 'PARTIAL VIEW \u2014 unreachable: ' + t.unreachable : '',\n  '',\n  advice ? advice.slice(0, 2500)\n         : 'The agent returned no text. This is a lane fault, not a quiet week.',\n  '',\n  `_tier ${dec.tier || '?'} \u00b7 ${dec.model || '?'}${dec.clamped ? ' (clamped by cost ceiling)' : ''}` +\n  ` \u00b7 notional $${(usage.notionalCostUsd || 0).toFixed(4)}_`,\n].filter(Boolean).join(NL);\n\n// Contract fields (notification-taxonomy.md \"The message contract\"):\n// an explicit action line, the one number that matters, and a deep link to\n// THIS execution. $execution.id here is the lane's own run, which is what a\n// reader needs \u2014 the Notify sub-workflow has its own id and cannot derive it.\nconst base = ($env.N8N_BASE_URL || 'http://localhost:5678').replace(/\\/+$/, '');\nconst action = t.driftCount\n  ? 'Act by the next pin review (2026-08-24) \u2014 queued in context/image-policy.json'\n  : (t.notableCount ? 'Decide this week' : 'No action');\nreturn [{ json: {\n  severity: t.severity || 'info',\n  title: head,\n  action,\n  metric: `${t.notableCount} of ${t.releases} release(s) carry a signal` +\n          (t.driftCount ? `, ${t.driftCount} pin behind` : ''),\n  runUrl: `${base}/workflow/${$workflow.id}/executions/${$execution.id}`,\n  message: body,\n  source: 'research-watch',\n} }];"
      }
    },
    {
      "id": "quiet",
      "name": "Format quiet week",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1100,
        420
      ],
      "parameters": {
        "jsCode": "// A quiet week still posts. Coverage-first (notification-taxonomy.md): a lane\n// that only speaks when it finds something is indistinguishable from a lane that\n// stopped running, and the second failure is the dangerous one.\nconst t = $input.first().json || {};\nconst NL = String.fromCharCode(10);\nif (t.severity === 'warn' && t.title) return [{ json: t }];   // lane fault, pass through\nreturn [{ json: {\n  severity: t.partial ? 'warn' : 'info',\n  title: 'Nothing moved a recorded decision',\n  action: 'No action',\n  metric: `0 of ${t.releases} release(s) carry a signal`,\n  message: [\n    `Checked 6 sources; ${t.releases} release(s) in the window, none carrying a`,\n    'decision-moving signal, and no version drift against context/image-policy.json.',\n    t.partial ? 'PARTIAL VIEW \u2014 these sources were NOT checked: ' + t.unreachable : '',\n    '_No model call was made \u2014 nothing to reason about._',\n  ].filter(Boolean).join(NL),\n  source: 'research-watch',\n} }];"
      }
    },
    {
      "id": "notify",
      "name": "Notify",
      "type": "n8n-nodes-base.executeWorkflow",
      "typeVersion": 1.2,
      "position": [
        1560,
        300
      ],
      "parameters": {
        "source": "database",
        "workflowId": {
          "__rl": true,
          "value": "aicp-notify",
          "mode": "id"
        },
        "options": {
          "waitForSubWorkflow": true
        },
        "workflowInputs": {
          "mappingMode": "autoMapInputData",
          "value": {},
          "matchingColumns": [],
          "schema": [
            {
              "id": "severity",
              "displayName": "severity",
              "type": "string",
              "required": false,
              "defaultMatch": false,
              "display": true,
              "canBeUsedToMatch": true
            },
            {
              "id": "title",
              "displayName": "title",
              "type": "string",
              "required": false,
              "defaultMatch": false,
              "display": true,
              "canBeUsedToMatch": true
            },
            {
              "id": "message",
              "displayName": "message",
              "type": "string",
              "required": false,
              "defaultMatch": false,
              "display": true,
              "canBeUsedToMatch": true
            },
            {
              "id": "source",
              "displayName": "source",
              "type": "string",
              "required": false,
              "defaultMatch": false,
              "display": true,
              "canBeUsedToMatch": true
            }
          ],
          "attemptToConvertTypes": false,
          "convertFieldsToString": true
        }
      }
    }
  ],
  "connections": {
    "Weekly (Mon 07:30)": {
      "main": [
        [
          {
            "node": "Fetch research",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Run on demand": {
      "main": [
        [
          {
            "node": "Fetch research",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Fetch research": {
      "main": [
        [
          {
            "node": "Triage findings",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Triage findings": {
      "main": [
        [
          {
            "node": "Anything to reason about?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Anything to reason about?": {
      "main": [
        [
          {
            "node": "Ask the agent",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Format quiet week",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Ask the agent": {
      "main": [
        [
          {
            "node": "Format proposal",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Format proposal": {
      "main": [
        [
          {
            "node": "Notify",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Format quiet week": {
      "main": [
        [
          {
            "node": "Notify",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "meta": {
    "description": "Weekly watch over the six projects tied to a recorded decision. Fetches in `lanes` (the runners are firewalled), asks an agent which decisions actually move, and skips the model call entirely on a quiet week."
  }
}
Pro

For the full experience including quality scoring and batch install features for each workflow upgrade to Pro

About this workflow

Research watch (scheduled). Uses executeWorkflowTrigger, httpRequest. Scheduled trigger; 9 nodes.

Source: https://github.com/jgobuilds/ai-control-plane-public/blob/main/n8n-workflows/research-watch.workflow.json — original creator credit. Request a take-down →

More Web Scraping workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.

Web Scraping

Tired of being let down by the Google Drive Trigger? Rather not exhaust system resources by polling every minute? Then this workflow is for you!

HTTP Request, Execute Workflow Trigger
Web Scraping

Proactively alert to service endpoint changes and pod/container issues (Pending, Not Ready, Restart spikes) using Prometheus metrics, formatted and sent to Slack.

HTTP Request
Web Scraping

Triggers at a regular interval or via a webhook request. Solves AWS WAF challenge then makes a request to fetch the product page. Extracts product data from the retrieved HTML page. Compares the curre

N8N Nodes Capsolver, HTTP Request
Web Scraping

🔄 Monitor Container Images from Docker Hub or GHCR.

HTTP Request
Web Scraping

Automatically monitor billable Kimai projects every weekday morning and receive a formatted HTML email when a project deadline is approaching or its hour budget is running low. If nothing requires att

Email Send, HTTP Request