AutomationFlowsWeb Scraping › Wf-main - Xhs Pipeline Controller

Wf-main - Xhs Pipeline Controller

WF-Main - XHS Pipeline Controller. Uses httpRequest. Scheduled trigger; 22 nodes.

Cron / scheduled trigger★★★★☆ complexity22 nodesHTTP Request
Web Scraping Trigger: Cron / scheduled Nodes: 22 Complexity: ★★★★☆ Added:

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

Download .json
{
  "name": "WF-Main - XHS Pipeline Controller",
  "nodes": [
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "minutes",
              "minutesInterval": 5
            }
          ]
        }
      },
      "id": "schedule-trigger",
      "name": "Schedule Trigger",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.3,
      "position": [
        0,
        0
      ],
      "notes": "\u6bcf5\u5206\u949f\u81ea\u52a8\u89e6\u53d1"
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "path": "xhs-pipeline",
        "responseMode": "onReceived",
        "options": {}
      },
      "id": "webhook-trigger",
      "name": "Webhook Trigger",
      "type": "n8n-nodes-base.webhook",
      "typeVersion": 2.1,
      "position": [
        0,
        200
      ],
      "onError": "continueRegularOutput",
      "notes": "\u624b\u52a8\u89e6\u53d1: POST /webhook/xhs-pipeline\nBody: { action: 'discovery|extraction|generation|publish|all', target_ids?: [], force?: true }"
    },
    {
      "parameters": {
        "mode": "chooseBranch",
        "output": "empty"
      },
      "id": "merge-triggers",
      "name": "Merge Triggers",
      "type": "n8n-nodes-base.merge",
      "typeVersion": 3.2,
      "position": [
        250,
        100
      ],
      "notes": "\u5408\u5e76\u5b9a\u65f6\u548c\u624b\u52a8\u89e6\u53d1\u5165\u53e3"
    },
    {
      "parameters": {
        "method": "GET",
        "url": "={{ $env.CRAWLER_API_URL || 'http://124.221.251.8:8080' }}/api/health",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "X-API-Key",
              "value": "={{ $env.CRAWLER_API_KEY || 'dev-key' }}"
            }
          ]
        },
        "options": {
          "timeout": 10000
        }
      },
      "id": "health-check",
      "name": "Health Check",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        450,
        100
      ],
      "onError": "continueRegularOutput",
      "notes": "\u68c0\u67e5 MediaCrawler API \u5065\u5eb7\u72b6\u6001"
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict"
          },
          "conditions": [
            {
              "id": "crawler-ready-check",
              "leftValue": "={{ $json.crawler_ready }}",
              "rightValue": true,
              "operator": {
                "type": "boolean",
                "operation": "equals"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "if-crawler-ready",
      "name": "IF Crawler Ready",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.3,
      "position": [
        650,
        100
      ],
      "notes": "\u68c0\u67e5 crawler_ready \u662f\u5426\u4e3a true"
    },
    {
      "parameters": {
        "method": "POST",
        "url": "={{ $env.CRAWLER_API_URL || 'http://124.221.251.8:8080' }}/api/crawler/cookies",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "X-API-Key",
              "value": "={{ $env.CRAWLER_API_KEY || 'dev-key' }}"
            },
            {
              "name": "Content-Type",
              "value": "application/json"
            }
          ]
        },
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            {
              "name": "cookies",
              "value": "[]"
            }
          ]
        },
        "options": {}
      },
      "id": "trigger-cookie-refresh",
      "name": "Trigger Cookie Refresh",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        850,
        250
      ],
      "onError": "continueRegularOutput",
      "notes": "\u5f53\u722c\u866b\u4e0d\u5065\u5eb7\u65f6\uff0c\u89e6\u53d1 Cookie \u5237\u65b0"
    },
    {
      "parameters": {
        "jsCode": "// \u83b7\u53d6\u89e6\u53d1\u6765\u6e90\u6570\u636e\nconst triggerData = $input.first().json;\nconst webhookBody = triggerData.body || {};\nconst forceAction = webhookBody.action || null;\nconst forceTargetIds = webhookBody.target_ids || [];\nconst forceMode = webhookBody.force || false;\n\n// \u914d\u7f6e\u98de\u4e66 API\nconst LARK_API_BASE = $env.LARK_API_BASE || 'https://open.feishu.cn/open-apis/bitable/v1';\nconst LARK_APP_TOKEN = $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh';\n\n// \u8868\u683c ID \u914d\u7f6e\nconst TABLES = {\n  keywords: $env.KEYWORDS_TABLE_ID || 'tblKeywords',\n  topics: $env.TOPICS_TABLE_ID || 'tblE2SypBdIhJVrR',\n  source: $env.SOURCE_TABLE_ID || 'tblPCp5gqgVFnhLc',\n  content: $env.CONTENT_TABLE_ID || 'tblMYjwzOkYpW4AX'\n};\n\n// \u67e5\u8be2\u6761\u4ef6\u914d\u7f6e\nconst QUERIES = {\n  discovery: { table: 'keywords', status: '\u5f85\u91c7\u96c6', limit: 5 },\n  extraction: { table: 'topics', status: '\u5f85\u63d0\u53d6', limit: 10 },\n  generation: { table: 'source', status: '\u5f85\u751f\u6210', limit: 5 },\n  publish: { table: 'content', status: '\u5f85\u53d1\u5e03', limit: 3 }\n};\n\n// \u4f18\u5148\u7ea7\u987a\u5e8f (\u9ad8 \u2192 \u4f4e)\nconst PRIORITY_ORDER = ['publish', 'generation', 'extraction', 'discovery'];\n\n// \u6a21\u62df\u67e5\u8be2\u7ed3\u679c (\u5b9e\u9645\u5e94\u901a\u8fc7 HTTP Request \u8282\u70b9\u67e5\u8be2\u98de\u4e66)\n// \u8fd9\u91cc\u8fd4\u56de\u7ed3\u6784\u5316\u6570\u636e\u4f9b\u540e\u7eed\u8282\u70b9\u4f7f\u7528\nconst result = {\n  timestamp: new Date().toISOString(),\n  trigger_source: forceAction ? 'webhook' : 'schedule',\n  force_action: forceAction,\n  force_target_ids: forceTargetIds,\n  force_mode: forceMode,\n  queries: QUERIES,\n  tables: TABLES,\n  priority_order: PRIORITY_ORDER,\n  // \u5f85\u67e5\u8be2\u7684\u4efb\u52a1\n  pending_queries: PRIORITY_ORDER.map(action => ({\n    action,\n    table_id: TABLES[QUERIES[action].table],\n    status_filter: QUERIES[action].status,\n    limit: QUERIES[action].limit\n  }))\n};\n\n// \u5982\u679c\u662f\u5f3a\u5236\u6a21\u5f0f\uff0c\u76f4\u63a5\u8fd4\u56de\u6307\u5b9a\u7684 action\nif (forceAction && forceAction !== 'all') {\n  result.selected_action = forceAction;\n  result.skip_query = true;\n}\n\nreturn [result];"
      },
      "id": "prepare-queries",
      "name": "Prepare Queries",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        850,
        0
      ],
      "notes": "\u51c6\u5907\u98de\u4e66\u67e5\u8be2\u53c2\u6570\u548c\u4f18\u5148\u7ea7\u914d\u7f6e"
    },
    {
      "parameters": {
        "method": "GET",
        "url": "=https://open.feishu.cn/open-apis/bitable/v1/apps/{{ $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh' }}/tables/{{ $json.tables.content }}/records?filter=CurrentValue.[status]=\"\u5f85\u53d1\u5e03\"&page_size=3",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "=Bearer {{ $env.LARK_ACCESS_TOKEN }}"
            }
          ]
        },
        "options": {
          "timeout": 15000
        }
      },
      "id": "query-content",
      "name": "Query Content (\u5f85\u53d1\u5e03)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        1100,
        -150
      ],
      "onError": "continueRegularOutput",
      "notes": "\u67e5\u8be2\u5f85\u53d1\u5e03\u5185\u5bb9 (\u4f18\u5148\u7ea71)"
    },
    {
      "parameters": {
        "method": "GET",
        "url": "=https://open.feishu.cn/open-apis/bitable/v1/apps/{{ $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh' }}/tables/{{ $json.tables.source }}/records?filter=CurrentValue.[status]=\"\u5f85\u751f\u6210\"&page_size=5",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "=Bearer {{ $env.LARK_ACCESS_TOKEN }}"
            }
          ]
        },
        "options": {
          "timeout": 15000
        }
      },
      "id": "query-source",
      "name": "Query Source (\u5f85\u751f\u6210)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        1100,
        0
      ],
      "onError": "continueRegularOutput",
      "notes": "\u67e5\u8be2\u5f85\u751f\u6210\u5185\u5bb9 (\u4f18\u5148\u7ea72)"
    },
    {
      "parameters": {
        "method": "GET",
        "url": "=https://open.feishu.cn/open-apis/bitable/v1/apps/{{ $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh' }}/tables/{{ $json.tables.topics }}/records?filter=CurrentValue.[status]=\"\u5f85\u63d0\u53d6\"&page_size=10",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "=Bearer {{ $env.LARK_ACCESS_TOKEN }}"
            }
          ]
        },
        "options": {
          "timeout": 15000
        }
      },
      "id": "query-topics",
      "name": "Query Topics (\u5f85\u63d0\u53d6)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        1100,
        150
      ],
      "onError": "continueRegularOutput",
      "notes": "\u67e5\u8be2\u5f85\u63d0\u53d6\u9009\u9898 (\u4f18\u5148\u7ea73)"
    },
    {
      "parameters": {
        "method": "GET",
        "url": "=https://open.feishu.cn/open-apis/bitable/v1/apps/{{ $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh' }}/tables/{{ $json.tables.keywords }}/records?filter=CurrentValue.[status]=\"\u5f85\u91c7\u96c6\"&page_size=5",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "=Bearer {{ $env.LARK_ACCESS_TOKEN }}"
            }
          ]
        },
        "options": {
          "timeout": 15000
        }
      },
      "id": "query-keywords",
      "name": "Query Keywords (\u5f85\u91c7\u96c6)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        1100,
        300
      ],
      "onError": "continueRegularOutput",
      "notes": "\u67e5\u8be2\u5f85\u91c7\u96c6\u5173\u952e\u8bcd (\u4f18\u5148\u7ea74)"
    },
    {
      "parameters": {
        "mode": "combine",
        "mergeByFields": {
          "values": []
        },
        "options": {}
      },
      "id": "merge-queries",
      "name": "Merge Query Results",
      "type": "n8n-nodes-base.merge",
      "typeVersion": 3.2,
      "position": [
        1350,
        100
      ],
      "notes": "\u5408\u5e764\u4e2a\u8868\u7684\u67e5\u8be2\u7ed3\u679c"
    },
    {
      "parameters": {
        "jsCode": "// \u83b7\u53d6\u6240\u6709\u67e5\u8be2\u7ed3\u679c\nconst items = $input.all();\nconst prepareData = items[0]?.json || {};\n\n// \u89e3\u6790\u67e5\u8be2\u7ed3\u679c\nconst queryResults = {\n  publish: items[1]?.json?.data?.items || [],\n  generation: items[2]?.json?.data?.items || [],\n  extraction: items[3]?.json?.data?.items || [],\n  discovery: items[4]?.json?.data?.items || []\n};\n\n// \u4f18\u5148\u7ea7\u987a\u5e8f\nconst PRIORITY_ORDER = ['publish', 'generation', 'extraction', 'discovery'];\n\n// \u5982\u679c\u662f\u5f3a\u5236\u6a21\u5f0f\uff0c\u76f4\u63a5\u4f7f\u7528\u6307\u5b9a\u7684 action\nif (prepareData.skip_query && prepareData.selected_action) {\n  return [{\n    action: prepareData.selected_action,\n    target_ids: prepareData.force_target_ids || [],\n    records: [],\n    force_mode: true,\n    timestamp: new Date().toISOString()\n  }];\n}\n\n// \u6309\u4f18\u5148\u7ea7\u9009\u62e9\u8981\u6267\u884c\u7684\u4efb\u52a1\nlet selectedAction = 'none';\nlet selectedRecords = [];\n\nfor (const action of PRIORITY_ORDER) {\n  const records = queryResults[action];\n  if (records && records.length > 0) {\n    selectedAction = action;\n    selectedRecords = records.map(r => ({\n      record_id: r.record_id,\n      fields: r.fields\n    }));\n    break;\n  }\n}\n\n// \u751f\u6210\u6267\u884c\u65e5\u5fd7\nconst summary = {\n  publish_count: queryResults.publish.length,\n  generation_count: queryResults.generation.length,\n  extraction_count: queryResults.extraction.length,\n  discovery_count: queryResults.discovery.length,\n  total_pending: Object.values(queryResults).reduce((sum, arr) => sum + arr.length, 0)\n};\n\nreturn [{\n  action: selectedAction,\n  records: selectedRecords,\n  record_count: selectedRecords.length,\n  summary: summary,\n  force_mode: false,\n  timestamp: new Date().toISOString()\n}];"
      },
      "id": "priority-dispatch",
      "name": "Priority Dispatch",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1550,
        100
      ],
      "notes": "\u4f18\u5148\u7ea7\u5206\u53d1\u903b\u8f91: Publish > Generation > Extraction > Discovery"
    },
    {
      "parameters": {
        "mode": "rules",
        "rules": {
          "values": [
            {
              "conditions": {
                "options": {
                  "caseSensitive": true,
                  "leftValue": "",
                  "typeValidation": "strict"
                },
                "conditions": [
                  {
                    "leftValue": "={{ $json.action }}",
                    "rightValue": "publish",
                    "operator": {
                      "type": "string",
                      "operation": "equals"
                    }
                  }
                ],
                "combinator": "and"
              },
              "renameOutput": true,
              "outputKey": "Publish"
            },
            {
              "conditions": {
                "options": {
                  "caseSensitive": true,
                  "leftValue": "",
                  "typeValidation": "strict"
                },
                "conditions": [
                  {
                    "leftValue": "={{ $json.action }}",
                    "rightValue": "generation",
                    "operator": {
                      "type": "string",
                      "operation": "equals"
                    }
                  }
                ],
                "combinator": "and"
              },
              "renameOutput": true,
              "outputKey": "Generation"
            },
            {
              "conditions": {
                "options": {
                  "caseSensitive": true,
                  "leftValue": "",
                  "typeValidation": "strict"
                },
                "conditions": [
                  {
                    "leftValue": "={{ $json.action }}",
                    "rightValue": "extraction",
                    "operator": {
                      "type": "string",
                      "operation": "equals"
                    }
                  }
                ],
                "combinator": "and"
              },
              "renameOutput": true,
              "outputKey": "Extraction"
            },
            {
              "conditions": {
                "options": {
                  "caseSensitive": true,
                  "leftValue": "",
                  "typeValidation": "strict"
                },
                "conditions": [
                  {
                    "leftValue": "={{ $json.action }}",
                    "rightValue": "discovery",
                    "operator": {
                      "type": "string",
                      "operation": "equals"
                    }
                  }
                ],
                "combinator": "and"
              },
              "renameOutput": true,
              "outputKey": "Discovery"
            }
          ]
        },
        "options": {
          "fallbackOutput": "extra",
          "renameFallbackOutput": "No Task"
        }
      },
      "id": "switch-router",
      "name": "Switch Router",
      "type": "n8n-nodes-base.switch",
      "typeVersion": 3.4,
      "position": [
        1750,
        100
      ],
      "notes": "\u6839\u636e action \u8def\u7531\u5230\u5bf9\u5e94\u7684\u5b50\u5de5\u4f5c\u6d41"
    },
    {
      "parameters": {
        "workflowId": "={{ $env.WF_PUBLISH_ID || 'WF-Publish' }}",
        "options": {}
      },
      "id": "execute-publish",
      "name": "Execute WF-Publish",
      "type": "n8n-nodes-base.executeWorkflow",
      "typeVersion": 1.3,
      "position": [
        2000,
        -100
      ],
      "notes": "\u6267\u884c\u53d1\u5e03\u5b50\u5de5\u4f5c\u6d41"
    },
    {
      "parameters": {
        "workflowId": "={{ $env.WF_GENERATION_ID || 'WF-Generation' }}",
        "options": {}
      },
      "id": "execute-generation",
      "name": "Execute WF-Generation",
      "type": "n8n-nodes-base.executeWorkflow",
      "typeVersion": 1.3,
      "position": [
        2000,
        50
      ],
      "notes": "\u6267\u884c AI \u5185\u5bb9\u751f\u6210\u5b50\u5de5\u4f5c\u6d41"
    },
    {
      "parameters": {
        "workflowId": "={{ $env.WF_EXTRACTION_ID || 'WF-Extraction' }}",
        "options": {}
      },
      "id": "execute-extraction",
      "name": "Execute WF-Extraction",
      "type": "n8n-nodes-base.executeWorkflow",
      "typeVersion": 1.3,
      "position": [
        2000,
        200
      ],
      "notes": "\u6267\u884c\u5185\u5bb9\u63d0\u53d6\u5b50\u5de5\u4f5c\u6d41"
    },
    {
      "parameters": {
        "workflowId": "={{ $env.WF_DISCOVERY_ID || 'WF-Discovery' }}",
        "options": {}
      },
      "id": "execute-discovery",
      "name": "Execute WF-Discovery",
      "type": "n8n-nodes-base.executeWorkflow",
      "typeVersion": 1.3,
      "position": [
        2000,
        350
      ],
      "notes": "\u6267\u884c\u5185\u5bb9\u53d1\u73b0\u5b50\u5de5\u4f5c\u6d41"
    },
    {
      "parameters": {
        "jsCode": "// \u65e0\u5f85\u5904\u7406\u4efb\u52a1\u65f6\u7684\u65e5\u5fd7\nconst input = $input.first().json;\n\nreturn [{\n  status: 'idle',\n  message: 'No pending tasks',\n  summary: input.summary || {},\n  timestamp: new Date().toISOString()\n}];"
      },
      "id": "no-task-handler",
      "name": "No Task Handler",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        2000,
        500
      ],
      "notes": "\u65e0\u5f85\u5904\u7406\u4efb\u52a1\u65f6\u7684\u5904\u7406"
    },
    {
      "parameters": {
        "mode": "chooseBranch",
        "output": "empty"
      },
      "id": "merge-results",
      "name": "Merge Results",
      "type": "n8n-nodes-base.merge",
      "typeVersion": 3.2,
      "position": [
        2250,
        200
      ],
      "notes": "\u5408\u5e76\u6240\u6709\u5b50\u5de5\u4f5c\u6d41\u6267\u884c\u7ed3\u679c"
    },
    {
      "parameters": {
        "jsCode": "// \u751f\u6210\u6267\u884c\u65e5\u5fd7\nconst results = $input.all();\nconst firstResult = results[0]?.json || {};\n\nconst logEntry = {\n  workflow_id: $workflow.id,\n  workflow_name: 'WF-Main',\n  execution_id: $execution.id,\n  timestamp: new Date().toISOString(),\n  action: firstResult.action || 'unknown',\n  status: firstResult.status || 'completed',\n  record_count: firstResult.record_count || 0,\n  summary: firstResult.summary || {},\n  duration_ms: Date.now() - new Date($execution.startedAt).getTime(),\n  error: firstResult.error || null\n};\n\nreturn [logEntry];"
      },
      "id": "prepare-log",
      "name": "Prepare Log Entry",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        2450,
        200
      ],
      "notes": "\u51c6\u5907\u6267\u884c\u65e5\u5fd7\u6570\u636e"
    },
    {
      "parameters": {
        "method": "POST",
        "url": "=https://open.feishu.cn/open-apis/bitable/v1/apps/{{ $env.LARK_APP_TOKEN || 'Gq93bAlZ7aSSclsLKdTcYCO2nwh' }}/tables/{{ $env.LOGS_TABLE_ID || 'tbl8xTUEtAQjxP4k' }}/records",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "=Bearer {{ $env.LARK_ACCESS_TOKEN }}"
            },
            {
              "name": "Content-Type",
              "value": "application/json"
            }
          ]
        },
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={\n  \"fields\": {\n    \"workflow_name\": \"{{ $json.workflow_name }}\",\n    \"execution_id\": \"{{ $json.execution_id }}\",\n    \"action\": \"{{ $json.action }}\",\n    \"status\": \"{{ $json.status }}\",\n    \"record_count\": {{ $json.record_count }},\n    \"duration_ms\": {{ $json.duration_ms }},\n    \"timestamp\": \"{{ $json.timestamp }}\",\n    \"summary\": \"{{ JSON.stringify($json.summary) }}\"\n  }\n}",
        "options": {
          "timeout": 10000
        }
      },
      "id": "log-to-feishu",
      "name": "Log to Feishu",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        2650,
        200
      ],
      "onError": "continueRegularOutput",
      "notes": "\u5c06\u6267\u884c\u65e5\u5fd7\u5199\u5165\u98de\u4e66 ExecutionLogs \u8868"
    }
  ],
  "connections": {
    "Schedule Trigger": {
      "main": [
        [
          {
            "node": "Merge Triggers",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Webhook Trigger": {
      "main": [
        [
          {
            "node": "Merge Triggers",
            "type": "main",
            "index": 1
          }
        ]
      ]
    },
    "Merge Triggers": {
      "main": [
        [
          {
            "node": "Health Check",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Health Check": {
      "main": [
        [
          {
            "node": "IF Crawler Ready",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF Crawler Ready": {
      "main": [
        [
          {
            "node": "Prepare Queries",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Trigger Cookie Refresh",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Trigger Cookie Refresh": {
      "main": [
        [
          {
            "node": "Prepare Log Entry",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prepare Queries": {
      "main": [
        [
          {
            "node": "Query Content (\u5f85\u53d1\u5e03)",
            "type": "main",
            "index": 0
          },
          {
            "node": "Query Source (\u5f85\u751f\u6210)",
            "type": "main",
            "index": 0
          },
          {
            "node": "Query Topics (\u5f85\u63d0\u53d6)",
            "type": "main",
            "index": 0
          },
          {
            "node": "Query Keywords (\u5f85\u91c7\u96c6)",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Query Content (\u5f85\u53d1\u5e03)": {
      "main": [
        [
          {
            "node": "Merge Query Results",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Query Source (\u5f85\u751f\u6210)": {
      "main": [
        [
          {
            "node": "Merge Query Results",
            "type": "main",
            "index": 1
          }
        ]
      ]
    },
    "Query Topics (\u5f85\u63d0\u53d6)": {
      "main": [
        [
          {
            "node": "Merge Query Results",
            "type": "main",
            "index": 2
          }
        ]
      ]
    },
    "Query Keywords (\u5f85\u91c7\u96c6)": {
      "main": [
        [
          {
            "node": "Merge Query Results",
            "type": "main",
            "index": 3
          }
        ]
      ]
    },
    "Merge Query Results": {
      "main": [
        [
          {
            "node": "Priority Dispatch",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Priority Dispatch": {
      "main": [
        [
          {
            "node": "Switch Router",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Switch Router": {
      "main": [
        [
          {
            "node": "Execute WF-Publish",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Execute WF-Generation",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Execute WF-Extraction",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Execute WF-Discovery",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "No Task Handler",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Execute WF-Publish": {
      "main": [
        [
          {
            "node": "Merge Results",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Execute WF-Generation": {
      "main": [
        [
          {
            "node": "Merge Results",
            "type": "main",
            "index": 1
          }
        ]
      ]
    },
    "Execute WF-Extraction": {
      "main": [
        [
          {
            "node": "Merge Results",
            "type": "main",
            "index": 2
          }
        ]
      ]
    },
    "Execute WF-Discovery": {
      "main": [
        [
          {
            "node": "Merge Results",
            "type": "main",
            "index": 3
          }
        ]
      ]
    },
    "No Task Handler": {
      "main": [
        [
          {
            "node": "Merge Results",
            "type": "main",
            "index": 4
          }
        ]
      ]
    },
    "Merge Results": {
      "main": [
        [
          {
            "node": "Prepare Log Entry",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prepare Log Entry": {
      "main": [
        [
          {
            "node": "Log to Feishu",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "settings": {
    "executionOrder": "v1",
    "saveManualExecutions": true,
    "callerPolicy": "workflowsFromSameOwner",
    "errorWorkflow": ""
  },
  "staticData": null,
  "tags": [
    {
      "name": "XHS-Pipeline"
    },
    {
      "name": "Main-Controller"
    }
  ],
  "triggerCount": 2,
  "versionId": "1.0.0"
}
Pro

For the full experience including quality scoring and batch install features for each workflow upgrade to Pro

About this workflow

WF-Main - XHS Pipeline Controller. Uses httpRequest. Scheduled trigger; 22 nodes.

Source: https://github.com/wade56754/XHS/blob/b5b999f35721bcfe8ffc6b518245e1d8dbaab5c5/workflows/WF-Main.json — original creator credit. Request a take-down →

More Web Scraping workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.

Web Scraping

As n8n instances scale, teams often lose track of sub-workflows—who uses them, where they are referenced, and whether they can be safely updated. This leads to inefficiencies like unnecessary copies o

HTTP Request, n8n, N8N Trigger +1
Web Scraping

This workflow is an improvement of this workflow by Greg Brzezinka.

HTTP Request, Email Send, XML +1
Web Scraping

N8N-Workflow-Github-Manager. Uses github, httpRequest, n8n. Scheduled trigger; 38 nodes.

GitHub, HTTP Request, n8n
Web Scraping

This workflow uses KlickTipp community nodes, available for self-hosted n8n instances only.

N8N Nodes Klicktipp, Salesforce, Salesforce Trigger +1
Web Scraping

This workflow acts as an automated engagement bot. It sends a Direct Message (DM) with a link or resource to any follower who replies to your post with a specific target keyword.

HTTP Request