[fc Sub] Scrape URL with 24h Cache

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

{
  "name": "[FC Sub] Scrape URL with 24h Cache",
  "settings": {
    "executionOrder": "v1"
  },
  "nodes": [
    {
      "parameters": {
        "content": "## [FC Sub] Scrape URL with 24h Cache\n**Purpose:** Scrapes a single URL via Firecrawl `/v1/scrape`, caches the result in Postgres for 24h, logs credit burn to the ledger, and returns a compact response.\n\n**Called by:** main agent's `scrape_url` tool.\n\n**Inputs:** `url` (required), `formats[]?` (default `['markdown','links']`), `only_main_content?` (default true), `include_screenshot?` (default false), `force_refresh?` (default false), `session_id?`.\n\n**Flow:**\n1. **Trigger** \u2014 receives scrape params from parent agent.\n2. **Prep + Hash URL** \u2014 validates URL, computes SHA-256 hash, normalizes inputs.\n3. **Cache Lookup** \u2014 Postgres SELECT on `firecrawl_scrape_cache` where `scraped_at > NOW() - 24h`.\n4. **IF Cache Hit?** \u2014 if hit AND not `force_refresh`, short-circuit with `cache_hit: true, credits_used: 0`.\n5. **Allowlist Guard** \u2014 if `firecrawl_allowed_domains` has rows, block URLs not in the list.\n6. **Call Firecrawl** \u2014 POST to `https://api.firecrawl.dev/v1/scrape` with the Firecrawl API credential.\n7. **Shape Response** \u2014 extract markdown/title/links, cap markdown at 50K chars.\n8. **Upsert Cache** \u2014 INSERT into cache, overwriting older rows for the same URL.\n9. **Log Ledger** \u2014 INSERT one row into `firecrawl_credit_ledger` with `credits_used: 1` (Firecrawl's per-scrape cost).\n10. **Return** \u2014 `{ url, markdown, title, links, credits_used, cache_hit, scraped_at }`.\n\n**Credentials required:** Firecrawl API (HTTP Header Auth, `Authorization: Bearer fc-...`), Postgres RW.",
        "height": 620,
        "width": 700,
        "color": 6
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -40,
        -660
      ],
      "id": "sticky-fc-scrape",
      "name": "README"
    },
    {
      "parameters": {
        "inputSource": "passthrough"
      },
      "type": "n8n-nodes-base.executeWorkflowTrigger",
      "typeVersion": 1.1,
      "position": [
        0,
        0
      ],
      "id": "fc-scrape-trigger",
      "name": "When Executed by Another Workflow"
    },
    {
      "parameters": {
        "jsCode": "const crypto = require('crypto');\nconst input = $input.first().json;\nconst q = input.query || {};\nconst pick = (k, def) => (input[k] !== undefined ? input[k] : (q[k] !== undefined ? q[k] : def));\n\nlet url = (pick('url') || '').toString().trim();\nif (!url) throw new Error('scrape_url requires a `url` parameter.');\nif (!/^https?:\\/\\//i.test(url)) url = 'https://' + url;\ntry { new URL(url); } catch { throw new Error(`Invalid URL: ${url}`); }\n\nconst parseIfString = (v, fallback) => {\n  if (v === undefined || v === null) return fallback;\n  if (Array.isArray(v)) return v;\n  if (typeof v !== 'string') return v;\n  try { return JSON.parse(v); } catch { return fallback; }\n};\n\nconst formats = parseIfString(pick('formats'), ['markdown','links']);\nconst only_main_content = pick('only_main_content', true) !== false;\nconst include_screenshot = pick('include_screenshot', false) === true;\nconst force_refresh = pick('force_refresh', false) === true;\nconst session_id = (pick('session_id') || $execution.id || 'no-session').toString();\n\nconst url_hash = crypto.createHash('sha256').update(url.toLowerCase()).digest('hex');\n\nconst effective_formats = include_screenshot\n  ? Array.from(new Set([...formats, 'screenshot']))\n  : formats;\n\nreturn [{ json: {\n  url,\n  url_hash,\n  formats: effective_formats,\n  only_main_content,\n  include_screenshot,\n  force_refresh,\n  session_id,\n  execution_id: $execution.id\n} }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        220,
        0
      ],
      "id": "fc-scrape-prep",
      "name": "Prep + Hash URL"
    },
    {
      "parameters": {
        "operation": "executeQuery",
        "query": "SELECT url, title, markdown, links, metadata, scraped_at\nFROM public.firecrawl_scrape_cache\nWHERE url_hash = $1\n  AND scraped_at > NOW() - INTERVAL '24 hours'\nLIMIT 1;",
        "options": {
          "queryReplacement": "={{ $json.url_hash }}"
        }
      },
      "type": "n8n-nodes-base.postgres",
      "typeVersion": 2.6,
      "position": [
        440,
        0
      ],
      "id": "fc-scrape-cache-lookup",
      "name": "Cache Lookup"
    },
    {
      "parameters": {
        "jsCode": "const prep = $('Prep + Hash URL').first().json;\nconst hits = $input.all().map(i => i.json).filter(r => r && r.url);\nconst hit = hits.length > 0 && !prep.force_refresh ? hits[0] : null;\nreturn [{ json: { ...prep, cache_hit: !!hit, cached: hit || null } }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        660,
        0
      ],
      "id": "fc-scrape-merge-cache",
      "name": "Merge Cache Result"
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict"
          },
          "conditions": [
            {
              "id": "cache-hit-condition",
              "leftValue": "={{ $json.cache_hit }}",
              "rightValue": true,
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        880,
        0
      ],
      "id": "fc-scrape-if-cache",
      "name": "IF Cache Hit"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "a1",
              "name": "url",
              "value": "={{ $json.cached.url }}",
              "type": "string"
            },
            {
              "id": "a2",
              "name": "title",
              "value": "={{ $json.cached.title }}",
              "type": "string"
            },
            {
              "id": "a3",
              "name": "markdown",
              "value": "={{ $json.cached.markdown }}",
              "type": "string"
            },
            {
              "id": "a4",
              "name": "links",
              "value": "={{ $json.cached.links }}",
              "type": "array"
            },
            {
              "id": "a5",
              "name": "scraped_at",
              "value": "={{ $json.cached.scraped_at }}",
              "type": "string"
            },
            {
              "id": "a6",
              "name": "cache_hit",
              "value": "={{ true }}",
              "type": "boolean"
            },
            {
              "id": "a7",
              "name": "credits_used",
              "value": "={{ 0 }}",
              "type": "number"
            },
            {
              "id": "a8",
              "name": "note",
              "value": "Served from 24h cache \u2014 no Firecrawl credit consumed. Pass force_refresh=true to re-fetch.",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1100,
        -160
      ],
      "id": "fc-scrape-return-cached",
      "name": "Return Cached"
    },
    {
      "parameters": {
        "operation": "executeQuery",
        "query": "SELECT COUNT(*)::int AS total,\n       (SELECT COUNT(*)::int FROM public.firecrawl_allowed_domains\n          WHERE $1 ILIKE '%' || domain) AS allowed_count\nFROM public.firecrawl_allowed_domains;",
        "options": {
          "queryReplacement": "={{ new URL($json.url).hostname }}"
        }
      },
      "type": "n8n-nodes-base.postgres",
      "typeVersion": 2.6,
      "position": [
        1100,
        160
      ],
      "id": "fc-scrape-allowlist-check",
      "name": "Check Allowlist"
    },
    {
      "parameters": {
        "jsCode": "const prep = $('Merge Cache Result').first().json;\nconst row = $input.first().json;\nconst total = Number(row.total || 0);\nconst allowed = Number(row.allowed_count || 0);\nif (total > 0 && allowed === 0) {\n  const hostname = new URL(prep.url).hostname;\n  throw new Error(`Domain '${hostname}' is not in firecrawl_allowed_domains. Add it via SQL (INSERT INTO firecrawl_allowed_domains) or leave the table empty to allow all domains.`);\n}\nreturn [{ json: prep }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1320,
        160
      ],
      "id": "fc-scrape-allowlist-guard",
      "name": "Allowlist Guard"
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://api.firecrawl.dev/v1/scrape",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={\n  \"url\": {{ JSON.stringify($json.url) }},\n  \"formats\": {{ JSON.stringify($json.formats) }},\n  \"onlyMainContent\": {{ $json.only_main_content }}\n}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "json"
            }
          },
          "timeout": 120000
        }
      },
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        1540,
        160
      ],
      "id": "fc-scrape-http",
      "name": "Call Firecrawl Scrape"
    },
    {
      "parameters": {
        "jsCode": "const prep = $('Merge Cache Result').first().json;\nconst resp = $input.first().json;\nconst data = resp.data || resp;\nconst MAX_MD = 50000;\nconst markdown = (data.markdown || '').toString();\nconst truncated = markdown.length > MAX_MD;\nconst links = Array.isArray(data.links) ? data.links.slice(0, 200) : [];\nconst title = (data.metadata && data.metadata.title) || data.title || '';\nconst description = (data.metadata && data.metadata.description) || '';\nconst screenshot = data.screenshot || null;\n\nreturn [{ json: {\n  url: prep.url,\n  url_hash: prep.url_hash,\n  session_id: prep.session_id,\n  execution_id: prep.execution_id,\n  title,\n  description,\n  markdown: truncated ? markdown.slice(0, MAX_MD) + '\\n\\n[...truncated at 50,000 chars]' : markdown,\n  links,\n  screenshot,\n  credits_used: 1,\n  cache_hit: false,\n  scraped_at: new Date().toISOString(),\n  metadata: data.metadata || null\n} }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1760,
        160
      ],
      "id": "fc-scrape-shape",
      "name": "Shape Response"
    },
    {
      "parameters": {
        "operation": "executeQuery",
        "query": "INSERT INTO public.firecrawl_scrape_cache (url_hash, url, title, markdown, links, metadata, scraped_at)\nVALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, NOW())\nON CONFLICT (url_hash) DO UPDATE\n  SET url        = EXCLUDED.url,\n      title      = EXCLUDED.title,\n      markdown   = EXCLUDED.markdown,\n      links      = EXCLUDED.links,\n      metadata   = EXCLUDED.metadata,\n      scraped_at = NOW();",
        "options": {
          "queryReplacement": "={{ $json.url_hash }}, {{ $json.url }}, {{ $json.title }}, {{ $json.markdown }}, {{ JSON.stringify($json.links) }}, {{ JSON.stringify($json.metadata) }}"
        }
      },
      "type": "n8n-nodes-base.postgres",
      "typeVersion": 2.6,
      "position": [
        1980,
        160
      ],
      "id": "fc-scrape-cache-write",
      "name": "Upsert Cache"
    },
    {
      "parameters": {
        "operation": "executeQuery",
        "query": "INSERT INTO public.firecrawl_credit_ledger (session_id, execution_id, operation, credits_used, url, status, metadata)\nVALUES ($1, $2, 'scrape', $3, $4, 'ok', $5::jsonb);",
        "options": {
          "queryReplacement": "={{ $('Shape Response').first().json.session_id }}, {{ $('Shape Response').first().json.execution_id }}, {{ $('Shape Response').first().json.credits_used }}, {{ $('Shape Response').first().json.url }}, {{ JSON.stringify({ cache_hit: false, title: $('Shape Response').first().json.title }) }}"
        }
      },
      "type": "n8n-nodes-base.postgres",
      "typeVersion": 2.6,
      "position": [
        2200,
        160
      ],
      "id": "fc-scrape-ledger",
      "name": "Log Credit Ledger"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "b1",
              "name": "url",
              "value": "={{ $('Shape Response').first().json.url }}",
              "type": "string"
            },
            {
              "id": "b2",
              "name": "title",
              "value": "={{ $('Shape Response').first().json.title }}",
              "type": "string"
            },
            {
              "id": "b3",
              "name": "markdown",
              "value": "={{ $('Shape Response').first().json.markdown }}",
              "type": "string"
            },
            {
              "id": "b4",
              "name": "links",
              "value": "={{ $('Shape Response').first().json.links }}",
              "type": "array"
            },
            {
              "id": "b5",
              "name": "scraped_at",
              "value": "={{ $('Shape Response').first().json.scraped_at }}",
              "type": "string"
            },
            {
              "id": "b6",
              "name": "cache_hit",
              "value": "={{ false }}",
              "type": "boolean"
            },
            {
              "id": "b7",
              "name": "credits_used",
              "value": "={{ $('Shape Response').first().json.credits_used }}",
              "type": "number"
            },
            {
              "id": "b8",
              "name": "screenshot",
              "value": "={{ $('Shape Response').first().json.screenshot }}",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        2420,
        160
      ],
      "id": "fc-scrape-return",
      "name": "Return Scraped"
    }
  ],
  "connections": {
    "When Executed by Another Workflow": {
      "main": [
        [
          {
            "node": "Prep + Hash URL",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prep + Hash URL": {
      "main": [
        [
          {
            "node": "Cache Lookup",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Cache Lookup": {
      "main": [
        [
          {
            "node": "Merge Cache Result",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Merge Cache Result": {
      "main": [
        [
          {
            "node": "IF Cache Hit",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF Cache Hit": {
      "main": [
        [
          {
            "node": "Return Cached",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Check Allowlist",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Check Allowlist": {
      "main": [
        [
          {
            "node": "Allowlist Guard",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Allowlist Guard": {
      "main": [
        [
          {
            "node": "Call Firecrawl Scrape",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Call Firecrawl Scrape": {
      "main": [
        [
          {
            "node": "Shape Response",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Shape Response": {
      "main": [
        [
          {
            "node": "Upsert Cache",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Upsert Cache": {
      "main": [
        [
          {
            "node": "Log Credit Ledger",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Log Credit Ledger": {
      "main": [
        [
          {
            "node": "Return Scraped",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

About this workflow

[FC Sub] Scrape URL with 24h Cache. Uses executeWorkflowTrigger, postgres, httpRequest. Event-driven trigger; 14 nodes.

Source: https://github.com/MinaSaad1/n8n-firecrawl-web-crawler-agent/blob/main/workflows/03-sub-scrape-cached.json — original creator credit. Request a take-down →

More Web Scraping workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.