AutomationFlowsWeb Scraping › Generate Llms.txt From a Website Sitemap Using Forms and Http Requests

Generate Llms.txt From a Website Sitemap Using Forms and Http Requests

ByisaWOW @isawow on n8n.io

This workflow uses an n8n Form and HTTP requests to read a website sitemap, fetch each listed page, extract the page title and meta description, and generate an llms.txt file that you can preview, copy, or download. Receives a form submission with the site name, site…

Event trigger★★★★☆ complexity13 nodesForm TriggerHTTP RequestForm
Web Scraping Trigger: Event Nodes: 13 Complexity: ★★★★☆ Added:

This workflow corresponds to n8n.io template #16972 — we link there as the canonical source.

This workflow follows the Form → Form Trigger recipe pattern — see all workflows that pair these two integrations.

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

Download .json
{
  "id": "hc3pIv1ODZdg8iwV",
  "name": "Auto llms.txt Generator from Sitemap - Form + Page Titles + Copy or Download",
  "tags": [],
  "nodes": [
    {
      "id": "c955f268-604d-4a0a-bab5-1dd3d5367223",
      "name": "Sticky Note",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -896,
        -208
      ],
      "parameters": {
        "width": 432,
        "height": 1032,
        "content": "## Auto llms.txt Generator from Sitemap\n\nGive AI answer engines a clean map of your website. Fill a short form with your sitemap URL, and this workflow reads the live sitemap, visits each page, pulls the real title and meta description, and builds a ready to host llms.txt file. The result page lets you preview, copy, or download it. No crawler service, no CSV upload, no AI credits, and no credentials needed.\n\n### How it works\n1. **On Form Submission** collects the site name, description, sitemap URL and a page cap.\n2. The workflow fetches your sitemap and expands a sitemap index into its child sitemaps when needed.\n3. Page URLs are collected, deduplicated and grouped into sections based on the first URL folder.\n4. Each page is fetched once and its title and meta description are read from the HTML.\n5. A markdown llms.txt file is built and shown on a styled result page with Copy and Download buttons.\n\n### Setup steps\n1. No credentials are needed. Open node 1 and copy the form URL, or use the Test URL inside the editor.\n2. Open the form, enter your site name, description, sitemap URL and page cap, then submit.\n3. On the result page press Copy or Download to save the file.\n4. Upload llms.txt to your site root at yourdomain.com/llms.txt.\n\n### Customization\nRaise Max Pages for larger sites, change the section grouping rule in node 6, or edit the result page styling in node 8 to match your brand."
      },
      "typeVersion": 1
    },
    {
      "id": "eaf3d167-32b7-4603-9db9-5273ff442570",
      "name": "Sticky Note1",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -416,
        144
      ],
      "parameters": {
        "color": 7,
        "width": 512,
        "height": 360,
        "content": "## 1. Form and config\nCollects the site name, description, sitemap URL and page cap, then normalizes them for the rest of the flow."
      },
      "typeVersion": 1
    },
    {
      "id": "95d7c250-0a37-454f-8fd1-405c3ae555bc",
      "name": "Sticky Note2",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        128,
        144
      ],
      "parameters": {
        "color": 7,
        "width": 710,
        "height": 360,
        "content": "## 2. Read sitemaps\nFetches the sitemap, expands a sitemap index when present, and collects clean deduplicated page URLs."
      },
      "typeVersion": 1
    },
    {
      "id": "ba496578-1158-4e7c-9a88-21b46ca86b64",
      "name": "Sticky Note3",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        880,
        144
      ],
      "parameters": {
        "color": 7,
        "width": 924,
        "height": 360,
        "content": "## 3. Build and show\nReads each page title and description, writes the llms.txt file, and shows a copy or download result page."
      },
      "typeVersion": 1
    },
    {
      "id": "a8220c89-0199-4d80-994c-437586756fcd",
      "name": "1. On Form Submission",
      "type": "n8n-nodes-base.formTrigger",
      "position": [
        -336,
        320
      ],
      "parameters": {
        "options": {},
        "formTitle": "llms.txt Generator",
        "formFields": {
          "values": [
            {
              "fieldLabel": "Site Name",
              "placeholder": "My Website",
              "requiredField": true
            },
            {
              "fieldLabel": "Site Description",
              "placeholder": "One line about the site"
            },
            {
              "fieldLabel": "Sitemap URL",
              "placeholder": "https://example.com/sitemap.xml",
              "requiredField": true
            },
            {
              "fieldType": "number",
              "fieldLabel": "Max Pages",
              "placeholder": "50"
            }
          ]
        },
        "formDescription": "Enter your site details and get a ready to host llms.txt file."
      },
      "typeVersion": 2.2
    },
    {
      "id": "d5231a4c-dcc6-479c-a44f-c86867131b37",
      "name": "2. Set Config",
      "type": "n8n-nodes-base.set",
      "position": [
        -112,
        320
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "23145ef5-4e62-4af8-abc5-8e47c1a2f3b0",
              "name": "siteName",
              "type": "string",
              "value": "={{ $json['Site Name'] }}"
            },
            {
              "id": "90858002-f2ad-4925-a938-d5a180fe063e",
              "name": "siteDescription",
              "type": "string",
              "value": "={{ $json['Site Description'] }}"
            },
            {
              "id": "d582fb73-0120-4163-9a10-476231b1b773",
              "name": "sitemapUrl",
              "type": "string",
              "value": "={{ $json['Sitemap URL'] }}"
            },
            {
              "id": "f176194e-5d3a-4224-98c0-0d4776b7b423",
              "name": "maxPages",
              "type": "number",
              "value": "={{ $json['Max Pages'] || 50 }}"
            }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "fe8f7bf1-c993-4eff-ac5c-cbb89b9c31a3",
      "name": "3. Fetch Sitemap",
      "type": "n8n-nodes-base.httpRequest",
      "position": [
        192,
        320
      ],
      "parameters": {
        "url": "={{ $json.sitemapUrl }}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "text"
            }
          }
        },
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (compatible; n8n-llms-txt-generator/1.0)"
            }
          ]
        }
      },
      "typeVersion": 4.2
    },
    {
      "id": "f9653f4b-bd02-4db8-a6c7-8df47aa6cdfe",
      "name": "4. Resolve Leaf Sitemaps",
      "type": "n8n-nodes-base.code",
      "position": [
        400,
        320
      ],
      "parameters": {
        "jsCode": "const xml = $json.data || '';\nconst primary = $('2. Set Config').first().json.sitemapUrl;\nconst isIndex = /<sitemapindex[\\s>]/i.test(xml);\nconst out = [];\nif (isIndex) {\n  const re = /<loc>\\s*([^<]+?)\\s*<\\/loc>/gi;\n  let m;\n  while ((m = re.exec(xml)) !== null) {\n    out.push({ json: { sitemapUrl: m[1].trim() } });\n  }\n}\nif (out.length === 0) {\n  out.push({ json: { sitemapUrl: primary } });\n}\nreturn out;"
      },
      "typeVersion": 2
    },
    {
      "id": "b464ebf6-54fd-4f7f-9b4c-2a735908c550",
      "name": "5. Fetch Leaf Sitemaps",
      "type": "n8n-nodes-base.httpRequest",
      "onError": "continueRegularOutput",
      "position": [
        624,
        320
      ],
      "parameters": {
        "url": "={{ $json.sitemapUrl }}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "text"
            }
          }
        },
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (compatible; n8n-llms-txt-generator/1.0)"
            }
          ]
        }
      },
      "typeVersion": 4.2,
      "alwaysOutputData": true
    },
    {
      "id": "3a43760e-719d-47ca-925e-d3fcc52449a0",
      "name": "6. Extract Page URLs",
      "type": "n8n-nodes-base.code",
      "position": [
        928,
        320
      ],
      "parameters": {
        "jsCode": "const cfg = $('2. Set Config').first().json;\nconst maxPages = Number(cfg.maxPages) || 50;\nconst all = $input.all();\nconst seen = new Set();\nconst out = [];\nfor (const it of all) {\n  const xml = (it.json && it.json.data) ? it.json.data : '';\n  const re = /<loc>\\s*([^<]+?)\\s*<\\/loc>/gi;\n  let m;\n  while ((m = re.exec(xml)) !== null) {\n    let u = m[1].trim();\n    if (!u) continue;\n    if (/\\.xml($|\\?)/i.test(u)) continue;\n    if (/sitemap/i.test(u)) continue;\n    if (seen.has(u)) continue;\n    seen.add(u);\n    let section = 'Pages';\n    try {\n      const parts = new URL(u).pathname.split('/').filter(Boolean);\n      if (parts.length > 1) section = parts[0];\n    } catch (e) {}\n    out.push({ json: { url: u, section } });\n    if (out.length >= maxPages) break;\n  }\n  if (out.length >= maxPages) break;\n}\nreturn out;"
      },
      "typeVersion": 2
    },
    {
      "id": "c922c29a-4ec9-4e70-91bd-7c536eaf50f3",
      "name": "7. Fetch Page HTML",
      "type": "n8n-nodes-base.httpRequest",
      "onError": "continueRegularOutput",
      "position": [
        1152,
        320
      ],
      "parameters": {
        "url": "={{ $json.url }}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "text"
            }
          }
        },
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (compatible; n8n-llms-txt-generator/1.0)"
            }
          ]
        }
      },
      "typeVersion": 4.2,
      "alwaysOutputData": true
    },
    {
      "id": "1dcc4bdb-664c-476d-82e6-fedfecf41550",
      "name": "8. Build llms.txt File",
      "type": "n8n-nodes-base.code",
      "position": [
        1360,
        320
      ],
      "parameters": {
        "jsCode": "const cfg = $('2. Set Config').first().json;\nconst siteName = cfg.siteName || 'My Website';\nconst siteDescription = cfg.siteDescription || '';\nconst pages = $('6. Extract Page URLs').all();\nconst htmls = $input.all();\n\nfunction decode(s) {\n  return String(s)\n    .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>')\n    .replace(/&quot;/g, '\"').replace(/&#39;/g, \"'\").replace(/&nbsp;/g, ' ')\n    .replace(/\\s+/g, ' ').trim();\n}\nfunction titleFromSlug(u) {\n  try {\n    const parts = new URL(u).pathname.split('/').filter(Boolean);\n    const slug = parts.length ? parts[parts.length - 1] : u;\n    const clean = slug.replace(/\\.[a-z]+$/i, '').replace(/[-_]+/g, ' ').trim();\n    if (!clean) return siteName + ' home';\n    return clean.replace(/\\b\\w/g, c => c.toUpperCase());\n  } catch (e) { return u; }\n}\n\nconst sections = {};\nconst order = [];\nfor (let i = 0; i < pages.length; i++) {\n  const url = pages[i].json.url;\n  const section = pages[i].json.section || 'Pages';\n  const html = (htmls[i] && htmls[i].json && htmls[i].json.data) ? htmls[i].json.data : '';\n  let title = '';\n  let desc = '';\n  if (html) {\n    const t = html.match(/<title[^>]*>([\\s\\S]*?)<\\/title>/i);\n    if (t) title = decode(t[1]);\n    let d = html.match(/<meta[^>]+name=[\"']description[\"'][^>]*content=[\"']([\\s\\S]*?)[\"']/i);\n    if (!d) d = html.match(/<meta[^>]+content=[\"']([\\s\\S]*?)[\"'][^>]+name=[\"']description[\"']/i);\n    if (d) desc = decode(d[1]);\n  }\n  if (!title) title = titleFromSlug(url);\n  if (!sections[section]) { sections[section] = []; order.push(section); }\n  let line = '- [' + title + '](' + url + ')';\n  if (desc) line += ': ' + desc;\n  sections[section].push(line);\n}\n\nlet out = '# ' + siteName + '\\n';\nif (siteDescription) out += '\\n> ' + siteDescription + '\\n';\nout += '\\n';\nconst names = order.sort();\nfor (const s of names) {\n  const heading = s.charAt(0).toUpperCase() + s.slice(1);\n  out += '## ' + heading + '\\n';\n  for (const line of sections[s]) out += line + '\\n';\n  out += '\\n';\n}\n\nconst esc = (s) => String(s).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');\nconst pageCount = pages.length;\n\nconst pageHtml = `\n<style>\n  .llmswrap { max-width: 760px; margin: 0 auto; text-align: left; font-family: system-ui, -apple-system, Roboto, Arial, sans-serif; color: #0f172a; }\n  .llmswrap .badge { display: inline-block; font-size: 12px; font-weight: 600; color: #4338ca; background: #eef2ff; border: 1px solid #e0e7ff; padding: 3px 10px; border-radius: 999px; margin-bottom: 12px; }\n  .llmswrap .title { font-size: 22px; font-weight: 700; margin: 0 0 4px; }\n  .llmswrap .sub { font-size: 14px; color: #64748b; margin: 0 0 16px; line-height: 1.5; }\n  .llmswrap textarea { width: 100%; box-sizing: border-box; min-height: 340px; resize: vertical; font-family: ui-monospace, Menlo, Consolas, monospace; font-size: 13px; line-height: 1.55; padding: 16px; color: #e6edf3; background: #0d1117; border: 1px solid #30363d; border-radius: 12px; outline: none; }\n  .llmswrap textarea:focus { border-color: #6366f1; box-shadow: 0 0 0 3px rgba(99,102,241,0.25); }\n  .llmswrap .row { display: flex; gap: 12px; margin-top: 16px; flex-wrap: wrap; }\n  .llmswrap .btn { flex: 1 1 auto; min-width: 160px; font-size: 15px; font-weight: 600; padding: 13px 18px; border-radius: 12px; border: 0; cursor: pointer; transition: transform .16s ease, box-shadow .16s ease, background .16s ease; }\n  .llmswrap .btn:hover { transform: translateY(-1px); }\n  .llmswrap .btn:active { transform: translateY(0) scale(.98); }\n  .llmswrap .primary { color: #fff; background: linear-gradient(135deg, #6366f1, #8b5cf6); box-shadow: 0 6px 18px rgba(99,102,241,0.35); }\n  .llmswrap .primary.ok { background: linear-gradient(135deg, #16a34a, #22c55e); animation: llmspop .32s ease; }\n  .llmswrap .ghost { color: #0f172a; background: #fff; border: 1px solid #cbd5e1; box-shadow: 0 2px 8px rgba(15,23,42,0.06); }\n  .llmswrap .ghost:hover { border-color: #6366f1; color: #4338ca; }\n  @keyframes llmspop { 0% { transform: scale(1); } 45% { transform: scale(1.06); } 100% { transform: scale(1); } }\n  @media (max-width: 520px) { .llmswrap .btn { flex-basis: 100%; } }\n</style>\n<div class='llmswrap'>\n  <div class='badge'>${pageCount} pages mapped</div>\n  <div class='title'>Your llms.txt is ready</div>\n  <div class='sub'>Review it below, then copy or download. Upload the file to your site root at yourdomain.com/llms.txt</div>\n  <textarea id='llmsOut' readonly spellcheck='false'>${esc(out)}</textarea>\n  <div class='row'>\n    <button id='llmsCopy' type='button' class='btn primary'>Copy to clipboard</button>\n    <button id='llmsDl' type='button' class='btn ghost'>Download llms.txt</button>\n  </div>\n</div>\n<script>\n  (function () {\n    var out = document.getElementById('llmsOut');\n    var copy = document.getElementById('llmsCopy');\n    var dl = document.getElementById('llmsDl');\n    var reset;\n    copy.addEventListener('click', function () {\n      var done = function () {\n        copy.classList.add('ok');\n        copy.textContent = 'Copied';\n        clearTimeout(reset);\n        reset = setTimeout(function () { copy.classList.remove('ok'); copy.textContent = 'Copy to clipboard'; }, 1600);\n      };\n      if (navigator.clipboard && navigator.clipboard.writeText) {\n        navigator.clipboard.writeText(out.value).then(done, function () { out.select(); document.execCommand('copy'); done(); });\n      } else {\n        out.select(); document.execCommand('copy'); done();\n      }\n    });\n    dl.addEventListener('click', function () {\n      var blob = new Blob([out.value], { type: 'text/plain;charset=utf-8' });\n      var a = document.createElement('a');\n      a.href = URL.createObjectURL(blob);\n      a.download = 'llms.txt';\n      document.body.appendChild(a);\n      a.click();\n      setTimeout(function () { URL.revokeObjectURL(a.href); a.remove(); }, 1000);\n    });\n  })();\n</script>`;\n\nreturn [{ json: { pageCount: pageCount, fileName: 'llms.txt', content: out, pageHtml: pageHtml } }];"
      },
      "typeVersion": 2
    },
    {
      "id": "5c7d86ff-cbf6-4e2b-a2cf-fb7cd86c9067",
      "name": "9. Show Result Page",
      "type": "n8n-nodes-base.form",
      "position": [
        1584,
        320
      ],
      "parameters": {
        "operation": "completion",
        "respondWith": "showText",
        "responseText": "={{ $json.pageHtml }}"
      },
      "typeVersion": 1
    }
  ],
  "active": false,
  "settings": {
    "binaryMode": "separate",
    "executionOrder": "v1"
  },
  "versionId": "389f2789-5163-474e-a40d-764c81ff89b6",
  "nodeGroups": [],
  "connections": {
    "2. Set Config": {
      "main": [
        [
          {
            "node": "3. Fetch Sitemap",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "3. Fetch Sitemap": {
      "main": [
        [
          {
            "node": "4. Resolve Leaf Sitemaps",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "7. Fetch Page HTML": {
      "main": [
        [
          {
            "node": "8. Build llms.txt File",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "6. Extract Page URLs": {
      "main": [
        [
          {
            "node": "7. Fetch Page HTML",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "1. On Form Submission": {
      "main": [
        [
          {
            "node": "2. Set Config",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "5. Fetch Leaf Sitemaps": {
      "main": [
        [
          {
            "node": "6. Extract Page URLs",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "8. Build llms.txt File": {
      "main": [
        [
          {
            "node": "9. Show Result Page",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "4. Resolve Leaf Sitemaps": {
      "main": [
        [
          {
            "node": "5. Fetch Leaf Sitemaps",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}
Pro

For the full experience including quality scoring and batch install features for each workflow upgrade to Pro

About this workflow

This workflow uses an n8n Form and HTTP requests to read a website sitemap, fetch each listed page, extract the page title and meta description, and generate an llms.txt file that you can preview, copy, or download. Receives a form submission with the site name, site…

Source: https://n8n.io/workflows/16972/ — original creator credit. Request a take-down →

More Web Scraping workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.

Web Scraping

This workflow allows you to import any workflow from a file or another n8n instance and map the credentials easily. A multi-form setup guides you through the entire process At the beginning you have t

Execute Command, Read Write File, HTTP Request +3
Web Scraping

LDX hub All Services Demo. Uses formTrigger, httpRequest, form, n8n-nodes-ldxhub. Event-driven trigger; 53 nodes.

Form Trigger, HTTP Request, Form +1
Web Scraping

This workflow provides a single n8n form that routes to five LDXhub services (StructFlow, RefineLoop, RenderOCR, CastDoc, and ExtractDoc) to run one-off document processing jobs and return the generat

Form Trigger, HTTP Request, Form +1
Web Scraping

N8n recently introduced folders and it has been a big improvement on workflow management on top of the tags.

HTTP Request, n8n, Form Trigger +1
Web Scraping

Git Commit. Uses github, n8n, formTrigger, httpRequest. Event-driven trigger; 34 nodes.

GitHub, n8n, Form Trigger +2