{
  "id": "r5wvaRaYTgvWjmmK",
  "meta": {
    "templateCredsSetupCompleted": true
  },
  "name": "Walmart Scraping with ScrapeOps",
  "tags": [
    {
      "id": "tOnkaXwaFBiEoFQm",
      "name": "price-tracking",
      "createdAt": "2025-11-06T11:10:23.912Z",
      "updatedAt": "2025-11-06T11:10:23.912Z"
    },
    {
      "id": "yh7XX4GObvd3np2S",
      "name": "google sheets",
      "createdAt": "2025-11-06T11:10:29.979Z",
      "updatedAt": "2025-11-06T11:10:29.979Z"
    },
    {
      "id": "lZKSh2IoxHklnOUw",
      "name": "ScrapeOps",
      "createdAt": "2025-10-20T20:27:13.410Z",
      "updatedAt": "2025-10-20T20:27:13.410Z"
    },
    {
      "id": "tDTf2TyCxgErkuae",
      "name": "walmart-scraping",
      "createdAt": "2025-10-20T20:27:13.542Z",
      "updatedAt": "2025-10-20T20:27:13.542Z"
    }
  ],
  "nodes": [
    {
      "id": "840b4f5f-5dec-45dc-88c7-9ccf5b7c42ff",
      "name": "Sticky Note1",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -176,
        176
      ],
      "parameters": {
        "width": 576,
        "height": 480,
        "content": "### \ud83d\uded2 Walmart Product Scraper\n\nThis workflow automatically scrapes product data from Walmart search results and saves it to Google Sheets for price tracking and analysis.\n\n### How it works\n1. **Schedule**: Runs automatically every 4 hours.\n2. **Search**: Defines a search term (default: \"Storage\").\n3. **Scrape**: Uses **ScrapeOps** to fetch and parse Walmart search results.\n4. **Process**: Cleans data and filters invalid items.\n5. **Save**: Appends to **Google Sheets** and notifies **Slack**.\n\n### Setup steps\n1. **ScrapeOps**: Add your free API key in the **[ScrapeOps](https://scrapeops.io/app/register/n8n)** node credentials.\n2. **Google Sheets**:\n   - Duplicate this [Template Sheet](https://docs.google.com/spreadsheets/d/1WbIzWaXjuQeTxrn_C7PjGMNNibUyLIgY2hdeWh3gOLA/edit?gid=0#gid=0).\n   - Connect your account and select the new sheet in the **Save to Google Sheets** node.\n3. **Slack**: Connect your account or remove the node."
      },
      "typeVersion": 1
    },
    {
      "id": "78771e7f-9449-476a-bcf0-3532b45a76d0",
      "name": "ScrapeOps Parser",
      "type": "@scrapeops/n8n-nodes-scrapeops.ScrapeOps",
      "position": [
        1136,
        424
      ],
      "parameters": {
        "apiType": "parserApi",
        "parserUrl": "=https://www.walmart.com/search?q= {{ $('Set Search Parameters').item.json.search_term }}",
        "parserHtml": "={{ $json }}",
        "parserDomain": "walmart"
      },
      "credentials": {
        "scrapeOpsApi": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 1
    },
    {
      "id": "87879598-0fdd-477a-aae8-3aa60efc47a3",
      "name": "ScrapeOps Proxy",
      "type": "@scrapeops/n8n-nodes-scrapeops.ScrapeOps",
      "position": [
        912,
        424
      ],
      "parameters": {
        "url": "=https://www.walmart.com/search?q= {{ $json.search_term }}",
        "advancedOptions": {}
      },
      "credentials": {
        "scrapeOpsApi": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 1
    },
    {
      "id": "c65d4eea-6fcc-40fa-b046-20813e314f9f",
      "name": "Add Metadata",
      "type": "n8n-nodes-base.set",
      "position": [
        1360,
        424
      ],
      "parameters": {
        "fields": {
          "values": [
            {
              "name": "=  timestamp",
              "stringValue": "={{ new Date().toISOString() }}"
            },
            {
              "name": "search_term",
              "stringValue": "={{ $('Set Search Parameters').item.json.search_term }}"
            },
            {
              "name": "=total_products_found",
              "stringValue": "={{ $json.data.searchMetadata.totalResults || 0}}"
            },
            {
              "name": "=products",
              "stringValue": "={{ $json.data.products }}"
            },
            {
              "name": "=url_scraped",
              "stringValue": "={{ $json.url }}"
            },
            {
              "name": "=data_coverage",
              "stringValue": "={{ $json.data_coverage_percentage }}"
            }
          ]
        },
        "options": {}
      },
      "typeVersion": 3.2
    },
    {
      "id": "9635f959-cf12-460f-870c-582220fbb991",
      "name": "Save to Google Sheets",
      "type": "n8n-nodes-base.googleSheets",
      "position": [
        2256,
        352
      ],
      "parameters": {
        "columns": {
          "value": {
            "saving": "={{\n  $json.price.match(/was[^\\d]*(\\d+\\.\\d+)/i)\n    ? Number(($json.price.match(/was[^\\d]*(\\d+\\.\\d+)/i) || [\"\",\"0\"])[1]) \n        - Number(($json.price.match(/(?:current price|now)[^\\d]*(\\d+\\.\\d+)/i) || [\"\",\"0\"])[1])\n    : 0\n}}\n",
            "Image URL": "={{ $json.image_url }}",
            "Product URL": "={{$json.product_url || $json.url }}",
            "Search Term": "={{ $('Add Metadata').item.json.search_term }}",
            "Is Sponsored": "={{ $json.is_sponsored }}",
            "Product Name": "={{ $json.product_name || $json.title }}",
            "Review Count": "={{ $json.review_count || $json.reviews}}",
            "Price Display": "=${{ Number(($json.price.match(/(?:current price|now)[^\\d]*(\\d+\\.\\d+)/i) || [\"\",\"0\"])[1]) }}",
            "Price Numeric": "={{ Number(($json.price.match(/(?:current price|now)[^\\d]*(\\d+\\.\\d+)/i) || [\"\",\"0\"])[1]) }}",
            "Popularity Level": "={{ parseInt($json.review_count ) > 1000 ? 'High' : (parseInt($json.review_count || 0) > 100 ? 'Medium' : (parseInt($json.review_count || 0) > 10 ? 'Low' : 'Very Low')) }}",
            "Scraped Timestamp": "={{ $json.scraped_at }}"
          },
          "schema": [
            {
              "id": "Product Name",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Product Name",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Price Display",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Price Display",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Price Numeric",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Price Numeric",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Review Count",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Review Count",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Popularity Level",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Popularity Level",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Is Sponsored",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Is Sponsored",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Product URL",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Product URL",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Image URL",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Image URL",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Search Term",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Search Term",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Scraped Timestamp",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Scraped Timestamp",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "saving",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "saving",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [],
          "attemptToConvertTypes": false,
          "convertFieldsToString": false
        },
        "options": {},
        "operation": "append",
        "sheetName": {
          "__rl": true,
          "mode": "list",
          "value": 460852058,
          "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1WbIzWaXjuQeTxrn_C7PjGMNNibUyLIgY2hdeWh3gOLA/edit#gid=460852058",
          "cachedResultName": "Sheet1"
        },
        "documentId": {
          "__rl": true,
          "mode": "url",
          "value": "https://docs.google.com/spreadsheets/d/1WbIzWaXjuQeTxrn_C7PjGMNNibUyLIgY2hdeWh3gOLA/edit?gid=0#gid=0"
        }
      },
      "credentials": {
        "googleSheetsOAuth2Api": {
          "name": "<your credential>"
        }
      },
      "typeVersion": 4.4
    },
    {
      "id": "338cf225-db7f-46dc-87bd-ed90b8931d0f",
      "name": "Sticky Note",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        416,
        296
      ],
      "parameters": {
        "color": 7,
        "width": 416,
        "height": 288,
        "content": "## 1. Setup & Config\nDefine your search keywords and schedule frequency."
      },
      "typeVersion": 1
    },
    {
      "id": "537940b2-b718-4aa1-8e0d-d10a039f5750",
      "name": "Sticky Note2",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        872,
        296
      ],
      "parameters": {
        "color": 7,
        "width": 400,
        "height": 288,
        "content": "## 2. Scraping Pipeline\n[Fetches Walmart](https://scrapeops.io/docs/n8n/overview/) search results and parses the HTML using ScrapeOps."
      },
      "typeVersion": 1
    },
    {
      "id": "5932945d-e0b8-42ba-a99e-e926d51a63e7",
      "name": "Sticky Note3",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        1320,
        296
      ],
      "parameters": {
        "color": 7,
        "width": 848,
        "height": 288,
        "content": "## 3. Data Processing\nStandardizes product data, splits arrays, and filters out invalid results."
      },
      "typeVersion": 1
    },
    {
      "id": "135308a1-f69a-471f-93f7-5445528925a8",
      "name": "Sticky Note4",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2184,
        152
      ],
      "parameters": {
        "color": 7,
        "width": 464,
        "height": 432,
        "content": "## 4. Storage & Alerts\nSaves valid data to Google Sheets and sends a summary to Slack."
      },
      "typeVersion": 1
    },
    {
      "id": "cf41f4bf-605b-48a7-98ff-b72e347cf988",
      "name": "Format Product Data",
      "type": "n8n-nodes-base.set",
      "position": [
        1808,
        424
      ],
      "parameters": {
        "fields": {
          "values": [
            {
              "name": "product_name",
              "stringValue": "={{ $json.name || $json.title || 'N/A' }}"
            },
            {
              "name": "price",
              "stringValue": "={{ $json.price || 'N/A' }}"
            },
            {
              "name": "image_url",
              "stringValue": "={{ $json.image?.url || $json.image || 'N/A' }}"
            },
            {
              "name": "product_url",
              "stringValue": "={{ $json.url || 'N/A' }}"
            },
            {
              "name": "review_count",
              "type": "numberValue",
              "numberValue": "={{ $json.rating?.reviewCount || $json.reviews || 0 }}"
            },
            {
              "name": "rating",
              "stringValue": "={{ $json.rating?.ratingValue || $json.rating || 'N/A' }}"
            },
            {
              "name": "saving",
              "stringValue": "={{ $json.saving || 'N/A' }}"
            },
            {
              "name": "is_sponsored",
              "stringValue": "={{ $json.isSponsored || $json.sponsored || false }}"
            },
            {
              "name": "=scraped_at",
              "stringValue": "={{ $('Set Search Parameters').item.json.Month }}/{{ $('Set Search Parameters').item.json[\"Day of month\"] }}/{{ $('Set Search Parameters').item.json.Year }}"
            },
            {
              "name": "search_term_used",
              "stringValue": "={{ $('Add Metadata').item.json.search_term }}"
            }
          ]
        },
        "options": {}
      },
      "typeVersion": 3.2
    },
    {
      "id": "ece9febe-78af-47d6-a7a5-04c2428ae9df",
      "name": "Split Products Array",
      "type": "n8n-nodes-base.code",
      "position": [
        1584,
        424
      ],
      "parameters": {
        "jsCode": "// Parse the products array and return individual product items\nconst inputData = $input.first().json;\n\n// Try multiple possible field names (with and without = prefix)\nconst productsArray = inputData['=products'] || inputData.products || inputData.data?.products || null;\n\n// Handle both string and array cases\nlet products;\nif (!productsArray) {\n  return [];\n}\n\nif (typeof productsArray === 'string') {\n  try {\n    products = JSON.parse(productsArray);\n  } catch (e) {\n    return [];\n  }\n} else {\n  products = productsArray;\n}\n\n// Ensure products is an array\nif (!Array.isArray(products)) {\n  return [];\n}\n\n// Filter out any null or undefined products\n// Note: Walmart products use 'name' field, not 'title'\nconst validProducts = products.filter(product => product && (product.name || product.title));\n\nreturn validProducts.map(product => ({ json: product }));"
      },
      "typeVersion": 2
    },
    {
      "id": "e791c0f1-5c60-48a7-9166-a52feae77a34",
      "name": "Set Search Parameters",
      "type": "n8n-nodes-base.set",
      "position": [
        688,
        424
      ],
      "parameters": {
        "fields": {
          "values": [
            {
              "name": "search_term",
              "stringValue": "Raspberry Pi"
            }
          ]
        },
        "options": {}
      },
      "typeVersion": 3.2
    },
    {
      "id": "433ab6ca-5e97-4fe4-8a9e-c4cc2e32be5f",
      "name": "Schedule Trigger",
      "type": "n8n-nodes-base.scheduleTrigger",
      "position": [
        464,
        424
      ],
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "hours",
              "hoursInterval": 4
            }
          ]
        }
      },
      "typeVersion": 1.1
    },
    {
      "id": "0f0a0663-34d9-4521-91ff-8e3b62e6a0c9",
      "name": "Filter Valid Products",
      "type": "n8n-nodes-base.if",
      "position": [
        2032,
        424
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 1,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "strict"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "valid_product_check",
              "operator": {
                "type": "string",
                "operation": "notEquals"
              },
              "leftValue": "={{ $json.product_name }}",
              "rightValue": "N/A"
            },
            {
              "id": "valid_price_check",
              "operator": {
                "type": "string",
                "operation": "notEquals"
              },
              "leftValue": "={{ $json.price }}",
              "rightValue": "N/A"
            }
          ]
        }
      },
      "typeVersion": 2
    },
    {
      "id": "a539163d-ee9f-4995-be57-aa7e0b45dc14",
      "name": "Send Slack Summary",
      "type": "n8n-nodes-base.slack",
      "position": [
        2480,
        424
      ],
      "parameters": {
        "text": "\ud83d\uded2 Walmart Scrape Complete! | Results Summary: \u2022https://docs.google.com/spreadsheets/d/1WbIzWaXjuQeTxrn_C7PjGMNNibUyLIgY2hdeWh3gOLA/edit?gid=460852058#gid=460852058",
        "select": "channel",
        "channelId": {
          "__rl": true,
          "mode": "name",
          "value": "#testnoor"
        },
        "otherOptions": {
          "includeLinkToWorkflow": true
        }
      },
      "credentials": {
        "slackApi": {
          "name": "<your credential>"
        }
      },
      "executeOnce": true,
      "typeVersion": 2.1
    }
  ],
  "active": false,
  "settings": {
    "executionOrder": "v1"
  },
  "versionId": "81faa070-d36e-4525-96e3-69d4865b9f64",
  "connections": {
    "Add Metadata": {
      "main": [
        [
          {
            "node": "Split Products Array",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "ScrapeOps Proxy": {
      "main": [
        [
          {
            "node": "ScrapeOps Parser",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Schedule Trigger": {
      "main": [
        [
          {
            "node": "Set Search Parameters",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "ScrapeOps Parser": {
      "main": [
        [
          {
            "node": "Add Metadata",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Format Product Data": {
      "main": [
        [
          {
            "node": "Filter Valid Products",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Split Products Array": {
      "main": [
        [
          {
            "node": "Format Product Data",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Filter Valid Products": {
      "main": [
        [
          {
            "node": "Save to Google Sheets",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Send Slack Summary",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Save to Google Sheets": {
      "main": [
        [
          {
            "node": "Send Slack Summary",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set Search Parameters": {
      "main": [
        [
          {
            "node": "ScrapeOps Proxy",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}