Deduplicate Data Records Using Javascript Array Methods

The workflow JSON

Copy or download the full n8n JSON below. Paste it into a new n8n workflow, add your credentials, activate. Full import guide →

{
  "id": "hQXDUS1zEZFI37kU",
  "name": "Javascript Masterclass2",
  "tags": [],
  "nodes": [
    {
      "id": "81dd117a-81f3-4dff-a7c7-664bc161987e",
      "name": "When clicking 'Test workflow'",
      "type": "n8n-nodes-base.manualTrigger",
      "position": [
        -420,
        80
      ],
      "parameters": {},
      "typeVersion": 1
    },
    {
      "id": "6f507b8c-8a8d-4c89-acdf-f0f4eda9fa94",
      "name": "Sticky Note",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -660,
        -260
      ],
      "parameters": {
        "color": 7,
        "width": 400,
        "height": 580,
        "content": "# \ud83e\uddf1 Data Deduplication Tutorial\n\n**Author:** David Olusola\n\n## \ud83d\ude80 n8n Coaching\nUnlock the full potential of n8n with personalized, one-on-one coaching. Whether you're just getting started or need help solving a specific challenge, I'll guide you step-by-step so you can build confidently and efficiently.\n\n\ud83d\udc49 [Book Coaching Session](mailto:david@daexai.com?subject=n8n%20Coaching%20Request)\n\n## \ud83d\udd27 n8n Consulting\nHave a complex automation project, unique integration needs, or want a custom workflow built from scratch? Let's collaborate to design and implement a robust solution tailored to your business.\n\n\ud83d\udce9 [Inquire About Consulting Services](mailto:david@daexai.com?subject=n8n%20Consultation%20Request)"
      },
      "typeVersion": 1
    },
    {
      "id": "eadab384-f0d6-48b1-b4dd-e0eb106228ac",
      "name": "Sticky Note1",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -240,
        220
      ],
      "parameters": {
        "color": 7,
        "width": 300,
        "height": 360,
        "content": "## \ud83d\udcca Sample Data\n\nThis node creates sample user data with intentional duplicates based on email addresses.\n\n**What it does:**\n- Creates an array of users\n- Includes duplicate entries (same email)\n- Simulates real-world messy data\n\n**Next Step:** Pass this data to the deduplication code node"
      },
      "typeVersion": 1
    },
    {
      "id": "745ba692-ffa8-4ac6-851f-fb9dde8392f2",
      "name": "Sticky Note2",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        100,
        220
      ],
      "parameters": {
        "color": 7,
        "width": 300,
        "height": 380,
        "content": "## \ud83e\uddf9 Deduplication Logic\n\n**Key Concepts:**\n- `filter()` - Creates new array with filtered elements\n- `findIndex()` - Returns index of first matching element\n- `index === self.findIndex()` - Keeps only first occurrence\n\n**Algorithm:**\n1. Parse JSON data\n2. Filter array keeping first occurrence of each email\n3. Transform back to n8n format\n4. Return deduplicated results"
      },
      "typeVersion": 1
    },
    {
      "id": "8bc4bd06-b031-4909-8210-5ec1f11366ff",
      "name": "Sticky Note3",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        460,
        220
      ],
      "parameters": {
        "color": 7,
        "width": 300,
        "height": 380,
        "content": "## \ud83d\udcc8 Results\n\n**What you'll see:**\n- Original count: 6 items\n- Deduplicated count: 4 items\n- Duplicates removed: 2 items\n\n**Use Cases:**\n- Clean data before CRM import\n- Prevent duplicate database entries\n- Data quality improvement\n- ETL pipeline cleanup"
      },
      "typeVersion": 1
    },
    {
      "id": "75a624be-73e9-4b35-ae58-33bc45a0208c",
      "name": "Create Sample Data",
      "type": "n8n-nodes-base.code",
      "position": [
        -200,
        80
      ],
      "parameters": {
        "jsCode": "// Create sample data with intentional duplicates\nconst usersWithDuplicates = [\n  { id: 1, name: \"John Doe\", email: \"user@example.com\", department: \"Engineering\" },\n  { id: 2, name: \"Jane Smith\", email: \"user@example.com\", department: \"Marketing\" },\n  { id: 3, name: \"John Doe\", email: \"user@example.com\", department: \"Engineering\" }, // Duplicate\n  { id: 4, name: \"Bob Johnson\", email: \"user@example.com\", department: \"Sales\" },\n  { id: 5, name: \"Alice Brown\", email: \"user@example.com\", department: \"HR\" },\n  { id: 6, name: \"Jane Smith Updated\", email: \"user@example.com\", department: \"Marketing\" } // Duplicate\n];\n\n// Return the sample data as a single item with usersJson property\nreturn [{\n  json: {\n    usersJson: JSON.stringify(usersWithDuplicates),\n    totalCount: usersWithDuplicates.length,\n    message: \"Sample data with duplicates created\"\n  }\n}];"
      },
      "typeVersion": 2
    },
    {
      "id": "909634f2-52ad-4f98-aad0-7c3d8a8b8e43",
      "name": "Deduplicate Users",
      "type": "n8n-nodes-base.code",
      "position": [
        140,
        80
      ],
      "parameters": {
        "jsCode": "// \ud83e\uddf1 Data Deduplication Code Node\n// Use Case: Clean up duplicates before inserting into DB or CRM\n\n// Parse the users JSON from the previous node\nconst users = JSON.parse(items[0].json.usersJson);\n\n// Deduplicate users based on email address\n// This keeps the first occurrence of each unique email\nconst uniqueUsers = users.filter(\n  (user, index, self) => \n    index === self.findIndex(u => u.email === user.email)\n);\n\n// Log the deduplication results\nconsole.log(`Original count: ${users.length}`);\nconsole.log(`Deduplicated count: ${uniqueUsers.length}`);\nconsole.log(`Duplicates removed: ${users.length - uniqueUsers.length}`);\n\n// Return the deduplicated users in n8n format\n// Each user becomes a separate item in the workflow\nreturn uniqueUsers.map(user => ({ json: user }));"
      },
      "typeVersion": 2
    },
    {
      "id": "59855549-51e4-42df-b586-730ff1e59e67",
      "name": "Display Results",
      "type": "n8n-nodes-base.code",
      "position": [
        500,
        80
      ],
      "parameters": {
        "jsCode": "// Display the final results with statistics\nconst currentItems = items;\nconst uniqueCount = currentItems.length;\n\n// Create a summary of the deduplication process\nconst summary = {\n  deduplicated_users: currentItems.map(item => item.json),\n  statistics: {\n    unique_users_count: uniqueCount,\n    process_completed: true,\n    timestamp: new Date().toISOString()\n  },\n  message: `Successfully deduplicated data - ${uniqueCount} unique users remaining`\n};\n\nreturn [{ json: summary }];"
      },
      "typeVersion": 2
    },
    {
      "id": "2a5c40fc-ff56-4e54-b06e-4c56e2c4da0e",
      "name": "Sticky Note4",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        -140,
        -240
      ],
      "parameters": {
        "color": 7,
        "width": 430,
        "height": 330,
        "content": "## \ud83c\udfaf Learning Objectives\n\n**After this tutorial you'll understand:**\n\n\u2705 How to use JavaScript array methods in n8n\n\u2705 Data deduplication techniques\n\u2705 Working with JSON data in Code nodes\n\u2705 Transforming data between n8n items\n\u2705 Practical use cases for data cleaning\n\n**Best Practices:**\n- Always validate input data\n- Consider performance for large datasets\n- Test with various duplicate scenarios\n- Add error handling for production use"
      },
      "typeVersion": 1
    }
  ],
  "active": false,
  "settings": {
    "executionOrder": "v1"
  },
  "versionId": "03170dda-ff22-468b-be1c-e32424a27c66",
  "connections": {
    "Deduplicate Users": {
      "main": [
        [
          {
            "node": "Display Results",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Create Sample Data": {
      "main": [
        [
          {
            "node": "Deduplicate Users",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "When clicking 'Test workflow'": {
      "main": [
        [
          {
            "node": "Create Sample Data",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

About this workflow

This tutorial demonstrates how to remove duplicate records from a dataset using JavaScript logic inside n8n's Code nodes. It simulates real-world data cleaning by generating sample user data with intentional duplicates (based on email addresses) and walks you through the process…

Source: https://n8n.io/workflows/5730/ — original creator credit. Request a take-down →

More General workflows → · Browse all categories →

Related workflows

Workflows that share integrations, category, or trigger type with this one. All free to copy and import.