{ "name": "05 - Apify Search Candidate Discovery", "nodes": [ { "id": "n05-note-001", "name": "Overview Note RU", "type": "n8n-nodes-base.stickyNote", "typeVersion": 1, "position": [ -380, -200 ], "parameters": { "width": 520, "height": 520, "content": "## 05 \u2014 Apify: \u043f\u043e\u0438\u0441\u043a \u043a\u0430\u043d\u0434\u0438\u0434\u0430\u0442\u043e\u0432 URL (URL Supplier)\n\n\u041f\u043e\u0438\u0441\u043a\u043e\u0432\u044b\u0439 \u0437\u0430\u043f\u0440\u043e\u0441 \u2192 Apify Google Search Results Scraper \u2192 URL-\u043a\u0430\u043d\u0434\u0438\u0434\u0430\u0442\u044b \u2192 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 url_registry \u2192\n\u0437\u0430\u043f\u0438\u0441\u044c \u0432 url_candidates + \u0441\u0442\u0440\u043e\u043a\u0430 \u0432 discovery_requests.\n\n\u041d\u0415 \u0430\u043d\u0430\u043b\u0438\u0437\u0438\u0440\u0443\u0435\u0442 \u0441\u0430\u0439\u0442\u044b. \u041d\u0415 \u0442\u0440\u0430\u0442\u0438\u0442 Firecrawl/Claude. \u041d\u0415 \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438.\n\u0422\u043e\u043b\u044c\u043a\u043e \u0420\u0423\u0427\u041d\u041e\u0419 \u0437\u0430\u043f\u0443\u0441\u043a, \u041e\u0414\u0418\u041d \u0437\u0430\u043f\u0440\u043e\u0441 \u0437\u0430 \u043f\u0440\u043e\u0433\u043e\u043d (v0.1), \u043c\u0430\u043a\u0441\u0438\u043c\u0443\u043c 10 \u043a\u0430\u043d\u0434\u0438\u0434\u0430\u0442\u043e\u0432.\n\n\u0427\u0442\u043e \u0434\u0435\u043b\u0430\u0435\u0442:\n1. Set Discovery Request \u2014 \u043e\u043f\u0435\u0440\u0430\u0442\u043e\u0440 \u0437\u0430\u0434\u0430\u0451\u0442 query / request_text (\u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e \u00ab\u0437\u0430\u0439\u043c \u043f\u043e\u0434 \u0437\u0430\u043b\u043e\u0433 \u041f\u0422\u0421 \u041c\u043e\u0441\u043a\u0432\u0430\u00bb).\n2. Apify Google Search (sync) \u2192 \u043e\u0440\u0433\u0430\u043d\u0438\u0447\u0435\u0441\u043a\u0438\u0435 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u044b (url, title, snippet, rank).\n3. \u041d\u043e\u0440\u043c\u0430\u043b\u0438\u0437\u0443\u0435\u0442 URL (\u043a\u0430\u043a \u0432 Workflow 04 \u2192 \u043a\u043b\u044e\u0447 \u0441\u043e\u0432\u043f\u0430\u0434\u0430\u0435\u0442 \u0441 url_registry), \u0447\u0438\u0441\u0442\u0438\u0442 \u043c\u0443\u0441\u043e\u0440\u043d\u044b\u0435 \u0441\u0441\u044b\u043b\u043a\u0438.\n4. \u0427\u0438\u0442\u0430\u0435\u0442 url_registry (\u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a \u043f\u0440\u0430\u0432\u0434\u044b \u0434\u0435\u0434\u0443\u043f\u0430) \u0438 \u043f\u043e\u043c\u0435\u0447\u0430\u0435\u0442 dedup_status / registry_status.\n5. \u0414\u0435\u0442\u0435\u0440\u043c\u0438\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u043e (\u0431\u0435\u0437 LLM) \u0441\u0447\u0438\u0442\u0430\u0435\u0442 confidence_score, region_hint, service_hint.\n6. \u041f\u0438\u0448\u0435\u0442 url_candidates (25 \u043a\u043e\u043b\u043e\u043d\u043e\u043a): approval_status=new \u0434\u043b\u044f \u0443\u043d\u0438\u043a\u0430\u043b\u044c\u043d\u044b\u0445, duplicate \u0434\u043b\u044f \u0434\u0443\u0431\u043b\u0435\u0439.\n7. \u041f\u0438\u0448\u0435\u0442 \u043e\u0434\u043d\u0443 \u0441\u0442\u0440\u043e\u043a\u0443 discovery_requests (18 \u043a\u043e\u043b\u043e\u043d\u043e\u043a): status=needs_review (\u0438\u043b\u0438 error).\n\n\u0414\u0430\u043b\u044c\u0448\u0435 \u2014 \u0420\u0423\u0427\u041d\u041e\u0415 \u043e\u0434\u043e\u0431\u0440\u0435\u043d\u0438\u0435: \u043e\u043f\u0435\u0440\u0430\u0442\u043e\u0440 \u0441\u0442\u0430\u0432\u0438\u0442 approval_status=approved \u0438 \u043f\u0435\u0440\u0435\u0434\u0430\u0451\u0442 \u22645 URL \u0432 Workflow 04.\nTelegram-\u0431\u043e\u0442 \u043f\u043e\u0437\u0436\u0435 \u0431\u0443\u0434\u0435\u0442 \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u043e\u043c \u043d\u0430\u0434 discovery_requests + url_candidates + Workflow 04 (\u043d\u0435 \u0434\u0443\u0431\u043b\u0438\u0440\u0443\u044f \u043b\u043e\u0433\u0438\u043a\u0443).\n\n\u041b\u0438\u043c\u0438\u0442\u044b: \u0440\u0443\u0447\u043d\u043e\u0439 \u0442\u0440\u0438\u0433\u0433\u0435\u0440, \u0431\u0435\u0437 \u0440\u0430\u0441\u043f\u0438\u0441\u0430\u043d\u0438\u044f, \u0431\u0435\u0437 Firecrawl/Claude, \u0431\u0435\u0437 \u0437\u0430\u043f\u0438\u0441\u0438 \u0432 monitor_queue/results \u0438 \u0442.\u0434." } }, { "id": "n05-note-002", "name": "Setup & Test RU", "type": "n8n-nodes-base.stickyNote", "typeVersion": 1, "position": [ 2080, -200 ], "parameters": { "width": 460, "height": 380, "content": "## \u041d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u0438 \u0442\u0435\u0441\u0442 (\u0432\u0440\u0443\u0447\u043d\u0443\u044e)\n\n1. \u041d\u0415 \u0430\u043a\u0442\u0438\u0432\u0438\u0440\u043e\u0432\u0430\u0442\u044c workflow (active=false).\n2. \u0421\u043e\u0437\u0434\u0430\u0442\u044c \u0432\u043a\u043b\u0430\u0434\u043a\u0438: discovery_requests (18 \u043a\u043e\u043b\u043e\u043d\u043e\u043a), url_candidates (25 \u043a\u043e\u043b\u043e\u043d\u043e\u043a). url_registry \u0443\u0436\u0435 \u0435\u0441\u0442\u044c (10 \u043a\u043e\u043b\u043e\u043d\u043e\u043a).\n3. \u0421\u043e\u0437\u0434\u0430\u0442\u044c \u043a\u0440\u0435\u0434\u0435\u043d\u0448\u043b Apify: 'Apify API - Marketing Scout' (Header Auth, Header Name=Authorization,\n Value=Bearer , \u0434\u043e\u043c\u0435\u043d api.apify.com). \u0422\u043e\u043a\u0435\u043d \u0432\u0432\u043e\u0434\u0438\u0442\u044c \u0422\u041e\u041b\u042c\u041a\u041e \u0432 n8n, \u043d\u0435 \u0432 \u0444\u0430\u0439\u043b\u044b.\n4. \u041f\u041e\u0421\u041b\u0415 \u0418\u041c\u041f\u041e\u0420\u0422\u0410 \u043f\u0435\u0440\u0435\u043f\u0440\u0438\u0432\u044f\u0437\u0430\u0442\u044c \u043a\u0440\u0435\u0434\u0435\u043d\u0448\u043b\u044b:\n - Apify Search API Request \u2192 Apify API - Marketing Scout\n - Read url_registry / Append url_candidates / Append discovery_requests \u2192 Google Sheets - Marketing Scout Service Account\n5. \u041d\u0430 3 \u043d\u043e\u0434\u0430\u0445 Google Sheets \u0432\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0440\u0435\u0430\u043b\u044c\u043d\u044b\u0439 Spreadsheet ID (\u0437\u0430\u043c\u0435\u043d\u0438\u0442\u044c PASTE_SPREADSHEET_ID_HERE).\n6. \u0412 'Set Discovery Request' \u043f\u0440\u0438 \u0436\u0435\u043b\u0430\u043d\u0438\u0438 \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c query / request_text.\n7. Execute Workflow \u043e\u0434\u0438\u043d \u0440\u0430\u0437.\n\n\u041e\u0436\u0438\u0434\u0430\u0435\u043c\u043e: 1 \u0441\u0442\u0440\u043e\u043a\u0430 discovery_requests (status=needs_review), \u0434\u043e 10 \u0441\u0442\u0440\u043e\u043a url_candidates,\n\u0434\u0443\u0431\u043b\u0438 \u043f\u043e\u043c\u0435\u0447\u0435\u043d\u044b approval_status=duplicate, \u043d\u0438\u043a\u0430\u043a\u0438\u0445 \u0437\u0430\u043f\u0438\u0441\u0435\u0439 \u0432 \u0431\u0438\u0437\u043d\u0435\u0441-\u0432\u043a\u043b\u0430\u0434\u043a\u0438, 0 Firecrawl/0 Claude." } }, { "id": "n05-manual-01", "name": "Manual Start", "type": "n8n-nodes-base.manualTrigger", "typeVersion": 1, "position": [ 120, 300 ], "parameters": {} }, { "id": "n05-set-req-01", "name": "Set Discovery Request", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 340, 300 ], "parameters": { "jsCode": "function pad(n){ return String(n).padStart(2,'0'); }\nfunction moscowIsoNow(){var m=new Date(Date.now()+10800000);return m.toISOString().replace('Z','+03:00');}\nfunction moscowStamp(){var z=function(n){return String(n).padStart(2,'0');};var m=new Date(Date.now()+10800000);return m.getUTCFullYear()+z(m.getUTCMonth()+1)+z(m.getUTCDate())+'_'+z(m.getUTCHours())+z(m.getUTCMinutes())+z(m.getUTCSeconds());}\nconst now = moscowIsoNow();\nconst stamp = moscowStamp();\n\n// OPERATOR: edit query / request_text below before running. One query per run (v0.1).\nreturn [{ json: {\n discovery_request_id: 'disc_'+stamp,\n created_at: now,\n requested_by: 'operator',\n request_text: '\u043d\u0430\u0439\u0442\u0438 \u043a\u043e\u043d\u043a\u0443\u0440\u0435\u043d\u0442\u043e\u0432 \u043f\u043e \u0437\u0430\u0439\u043c\u0430\u043c \u043f\u043e\u0434 \u041f\u0422\u0421 \u0432 \u041c\u043e\u0441\u043a\u0432\u0435',\n query: '\u0437\u0430\u0439\u043c \u043f\u043e\u0434 \u0437\u0430\u043b\u043e\u0433 \u041f\u0422\u0421 \u041c\u043e\u0441\u043a\u0432\u0430',\n region: '\u041c\u043e\u0441\u043a\u0432\u0430',\n service_focus: 'pts_loan',\n requested_limit: 10,\n source_mode: 'apify_search',\n source_api: 'apify/google-search-scraper',\n fixture_mode: false, // \u00a74 set true for a dry run (no Apify call)\n live_mode: true,\n require_approval_token: true, // \u00a74 paid Apify discovery requires an explicit approval token\n approval_token: '', // operator pastes the approved token for a live run (value never logged)\n expected_approval_token: 'WF05_LIVE_APPROVED',\n max_budget_usd: 0.20 // \u00a74 hard cost guard (>0 required)\n}}];" } }, { "id": "n05-build-apify-01", "name": "Build Apify Search Request", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 560, 300 ], "parameters": { "jsCode": "const r = $json;\n// \u00a74 EXECUTABLE APPROVAL/BUDGET GATE (before any Apify call). An unapproved run THROWS here, so the\n// downstream HTTP node never executes => zero external calls. The token VALUE is never emitted.\n(function(){ function s(v){return v==null?'':String(v).trim();}\n var fixture=(r.fixture_mode===true||s(r.fixture_mode).toLowerCase()==='true');\n if(fixture)return; // fixture/dry runs never call Apify\n var live=(r.live_mode!==false);\n var needTok=(r.require_approval_token!==false);\n var tokOk=(!needTok)||(s(r.approval_token)!==''&&s(r.approval_token)===s(r.expected_approval_token));\n var budget=Number(r.max_budget_usd)||0;\n if(!live)throw new Error('WF05 approval gate: live_mode=false \u2014 Apify discovery BLOCKED (zero external calls).');\n if(!tokOk)throw new Error('WF05 approval gate: approval token missing/mismatch \u2014 Apify discovery BLOCKED (approval_token_used='+(s(r.approval_token)!==''?'provided_invalid':'no')+'; token value never logged).');\n if(budget<=0)throw new Error('WF05 approval gate: max_budget_usd must be > 0 \u2014 Apify discovery BLOCKED.');\n})();\nconst query = String(r.query || '').trim();\nconst limit = Math.min(10, parseInt(r.requested_limit) || 10);\n\n// Apify Google Search Results Scraper input \u2014 minimal, low cost: one query, one page, organic only.\nconst apify_body = {\n queries: query,\n maxPagesPerQuery: 1,\n resultsPerPage: limit,\n countryCode: 'ru',\n languageCode: 'ru',\n includeUnfilteredResults: false,\n saveHtml: false,\n saveHtmlToKeyValueStore: false,\n geminiSearch: { enableGemini: false },\n perplexitySearch: { enablePerplexity: false, returnImages: false, returnRelatedQuestions: false },\n chatGptSearch: { enableChatGpt: false },\n copilotSearch: { enableCopilot: false },\n maximumLeadsEnrichmentRecords: 0\n};\n\nreturn [{ json: { ...r, apify_body } }];" } }, { "id": "n05-apify-http-01", "name": "Apify Search API Request", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.2, "position": [ 780, 300 ], "onError": "continueRegularOutput", "alwaysOutputData": true, "parameters": { "method": "POST", "url": "https://api.apify.com/v2/acts/apify~google-search-scraper/run-sync-get-dataset-items?format=json&clean=true", "authentication": "predefinedCredentialType", "nodeCredentialType": "httpHeaderAuth", "sendHeaders": true, "headerParameters": { "parameters": [ { "name": "Content-Type", "value": "application/json" } ] }, "sendBody": true, "contentType": "raw", "rawContentType": "application/json", "body": "={{ JSON.stringify($json.apify_body) }}", "options": { "timeout": 120000 } }, "credentials": { "httpHeaderAuth": { "name": "" } } }, { "id": "n05-normalize-01", "name": "Normalize Apify Results", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 1000, 300 ], "parameters": { "jsCode": "const meta = $('Set Discovery Request').first().json;\nconst limit = Math.min(10, parseInt(meta.requested_limit) || 10);\nfunction cap(s, n){ return (s == null ? '' : String(s)).substring(0, n); }\n\n// Gather raw dataset items from whatever shape the HTTP node produced.\nlet datasetItems = [];\ntry {\n const all = $input.all().map(i => i.json);\n for (const it of all) {\n if (it == null) continue;\n if (Array.isArray(it)) datasetItems.push(...it);\n else datasetItems.push(it);\n }\n} catch (e) { datasetItems = []; }\n\n// Detect Apify error / empty response.\nlet ok = true; let errorPreview = '';\nconst first = datasetItems[0] || {};\nif (datasetItems.length === 0) { ok = false; errorPreview = 'Empty Apify response (no dataset items).'; }\nelse if (first && (first.error || (first.statusCode && first.statusCode >= 400) || first.success === false)) {\n ok = false; errorPreview = cap('Apify error: ' + JSON.stringify(first.error || first.message || first.statusCode || first), 400);\n}\n\n// Collect organic-like results across dataset items.\nfunction collectOrganic(di){\n const out = [];\n if (Array.isArray(di.organicResults)) out.push(...di.organicResults);\n if (Array.isArray(di.organic_results)) out.push(...di.organic_results);\n if (Array.isArray(di.results)) out.push(...di.results);\n if (out.length === 0 && (di.url || di.link)) out.push(di); // item is itself a result row\n return out;\n}\nlet organic = [];\nfor (const di of datasetItems) { if (di && typeof di === 'object') organic.push(...collectOrganic(di)); }\n\n// Drop Google-internal / non-website / login junk URLs.\nfunction isJunk(u){\n const l = String(u || '').toLowerCase();\n if (!l) return true;\n if (!/^https?:\\/\\//.test(l)) return true;\n if (l.includes('google.') && (l.includes('/search') || l.includes('/url?') || l.includes('webcache') || l.includes('translate.google'))) return true;\n if (l.includes('googleusercontent') || l.includes('gstatic.com')) return true;\n if (l.includes('/maps') || l.startsWith('https://maps.')) return true;\n if (l.includes('accounts.google') || l.includes('/signin') || l.includes('/login')) return true;\n return false;\n}\n\n// URL normalizer \u2014 same rules as Workflow 04 (matches url_registry keys).\nfunction normalizeUrl(u){\n u = String(u || '').trim();\n if (!u) return '';\n try {\n const noFrag = u.split('#')[0];\n const url = new URL(noFrag);\n url.protocol = (url.protocol || '').toLowerCase();\n url.hostname = (url.hostname || '').toLowerCase();\n const drop = ['utm_source','utm_medium','utm_campaign','utm_term','utm_content','gclid','yclid','fbclid'];\n for (const p of drop) url.searchParams.delete(p);\n let path = url.pathname || '/';\n if (path.length > 1 && path.endsWith('/')) path = path.slice(0, -1);\n url.pathname = path;\n return url.toString();\n } catch (e) { return u; }\n}\n\nconst seen = new Set();\nconst candidates = [];\nlet rank = 0;\nfor (const r of organic) {\n if (candidates.length >= limit) break;\n const rawUrl = String((r && (r.url || r.link)) || '').trim();\n if (!rawUrl || isJunk(rawUrl)) continue;\n if (seen.has(rawUrl)) continue; // drop exact raw duplicates within the batch\n seen.add(rawUrl);\n rank++;\n const norm = normalizeUrl(rawUrl);\n let domain = '';\n try { domain = new URL(norm).hostname; } catch (e) { domain = ''; }\n if (!domain) { const m = String(rawUrl).match(/^https?:\\/\\/([^\\/?#]+)/i); domain = m ? m[1] : ''; }\n domain = domain.toLowerCase(); if (domain.indexOf('www.') === 0) domain = domain.slice(4);\n candidates.push({\n candidate_url: rawUrl,\n normalized_source_url: norm,\n title: cap(r.title || r.titleText || '', 300),\n snippet: cap(r.description || r.snippet || r.descriptionText || '', 600),\n domain: domain,\n rank: (r.position || r.rank || rank)\n });\n}\nif (ok && candidates.length === 0) { ok = false; errorPreview = errorPreview || 'Apify returned no usable organic results.'; }\n\nreturn [{ json: {\n ok: ok && candidates.length > 0,\n error_preview: errorPreview,\n candidate_count_raw: candidates.length,\n candidates: candidates\n}}];" }, "alwaysOutputData": true }, { "id": "n05-read-reg-01", "name": "Read url_registry", "type": "n8n-nodes-base.googleSheets", "typeVersion": 4, "position": [ 1220, 300 ], "alwaysOutputData": true, "parameters": { "authentication": "serviceAccount", "operation": "read", "documentId": { "__rl": true, "value": "PASTE_SPREADSHEET_ID_HERE", "mode": "id" }, "sheetName": { "__rl": true, "value": "url_registry", "mode": "name" }, "options": {} }, "credentials": { "googleApi": { "name": "" } }, "retryOnFail": true, "maxTries": 3, "waitBetweenTries": 5000 }, { "id": "n05-classify-01", "name": "Classify Candidates", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 1440, 300 ], "parameters": { "jsCode": "const meta = $('Set Discovery Request').first().json;\nconst norm = $('Normalize Apify Results').first().json;\nconst candidates = Array.isArray(norm.candidates) ? norm.candidates : [];\n\n// Read url_registry once (source of truth for dedup). Do NOT scan business tabs.\nlet registry = [];\ntry { registry = $('Read url_registry').all().map(i => i.json).filter(Boolean); } catch (e) { registry = []; }\nconst regSet = new Set(registry.map(r => String(r.normalized_source_url || '').trim()).filter(Boolean));\n\nfunction moscowIsoNow(){var m=new Date(Date.now()+10800000);return m.toISOString().replace('Z','+03:00');}\nfunction moscowStamp(){var z=function(n){return String(n).padStart(2,'0');};var m=new Date(Date.now()+10800000);return m.getUTCFullYear()+z(m.getUTCMonth()+1)+z(m.getUTCDate())+'_'+z(m.getUTCHours())+z(m.getUTCMinutes())+z(m.getUTCSeconds());}\nconst now = moscowIsoNow();\nconst stamp = moscowStamp();\nfunction clamp(n){ return Math.min(100, Math.max(1, Math.round(n))); }\nfunction low(v){ return String(v == null ? '' : v).toLowerCase(); }\n\nfunction extractDomain(u){\n let h = '';\n try { h = new URL(String(u||'').trim()).hostname; } catch (e) { h = ''; }\n if (!h) { const m = String(u||'').match(/^https?:\\/\\/([^\\/?#]+)/i); h = m ? m[1] : ''; }\n h = h.toLowerCase();\n if (h.indexOf('www.') === 0) h = h.slice(4);\n return h;\n}\n// Canonicalize a URL (S2-D6): lowercase scheme/host, strip www, drop fragment + tracking params, drop trailing slash.\nfunction canonicalize(u){\n u = String(u||'').trim(); if (!u) return '';\n try {\n const url = new URL(u.split('#')[0]);\n url.protocol = (url.protocol||'').toLowerCase();\n url.hostname = (url.hostname||'').toLowerCase().replace(/^www\\./,'');\n const drop = ['utm_source','utm_medium','utm_campaign','utm_term','utm_content','gclid','yclid','fbclid','_openstat'];\n for (const p of drop) url.searchParams.delete(p);\n let path = url.pathname || '/';\n if (path.length > 1 && path.endsWith('/')) path = path.slice(0,-1);\n url.pathname = path;\n let out = url.toString();\n if (out.endsWith('/') && (url.pathname === '/' || url.pathname === '')) out = out.replace(/\\/+$/,'');\n return out;\n } catch (e) { return u.split('#')[0].replace(/\\/+$/,''); }\n}\nfunction isRoot(u){\n try { const p = new URL(String(u||'').trim()).pathname || '/'; return (p === '' || p === '/'); } catch (e) { return /^https?:\\/\\/[^\\/?#]+\\/?$/i.test(String(u||'').trim()); }\n}\n\nfunction regionHint(l){\n if (l.includes('\u043c\u043e\u0441\u043a\u0432\u0430') || l.includes('\u043c\u043e\u0441\u043a\u043e\u0432\u0441\u043a') || l.includes('msk') || l.includes('moskva') || /(^|[^a-z\u0430-\u044f])\u043c\u043e([^a-z\u0430-\u044f]|$)/.test(l)) return '\u041c\u043e\u0441\u043a\u0432\u0430/\u041c\u041e';\n return '';\n}\n// Canonical service detection (S2-D4): map evidence to canonical taxonomy services; broad brokerage wins.\nfunction canonService(s){s=low(s);var A={'credit_broker':'credit_brokerage','kreditnyy_broker':'credit_brokerage','generic_lending':'credit_brokerage','secured_auto_loan':'pts_loan','auto_collateral_loan':'pts_loan','zaym_pod_pts':'pts_loan','secured_real_estate_loan':'real_estate_secured_loan','zalog_nedvizhimosti':'real_estate_secured_loan','refinancing':'debt_refinancing','credit_refinancing':'debt_refinancing','ipoteka':'mortgage_brokerage','consumer_loan':'consumer_credit'};return A[s]||s;}\nconst SVC = [\n ['credit_brokerage',['\u043a\u0440\u0435\u0434\u0438\u0442\u043d\u044b\u0439 \u0431\u0440\u043e\u043a\u0435\u0440','\u043f\u043e\u043c\u043e\u0449\u044c \u0432 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438','\u043f\u043e\u0434\u0431\u043e\u0440 \u0431\u0430\u043d\u043a','\u043f\u043e\u043c\u043e\u0449\u044c \u0441 \u043a\u0440\u0435\u0434\u0438\u0442','broker','\u0431\u0440\u043e\u043a\u0435\u0440']],\n ['credit_after_refusals',['\u043f\u043e\u0441\u043b\u0435 \u043e\u0442\u043a\u0430\u0437\u043e\u0432','\u043e\u0442\u043a\u0430\u0437\u0430\u043b\u0438','\u043f\u043b\u043e\u0445\u0430\u044f \u043a\u0440\u0435\u0434\u0438\u0442\u043d','\u043f\u0440\u043e\u0441\u0440\u043e\u0447\u043a']],\n ['pts_loan',['\u043f\u0442\u0441','pts','\u0437\u0430\u043b\u043e\u0433 \u0430\u0432\u0442\u043e','\u043f\u043e\u0434 \u0430\u0432\u0442\u043e','pod-zalog-pts','zalog-pts']],\n ['real_estate_secured_loan',['\u043d\u0435\u0434\u0432\u0438\u0436','\u043a\u0432\u0430\u0440\u0442\u0438\u0440','nedvizh','\u0437\u0430\u043b\u043e\u0433 \u043d\u0435\u0434\u0432\u0438\u0436']],\n ['mortgage_brokerage',['\u0438\u043f\u043e\u0442\u0435\u043a','\u0438\u043f\u043e\u0442\u0435\u0447\u043d','ipotek']],\n ['debt_refinancing',['\u0440\u0435\u0444\u0438\u043d\u0430\u043d\u0441','refinanc']],\n ['business_credit',['\u0434\u043b\u044f \u0431\u0438\u0437\u043d\u0435\u0441\u0430','\u043e\u0431\u043e\u0440\u043e\u0442\u043d','\u0442\u0435\u043d\u0434\u0435\u0440\u043d','\u0438\u043f \u0438 \u043e\u043e\u043e']],\n ['bank_guarantee',['\u0431\u0430\u043d\u043a\u043e\u0432\u0441\u043a \u0433\u0430\u0440\u0430\u043d\u0442','\u0433\u0430\u0440\u0430\u043d\u0442\u0438']],\n ['auto_credit',['\u0430\u0432\u0442\u043e\u043a\u0440\u0435\u0434\u0438\u0442']],\n ['consumer_credit',['\u043f\u043e\u0442\u0440\u0435\u0431\u0438\u0442\u0435\u043b\u044c\u0441\u043a','\u043f\u043e\u0442\u0440\u0435\u0431 \u043a\u0440\u0435\u0434\u0438\u0442']]\n];\nfunction detectServices(blob){ const out=[]; for (const p of SVC){ if (p[1].some(t=>blob.indexOf(t)>=0)) out.push(p[0]); } return out; }\nfunction looksLender(l){\n return l.includes('\u043b\u043e\u043c\u0431\u0430\u0440\u0434') || l.includes('lombard') || l.includes('autolombard') || l.includes('\u0437\u0430\u0439\u043c') || l.includes('\u0437\u0430\u0451\u043c') || l.includes('\u0437\u0430\u0438\u043c') || l.includes('\u043a\u0440\u0435\u0434\u0438\u0442') || l.includes('\u0444\u0438\u043d\u0430\u043d\u0441') || l.includes('finance') || l.includes('credit') || l.includes('loan');\n}\n\n// ---- request-level scope/service representation (S2-D1) ----\nconst queryLow = low(meta.query);\nconst queryServices = detectServices(queryLow);\nconst focusCanon = canonService(meta.service_focus);\nconst broadBrokerage = /\u0431\u0440\u043e\u043a\u0435\u0440|\u043f\u043e\u043c\u043e\u0449\u044c \u0432 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438|\u043f\u043e\u0434\u0431\u043e\u0440 \u0431\u0430\u043d\u043a/.test(queryLow);\nlet requestServicePrimary = broadBrokerage ? 'credit_brokerage' : (queryServices[0] || (focusCanon && focusCanon !== 'unknown' ? focusCanon : 'unknown'));\nconst requestServiceSecondaryArr = [];\nfor (const s of queryServices) if (s !== requestServicePrimary && requestServiceSecondaryArr.indexOf(s) < 0) requestServiceSecondaryArr.push(s);\nif (focusCanon && focusCanon !== 'unknown' && focusCanon !== requestServicePrimary && requestServiceSecondaryArr.indexOf(focusCanon) < 0) requestServiceSecondaryArr.push(focusCanon);\nconst requested_search_scope = broadBrokerage ? 'broad_credit_brokerage' : ('narrow:' + requestServicePrimary);\nconst query_terms = String(meta.query||'').toLowerCase().split(/[^a-z\u0430-\u044f\u04510-9]+/).filter(w => w && w.length >= 3).slice(0, 12);\n\n// ---- candidate classification (S2-D2): regulator / publisher / source_candidate / direct / indirect / irrelevant ----\nconst REGULATORS = ['cbr.ru','gov.ru','nalog.ru','nalog.gov.ru','consultant.ru','garant.ru','gosuslugi.ru','fssp.gov.ru','sudrf.ru','minfin.ru','asv.org.ru'];\nconst AGGREGATORS = ['banki.ru','vbr.ru','finuslugi.ru','sravni.ru','bankiros','myfin','vsezaimy','vse-zaimy','zaim.com','outbank','frbank','brobank'];\nconst DIRECTORIES = ['2gis.','zoon.','yell.ru','flamp.','orgpage','rusprofile','spr.ru','yandex.ru/maps','/maps'];\nconst MEDIA = ['kp.ru','rbc.ru','ria.ru','lenta.','vc.ru','dzen.ru','journal.tinkoff','pikabu','habr.','iz.ru','aif.ru','gazeta.ru'];\nconst MARKETPLACES = ['avito.','youla.','yula.','ozon.','wildberries','market.yandex','drom.ru','auto.ru'];\nconst SOCIALS = ['vk.com','ok.ru','instagram.','t.me','telegram','facebook.','youtube.','tiktok.','twitter.','x.com','dzen.ru/id'];\nfunction candidateType(domain, base){\n const dl = domain.toLowerCase();\n if (REGULATORS.some(x => dl === x || dl.endsWith('.' + x) || dl.includes(x))) return 'regulator';\n if (SOCIALS.some(x => dl.includes(x))) return 'social';\n if (MARKETPLACES.some(x => dl.includes(x))) return 'marketplace';\n if (DIRECTORIES.some(x => dl.includes(x) || base.includes(x))) return 'directory';\n if (AGGREGATORS.some(x => dl.includes(x))) return 'aggregator';\n if (MEDIA.some(x => dl.includes(x))) return 'publisher_article';\n if (/\\/(news|article|articles|blog|stati|statya|reviews|rating|reyting|top-?\\d|luchshi|obzor)(\\/|$)/.test(base)) return 'publisher_article';\n if (looksLender(base) || dl.includes('lombard') || dl.includes('zalog') || dl.includes('credit') || dl.includes('cash') || dl.includes('finans') || dl.includes('zaim')) return 'direct_competitor';\n return 'unknown';\n}\n// competitor_class: how this candidate may be USED. Regulators/publishers are NEVER competitor entities.\nfunction competitorClass(ct){\n if (ct === 'direct_competitor') return 'direct';\n if (ct === 'aggregator') return 'indirect';\n if (ct === 'regulator') return 'regulator';\n if (ct === 'publisher_article') return 'publisher';\n if (ct === 'directory' || ct === 'marketplace' || ct === 'social') return 'source_candidate';\n return 'irrelevant';\n}\n\nconst wantsMarket = /\u0430\u0432\u0438\u0442\u043e|\u043c\u0430\u0440\u043a\u0435\u0442\u043f\u043b\u0435\u0439\u0441|\u043e\u0431\u044a\u044f\u0432\u043b\u0435\u043d|avito|youla|\u044e\u043b\u0430/.test(queryLow);\nconst batchSeen = new Set();\nconst rows = [];\nlet unique = 0, dups = 0, estFc = 0, estClaude = 0;\nconst classCounts = {};\n\nfor (let i = 0; i < candidates.length; i++) {\n const c = candidates[i];\n const canonical_url = canonicalize(c.normalized_source_url || c.candidate_url);\n const key = String(c.normalized_source_url || '').trim() || canonical_url;\n const domain = extractDomain(canonical_url || c.candidate_url);\n const base = ((c.title||'') + ' ' + (c.snippet||'') + ' ' + domain + ' ' + (c.candidate_url||'')).toLowerCase();\n const hay = base + ' ' + queryLow;\n\n const inRegistry = key !== '' && regSet.has(key);\n let dedup_status = 'unique';\n if (inRegistry) dedup_status = 'duplicate_in_registry';\n else if (key !== '' && batchSeen.has(key)) dedup_status = 'duplicate_in_batch';\n if (!inRegistry && key !== '') batchSeen.add(key);\n const registry_status = inRegistry ? 'in_registry' : 'not_in_registry';\n const isDup = (dedup_status !== 'unique');\n const approval_status = isDup ? 'duplicate' : 'new';\n\n const candidate_type = candidateType(domain, base);\n const competitor_class = competitorClass(candidate_type);\n const is_competitor_entity = (competitor_class === 'direct' || competitor_class === 'indirect');\n classCounts[competitor_class] = (classCounts[competitor_class] || 0) + 1;\n\n // per-candidate canonical service hint(s) from candidate evidence (not the query alone)\n const candServices = detectServices(base);\n const service_primary = candServices[0] || (is_competitor_entity ? requestServicePrimary : 'unknown');\n const service_secondary = candServices.slice(1, 3).join(', ');\n\n // Confidence (S2-D3): domain/title/snippet/path/service + class. Regulators/publishers never look like a direct competitor.\n let s = 10;\n if (candidate_type === 'direct_competitor') s += 30;\n if (base.includes('\u0437\u0430\u043b\u043e\u0433')) s += 15;\n if (base.includes('\u043f\u0442\u0441') || base.includes('pts')) s += 15;\n if (base.includes('\u0430\u0432\u0442\u043e') || base.includes('\u0430\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b')) s += 10;\n if (regionHint(hay)) s += 12;\n if (base.includes('\u043a\u0440\u0435\u0434\u0438\u0442') || base.includes('\u0437\u0430\u0439\u043c') || base.includes('\u0437\u0430\u0451\u043c')) s += 8;\n if (base.includes('\u0441\u0442\u0430\u0432\u043a\u0430') || base.includes('\u0441\u0443\u043c\u043c\u0430') || base.includes('\u043e\u0434\u043e\u0431\u0440\u0435\u043d')) s += 8;\n if (service_primary !== 'unknown') s += 8;\n if (looksLender(base)) s += 8;\n if (dedup_status === 'duplicate_in_registry') s -= 50;\n if (candidate_type === 'regulator') s -= 45;\n if (candidate_type === 'publisher_article') s -= 30;\n if (candidate_type === 'directory') s -= 30;\n if (candidate_type === 'aggregator') s -= 20;\n if (candidate_type === 'marketplace' && !wantsMarket) s -= 20;\n if (candidate_type === 'social') s -= 28;\n const confidence_score = clamp(s);\n\n const est_fc = (approval_status === 'new') ? 1 : 0;\n const est_cl = (approval_status === 'new') ? 0.02 : 0;\n if (approval_status === 'new') { unique++; estFc += est_fc; estClaude += est_cl; } else { dups++; }\n\n let note;\n if (isDup) note = 'Duplicate; do not process unless force_reprocess later. Requires human approval before Workflow 04.';\n else if (candidate_type === 'regulator') note = 'Regulator/government source (e.g. cbr.ru) \u2014 NOT a competitor; useful as a monitoring/source candidate only. Do not write as a competitor entity.';\n else if (candidate_type === 'publisher_article') note = 'Publisher/article \u2014 NOT a competitor; may be a source candidate for audience/market signals. Do not write as a competitor entity.';\n else if (!is_competitor_entity) note = 'Source candidate (' + candidate_type + '); review manually. Not a competitor entity.';\n else if (candidate_type !== 'direct_competitor') note = 'Indirect competitor (' + candidate_type + '); review before Workflow 04.';\n else note = 'Direct competitor; requires human approval before Workflow 04.';\n\n rows.push({\n candidate_id: 'cand_' + stamp + '_' + (i + 1),\n discovery_request_id: meta.discovery_request_id,\n created_at: now,\n requested_by: meta.requested_by || 'operator',\n requested_limit: meta.requested_limit || 10,\n query: meta.query || '',\n requested_search_scope: requested_search_scope,\n source: 'apify_search',\n candidate_url: c.candidate_url || '',\n normalized_source_url: key,\n canonical_url: canonical_url,\n is_root: isRoot(c.candidate_url || canonical_url),\n domain: domain,\n candidate_type: candidate_type,\n competitor_class: competitor_class,\n is_competitor_entity: is_competitor_entity,\n title: c.title || '',\n snippet: c.snippet || '',\n rank: c.rank || (i + 1),\n region_hint: regionHint(hay),\n service_hint: service_primary,\n service_primary: service_primary,\n service_secondary: service_secondary,\n confidence_score: confidence_score,\n dedup_status: dedup_status,\n registry_status: registry_status,\n approval_status: approval_status,\n approved_by: '',\n approved_at: '',\n rejection_reason: '',\n estimated_firecrawl_credits: est_fc,\n estimated_claude_cost_usd: est_cl,\n notes: note\n });\n}\n\nreturn [{ json: {\n meta: meta,\n ok: (norm.ok === true) && rows.length > 0,\n error_preview: norm.error_preview || '',\n requested_search_scope: requested_search_scope,\n service_primary: requestServicePrimary,\n service_secondary: requestServiceSecondaryArr.join(', '),\n query_terms: query_terms,\n candidateRows: rows,\n candidate_count: rows.length,\n unique_candidate_count: unique,\n duplicate_count: dups,\n class_counts: classCounts,\n direct_competitor_count: classCounts.direct || 0,\n regulator_count: classCounts.regulator || 0,\n publisher_count: classCounts.publisher || 0,\n source_candidate_count: classCounts.source_candidate || 0,\n estimated_firecrawl_credits: estFc,\n estimated_claude_cost_usd: Math.round(estClaude * 1000) / 1000,\n estimated_apify_cost_usd: null,\n actual_apify_cost_usd: null,\n apify_cost_status: 'unknown'\n}}];" }, "alwaysOutputData": true }, { "id": "n05-expand-01", "name": "Expand Candidate Rows", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 1660, 300 ], "parameters": { "jsCode": "const c = $json;\nconst rows = Array.isArray(c.candidateRows) ? c.candidateRows : [];\n// Fan out 0..10 candidate rows for Append url_candidates. 0 \u2192 nothing appended.\nreturn rows.map(r => ({ json: r }));" } }, { "id": "n05-append-cand-01", "name": "Append url_candidates", "type": "n8n-nodes-base.googleSheets", "typeVersion": 4, "position": [ 1880, 300 ], "parameters": { "authentication": "serviceAccount", "operation": "append", "documentId": { "__rl": true, "value": "PASTE_SPREADSHEET_ID_HERE", "mode": "id" }, "sheetName": { "__rl": true, "value": "url_candidates", "mode": "name" }, "columns": { "mappingMode": "autoMapInputData", "value": {}, "matchingColumns": [], "schema": [] }, "options": {} }, "credentials": { "googleApi": { "name": "" } }, "retryOnFail": true, "maxTries": 3, "waitBetweenTries": 5000 }, { "id": "n05-summary-01", "name": "Build Discovery Request Summary", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 1660, 520 ], "parameters": { "jsCode": "const c = $('Classify Candidates').first().json;\nconst meta = c.meta || $('Set Discovery Request').first().json;\nfunction moscowIsoNow(){var m=new Date(Date.now()+10800000);return m.toISOString().replace('Z','+03:00');}\nconst now = moscowIsoNow();\nconst count = c.candidate_count || 0;\nconst status = (c.ok === true && count > 0) ? 'needs_review' : 'error';\n\nlet notes = (status === 'needs_review')\n ? ('Discovery via Apify Google Search. ' + count + ' candidates (' + (c.unique_candidate_count||0) + ' new, ' + (c.duplicate_count||0) + ' duplicate). Requires human approval before Workflow 04.')\n : ('Apify discovery returned no usable candidates. ' + (c.error_preview || ''));\nnotes = String(notes).substring(0, 500);\n\nreturn [{ json: {\n discovery_request_id: meta.discovery_request_id,\n created_at: meta.created_at || now,\n requested_by: meta.requested_by || 'operator',\n request_text: meta.request_text || '',\n query: meta.query || '',\n region: meta.region || '',\n service_focus: meta.service_focus || '',\n requested_limit: meta.requested_limit || 10,\n source_mode: 'apify_search',\n source_api: 'apify/google-search-scraper',\n status: status,\n candidate_count: count,\n unique_candidate_count: c.unique_candidate_count || 0,\n duplicate_count: c.duplicate_count || 0,\n approved_count: 0,\n estimated_firecrawl_credits: c.estimated_firecrawl_credits || 0,\n requested_search_scope: c.requested_search_scope || '',\n service_primary: c.service_primary || '',\n service_secondary: c.service_secondary || '',\n query_terms: Array.isArray(c.query_terms) ? c.query_terms.join(', ') : '',\n direct_competitor_count: c.direct_competitor_count || 0,\n regulator_count: c.regulator_count || 0,\n publisher_count: c.publisher_count || 0,\n source_candidate_count: c.source_candidate_count || 0,\n estimated_firecrawl_credits: c.estimated_firecrawl_credits || 0,\n estimated_claude_cost_usd: c.estimated_claude_cost_usd || 0,\n estimated_apify_cost_usd: (c.estimated_apify_cost_usd == null ? '' : c.estimated_apify_cost_usd),\n actual_apify_cost_usd: (c.actual_apify_cost_usd == null ? '' : c.actual_apify_cost_usd),\n apify_cost_status: c.apify_cost_status || 'unknown',\n notes: notes\n}}];" } }, { "id": "n05-append-req-01", "name": "Append discovery_requests", "type": "n8n-nodes-base.googleSheets", "typeVersion": 4, "position": [ 1880, 520 ], "parameters": { "authentication": "serviceAccount", "operation": "append", "documentId": { "__rl": true, "value": "PASTE_SPREADSHEET_ID_HERE", "mode": "id" }, "sheetName": { "__rl": true, "value": "discovery_requests", "mode": "name" }, "columns": { "mappingMode": "autoMapInputData", "value": {}, "matchingColumns": [], "schema": [] }, "options": {} }, "credentials": { "googleApi": { "name": "" } }, "retryOnFail": true, "maxTries": 3, "waitBetweenTries": 5000 }, { "id": "wf05-build-runlog", "name": "Build live_source_runs Row", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 1600, 820 ], "parameters": { "jsCode": "function moscowIsoNow(){var m=new Date(Date.now()+10800000);return m.toISOString().replace('Z','+03:00');}\n// WF05 live_source_runs ledger (23 cols, DEC-126). Apify search discovery run. No new external calls here.\nfunction str(v){return v==null?'':String(v).trim();}\nconst s=$('Build Discovery Request Summary').first().json;\nconst cfg=$('Set Discovery Request').first().json||{};\nconst cnt=Number(s.candidate_count)||0;\nconst directCompetitors=Number(s.direct_competitor_count)||0;\nvar _needTok=(cfg.require_approval_token!==false);\nvar _tokOk=(str(cfg.approval_token)!==''&&str(cfg.approval_token)===str(cfg.expected_approval_token));\nvar approvalTokenUsed=_needTok?(_tokOk?'yes':'no'):'not_required';\nconst st=(str(s.status)==='needs_review')?'completed':(cnt>0?'partial':'failed');\nreturn [{ json: {\n run_id:str(s.discovery_request_id)||('disc_'+Date.now()),\n created_at:moscowIsoNow(),\n workflow:'WF05 - Apify Search Candidate Discovery',\n source_family:'web_competitor',\n platform:'apify_search',\n mode:'live',\n approval_token_used:approvalTokenUsed,\n source_allowlist:str(s.query||cfg.query),\n max_items:Number(s.requested_limit)||Number(cfg.requested_limit)||10,\n items_received:cnt,\n items_relevant:directCompetitors,\n items_written_raw:0,\n items_unique:Number(s.unique_candidate_count)||0,\n items_duplicate:Number(s.duplicate_count)||0,\n hard_skipped:0,\n external_calls:1,\n apify_cost_status:'unknown',\n actual_source_cost_usd:null,\n estimated_source_cost_usd:'',\n llm_calls:0,\n estimated_llm_cost_usd:0,\n status:st,\n error_summary:(st==='failed'?str(s.notes):''),\n operator_next_action:'Approve direct_competitor candidates in url_candidates, then run WF06.',\n notes:'Apify google-search discovery; cost not recovered (apify_search). Writes url_candidates (not raw_market_records).'\n}}];" } }, { "id": "wf05-app-runlog", "name": "Append live_source_runs", "type": "n8n-nodes-base.googleSheets", "typeVersion": 4, "position": [ 1820, 820 ], "parameters": { "authentication": "serviceAccount", "operation": "append", "documentId": { "__rl": true, "value": "PASTE_SPREADSHEET_ID_HERE", "mode": "id" }, "sheetName": { "__rl": true, "value": "live_source_runs", "mode": "name" }, "columns": { "mappingMode": "autoMapInputData", "value": {}, "matchingColumns": [], "schema": [] }, "options": {} }, "credentials": { "googleApi": { "name": "" } }, "retryOnFail": true, "maxTries": 3, "waitBetweenTries": 5000 } ], "connections": { "Manual Start": { "main": [ [ { "node": "Set Discovery Request", "type": "main", "index": 0 } ] ] }, "Set Discovery Request": { "main": [ [ { "node": "Build Apify Search Request", "type": "main", "index": 0 } ] ] }, "Build Apify Search Request": { "main": [ [ { "node": "Apify Search API Request", "type": "main", "index": 0 } ] ] }, "Apify Search API Request": { "main": [ [ { "node": "Normalize Apify Results", "type": "main", "index": 0 } ] ] }, "Normalize Apify Results": { "main": [ [ { "node": "Read url_registry", "type": "main", "index": 0 } ] ] }, "Read url_registry": { "main": [ [ { "node": "Classify Candidates", "type": "main", "index": 0 } ] ] }, "Classify Candidates": { "main": [ [ { "node": "Expand Candidate Rows", "type": "main", "index": 0 }, { "node": "Build Discovery Request Summary", "type": "main", "index": 0 } ] ] }, "Expand Candidate Rows": { "main": [ [ { "node": "Append url_candidates", "type": "main", "index": 0 } ] ] }, "Build Discovery Request Summary": { "main": [ [ { "node": "Append discovery_requests", "type": "main", "index": 0 } ] ] }, "Append discovery_requests": { "main": [ [ { "node": "Build live_source_runs Row", "type": "main", "index": 0 } ] ] }, "Build live_source_runs Row": { "main": [ [ { "node": "Append live_source_runs", "type": "main", "index": 0 } ] ] } }, "active": false, "settings": { "executionOrder": "v1" }, "versionId": "05-apify-search-candidate-discovery-v0-1" }