How to Automate Market Research Without AI (and Why That's Often Better)

Not every workflow needs AI

Automating market research at scale doesn't need AI. Mine didn't, and it works better for it. The workflow I run today pulls public search results across 400+ keywords per run and turns them into a clean, deduplicated database of providers, with no vision model reading pages and no LLM parsing text. Plain JavaScript, rule-based, done.

That wasn't the first version. The first version used AI, and it failed for reasons that turned out to be predictable once I understood them. The lesson generalizes past this one workflow: the default move right now is to reach for AI first and ask questions later. Sometimes the boring tool wins, and it's worth knowing which situations those are before you build the expensive version.

The problem: manual research doesn't scale past a few keywords

A team needed to know, on a recurring basis, which providers show up for a given market and which of those look like real prospects. The information itself isn't hard to find. It's public: search results, directories, vertical portals.

Doing it by hand once, for ten keywords, is fine. An afternoon of browser tabs gets you a usable list. The trouble starts at volume: dozens of results per keyword, each one needing a human to open it, identify the provider, and compare it against everything already found.

At 400 to 500 keywords and several thousand results per run, that math stops working. You'd need a small team just to keep the list current, and by the time anyone finished a pass, the earlier results would already be stale.

Why I built it without AI

My first approach used a vision model. Screenshot the result pages, feed the images to the model, ask it to extract provider names and fields. It sounded reasonable going in, since the pages weren't uniform HTML I could just pattern-match against, or so I assumed.

It didn't survive contact with real data. The model hallucinated entries, invented provider names that weren't on the page, mismatched fields between neighboring listings, and dropped rows outright. The same site rendered its results as a carousel, a sidebar, or a grid depending on viewport, and the model couldn't keep pace with layouts that kept shifting under it.

So I rebuilt it around a fact I'd overlooked: the source data already has structure. It's HTML. A JavaScript parser looking for fixed patterns in the markup returns the same result regardless of how the page happens to render, no tokens spent, no hallucination risk, no dependency on a model having a good day.

The generalizable version of that lesson: reach for a model when a step genuinely needs judgment, interpreting free text, weighing ambiguous cases, writing something in natural language. When the input already has a schema, a rule-based parser will usually beat a model on cost, speed, and reliability. I go into the broader assistant-versus-agent framing in what's the difference between an AI assistant and an AI agent, if you want the fuller picture of where each approach actually fits.

The approach: collect first, then aggregate

The workflow runs in two phases with a staging store between them, so collection and analysis can each run and fail independently. That separation turned out to matter more than either phase on its own.

Phase 1 collects and extracts. It works through keywords in batches, respecting API limits, and pulls the relevant fields out of each result page. Requests route through residential proxies, since plenty of sites block anything that looks automated, and a cookie-banner handler runs first so the content underneath is actually reachable.

Phase 2 aggregates, deduplicates, and enriches. It reads from the staging store, not live from the source, so a slow or failed collection run doesn't corrupt the analysis step. If you're looking for other starter patterns built the same way, in n8n, the five beginner workflows I'd build first cover a few, and n8n vs. make.com is worth a look if you haven't settled on a platform yet.

The hard parts

Writing the parser was the easy part, an afternoon of work. Almost all the remaining time went into edge cases that only showed up once the workflow scaled from a handful of test keywords to the full 400+.

The index problem

Under ten results, everything stayed clean. The moment a batch of results came back with gaps, timeouts, blocked requests, empty responses, fake duplicates appeared out of nowhere. If nine of ten requests in a batch succeed, the array shifts by one, and keyword five silently gets matched to keyword six's results. The whole mapping breaks without throwing an error anywhere.

The fix was to stop trusting array position entirely. Every result ties back to its original keyword by explicit reference, not by index, so a result knows which keyword and which row it belongs to no matter how many neighboring requests failed.

The duplicate problem

A single provider often shows up more than once: once under their own account, once through a partner channel, sometimes both at the same time for different products. Dropping either row loses real information, the partner attribution.

Fixing the index problem mostly solved this too. The approach that worked was cluster-and-enrich: group every entry for a given provider, pick one as the master record (the most direct entry wins), then scan the rest of the cluster for fields the master is missing and merge them in. Two entries for the same provider get combined, not thrown away.

Fuzzy duplicates

Tiny differences break clean grouping before it even starts. Example GmbH and Example GmbH with a trailing space register as two separate providers. Casing and stray special characters cause the same problem. I normalize aggressively before grouping: lowercase everything, strip whitespace, strip special characters. Different-looking spellings of the same name end up in the same cluster instead of two.

API constraints

Several sites reject automated requests outright unless they're routed through residential proxies. Cookie banners have to be accepted programmatically before any content shows up at all. And export APIs commonly reject strings past a certain length, so a long tracking URL or a stray bit of HTML markup is enough to crash the whole export. Every field gets length-checked and sanitized before it's written.

What surprised me most: how much of the difficulty was reliability at scale, not the extraction logic itself. Getting the parser working took an afternoon. Getting 400+ keywords and several thousand rows to come through clean, every run, took weeks.

Proof: what this runs in production

The workflow runs on n8n, JavaScript, residential proxies, and Google Sheets as the delivery layer, built by one person over about four weeks and running in production since. Each run processes 400+ keywords and lands a clean provider list by email: no duplicates, partner attribution preserved where a provider shows up through more than one channel, direct links to each provider's profile, and a mapping back to whichever keyword surfaced them.

What used to take a team hours of manual browsing now runs unattended and produces a ready contact list on its own schedule. For a look at a different automation built on the opposite end of the spectrum, one where AI genuinely earns its place in the pipeline, the Content Machine case is a useful contrast: same platform, same "one person, a few weeks" scope, but the judgment calls in that workflow are exactly the kind a rule-based parser can't make.

The takeaway I keep coming back to: before reaching for a model, check whether the data you're working with already has a shape. If it does, a parser will probably get you there faster, cheaper, and with fewer surprises at 2 a.m.