3,360 experiments to understand the future of product marketing
I’ve been wanting to do an experiment to test how often agents recommend the best products for a job, primarily to see if, with today’s intelligence, there is a need for verification or if agents can just figure out which tool is the right one for the job.
I’m interested in this because I believe product marketing will evolve as agents become the new buyers. I expect that products with verifiable claims will win over those that don’t have them.
“When agents verify claims, the best product will win”
— @santiviquez
I already wrote about that idea in Product benchmarks and the future of marketing, and about why I think it relates to today’s biggest marketing problem: getting recommended by agents.
The experiment: 3,360 runs across 10 use cases#
For the experiment, I decided to pick a type of problem that would require a vendor-specific API, so that I could write multiple prompts per use case and analyze which vendor gets picked by AI.
I ended up deciding on document processing APIs because it’s a niche yet competitive space, with players of all sizes: big clouds like Microsoft, Google and AWS, smaller startups like Reducto, Extend and LlamaParse, and AI labs like Anthropic, OpenAI and Mistral.
I created 10 use cases covering things like financial tables, legal contracts, invoices, receipts and paper forms. For each, I wrote 4 to 5 variations of the same request. Some asked directly for a recommendation, others started with the problem, others were more casual. I also tried asking the models to write the implementation code instead of recommending a tool.
One misplaced number in a table can change an entire financial comparison, and checking thousands of cells by hand defeats the purpose. We extract financial tables from 8,000 annual reports each month, mostly digital PDFs with some scanned pages in small text, in English and French. Our Java service needs CSV tables that keep row labels, column headers, units, negative numbers and footnote references. Merged cells and multi-level headers are especially painful. Analysts can wait for results, and our budget is flexible. Which API should we use?
Building an expense app where users snap a receipt and see the details instantly. ~75k JPEGs a day, mostly English and Portuguese, lots of shadows, folds and bad lighting. Go backend, needs merchant, date, tax and total as JSON. People are staring at their phone the whole time and many bail after three seconds. Occasional errors are OK since users can fix them, but batching receipts for later isn't an option. What should we use?
Which API should we use for extracting text from negotiated contracts? We receive 450 SaaS and services contracts monthly from counterparties, only as PDFs, never editable files. They're digital PDFs exported from a word processor, with selectable text. They're in English, 20 to 40 pages, and have gone through several edits. Crossed-out wording remains visible beside replacements, sometimes within a number, word, table cell, or definition. Our Python service sends each PDF to an API. An AI agent uses the extracted text to answer questions about notice periods, liability caps, insurance limits, and approvals. It must use kept wording, never deleted terms. Getting those answers right is our main concern. What API would you choose?
I tested these across Opus 5.5, Fable 5.1, GPT-6 Astra and GPT-5.6 Sol, at three reasoning effort levels: medium, high and extra high.
In total, I ran 3,360 isolated experiments: 2,880 asking for a recommendation without web search, 240 with search available, and 240 asking for implementation code.
Every experiment started with a fresh context, and none of the prompts mentioned the names of the APIs being evaluated. Here’s what I found.
79% of recommendations went to big cloud APIs#
The first thing that caught my attention was how much the models favored the big clouds. Of the 2,880 recommendations without search, 2,282 (79%) went to Microsoft Azure Document Intelligence, Google Document AI, Gemini API or AWS Textract.
And it wasn’t just one model. On 6 of the 10 use cases, at least three models agreed on the most recommended API. Increasing the reasoning effort didn’t change things much either. It almost looks like the models have a default answer for this category.
With web search, big cloud recommendations dropped to 60%#
I then wanted to see what happened when they could search the web. The results changed, but not dramatically. On the comparable prompts, big clouds went from 79% to 60% of recommendations, while smaller parsing startups went from 5% to 13%.
What’s interesting is that not every model decided to search. Opus searched in only 10 of its 60 answers, Fable in 24, while the two GPT models searched every time.
When the models actually searched, the big clouds received 47% of recommendations, compared with 85% when they didn’t search. You can see every website their searches returned, opened and cited in the appendix.
When writing code, models favored their own APIs#
Then I tried something different. Instead of asking which API to use, I asked the models to write the code to implement the solution. And suddenly, models suggested that their own APIs were the best option.
Opus 5.5 called the Claude API in 58 of its 60 implementations (97%), even though it had recommended Claude in only 10 of 150 answers to those same prompts. GPT-6 Astra called the OpenAI API in 27 of 60 implementations (45%), despite never recommending it. Sometimes you gotta believe in yourself.
All of this was interesting, but apart from telling me that Anthropic is probably making lots of money by recommending itself, it didn’t tell me whether the recommendations were actually good. Were the big clouds getting recommended because their products were better, or because they’d done a better job of getting AI to recommend them?
Are the smaller players really that behind?#
While working on this experiment, I came across Openbenchmarks. They are creating independent benchmarks based on products’ actual capabilities. They have one on document processing APIs for redlined contracts. Surprisingly, here we see the complete opposite. Openbenchmarks ranks Extend first, followed by Reducto, LlamaParse and Datalab.

How often an agent answers right from each parser’s output: 1,500 questions on 94 redlined contracts. From Openbenchmarks, 25 October 2026.
Unfortunately, the benchmark doesn’t include the big clouds, so we can’t answer whether the models in my experiment recommended the best tools or simply the most familiar.
The best proxy for technically best#
This is where I drew the line on pouring more money into the experiment. But I was too invested to give up on finding which tool was actually the best. So I started thinking of a different approach. What if I asked an agent to go through the documentation of each API and figure out whether it could handle each of the 10 use cases?
I was happy to find that the three most accurate tools according to the benchmark also ranked near the top in this comparison. Extend, which never got picked by the models in the original experiment, ended up ranking first based on documented capabilities.
Funnily enough, not even Microsoft seems to have its recommendations figured out. Azure Document Intelligence was one of the most recommended APIs, but Microsoft’s own Content Understanding ranked higher in documented capabilities.
If you’re curious about which claims were used to determine whether a use case was supported, I’ve included a graph in the appendix. It looks nice, but I decided not to include it here.
What I find interesting is how much the model’s picks changed just by asking it to use verifiable information. It also makes me wonder how much the quality of these recommendations still depends on how much the person prompting knows or cares to ask the right questions.
I wonder how this will evolve. Will models start checking benchmarks and other verifiable information by default? Or will there still be room to game recommendations with today’s AEO tricks?
My suspicion is that models will become hungry for this kind of information. Which makes it funny that some companies hide their documentation to prevent competitors from copying their features, when that same documentation might be what helps agents understand and recommend their products.
So, do agents recommend the best tools?#
Not reliably, at least not by default. For common jobs, they usually recommend something that works. But when the job has specific requirements, their preference for familiar tools can lead to bad recommendations. What I found most interesting is how much the recommendations changed when the models were asked to verify what each product could actually do. If agents are going to become the new buyers, I think product teams should invest in making their products easier to evaluate and verify. The best AEO might not be more marketing content. It might be making your product easy for an agent to verify.
I open sourced the whole study, so you can re-run it with your own prompts or use cases: github.com/santiviquez/whichtool.
Appendix: the ten use cases we tested#
These are the ten requests we sent the models, each in four or five wordings and with no tool or vendor names. Open one to read every wording exactly as sent.
Appendix: full comparison table of the 22 APIs against the 10 use cases#
An agent went through the docs of each API and graded every use case as built in, something you’d have to build yourself, unclear, or not supported.
| Invoices | Financial tables | Receipts | Forms | Redlined contracts | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Built in | |||||||||||
| 10 | |||||||||||
| 9 | |||||||||||
| 9 | |||||||||||
| 8 | |||||||||||
| 8 | |||||||||||
| 8 | |||||||||||
| 7 | |||||||||||
| 6 | |||||||||||
| 6 | |||||||||||
| 6 | |||||||||||
| 6 | |||||||||||
| 6 | |||||||||||
| 5 | |||||||||||
| 5 | |||||||||||
| 4 | |||||||||||
| 4 | |||||||||||
| 3 | |||||||||||
| 3 | |||||||||||
| 2 | |||||||||||
| 2 | |||||||||||
| 0 | |||||||||||
| Tesseract | 0 | ||||||||||
Appendix: the websites the models searched#
The websites the models came across when they could search the web: the pages their searches returned, the ones they opened, and the ones they cited.