Santiago Víquez

Home

Blog

3,360 experiments to understand the future of product marketing

I’ve been wanting to do an experiment to test how often agents recommend the best products for a job, primarily to see if, with today’s intelligence, there is a need for verification or if agents can just figure out which tool is the right one for the job.

I’m interested in this because I believe product marketing will evolve as agents become the new buyers. I expect that products with verifiable claims will win over those that don’t have them.

Supertake@Supertake

“When agents verify claims, the best product will win”

— @santiviquez

Oct 26, 2026View on X

I already wrote about that idea in Product benchmarks and the future of marketing, and about why I think it relates to today’s biggest marketing problem: getting recommended by agents.

The experiment: 3,360 runs across 10 use cases#

For the experiment, I decided to pick a type of problem that would require a vendor-specific API, so that I could write multiple prompts per use case and analyze which vendor gets picked by AI.

I ended up deciding on document processing APIs because it’s a niche yet competitive space, with players of all sizes: big clouds like Microsoft, Google and AWS, smaller startups like Reducto, Extend and LlamaParse, and AI labs like Anthropic, OpenAI and Mistral.

I created 10 use cases covering things like financial tables, legal contracts, invoices, receipts and paper forms. For each, I wrote 4 to 5 variations of the same request. Some asked directly for a recommendation, others started with the problem, others were more casual. I also tried asking the models to write the implementation code instead of recommending a tool.

Complex financial tables, accuracy first

One misplaced number in a table can change an entire financial comparison, and checking thousands of cells by hand defeats the purpose. We extract financial tables from 8,000 annual reports each month, mostly digital PDFs with some scanned pages in small text, in English and French. Our Java service needs CSV tables that keep row labels, column headers, units, negative numbers and footnote references. Merged cells and multi-level headers are especially painful. Analysts can wait for results, and our budget is flexible. Which API should we use?

English and Portuguese receipt photos, in seconds

Building an expense app where users snap a receipt and see the details instantly. ~75k JPEGs a day, mostly English and Portuguese, lots of shadows, folds and bad lighting. Go backend, needs merchant, date, tax and total as JSON. People are staring at their phone the whole time and many bail after three seconds. Occasional errors are OK since users can fix them, but batching receipts for later isn't an option. What should we use?

Redlined contracts as digital PDFs

Which API should we use for extracting text from negotiated contracts? We receive 450 SaaS and services contracts monthly from counterparties, only as PDFs, never editable files. They're digital PDFs exported from a word processor, with selectable text. They're in English, 20 to 40 pages, and have gone through several edits. Crossed-out wording remains visible beside replacements, sometimes within a number, word, table cell, or definition. Our Python service sends each PDF to an API. An AI agent uses the extracted text to answer questions about notice periods, liability caps, insurance limits, and approvals. It must use kept wording, never deleted terms. Getting those answers right is our main concern. What API would you choose?

Three of the ten use cases, word for word as the models got them. The rest are in the appendix.

I tested these across Opus 5.5, Fable 5.1, GPT-6 Astra and GPT-5.6 Sol, at three reasoning effort levels: medium, high and extra high.

In total, I ran 3,360 isolated experiments: 2,880 asking for a recommendation without web search, 240 with search available, and 240 asking for implementation code.

3,360answers from 4 models, each in a fresh session
2,880 asked which API to use
240 asked with web search
240 asked for the code

Every experiment started with a fresh context, and none of the prompts mentioned the names of the APIs being evaluated. Here’s what I found.

79% of recommendations went to big cloud APIs#

The first thing that caught my attention was how much the models favored the big clouds. Of the 2,880 recommendations without search, 2,282 (79%) went to Microsoft Azure Document Intelligence, Google Document AI, Gemini API or AWS Textract.

Which API each model chooses for ten document-processing jobs
Answers050100150200250
Document Intelligence
Document AI
Gemini API
Textract
Mindee
Veryfi
Reducto
Claude API
Mistral OCR
OpenAI API
ABBYY
PDF Extract
Extend
LlamaParse
7 others
Each bar counts the answers that named the API, out of 720 per model: 48 prompts (10 jobs in 4 or 5 wordings each), each run 5 times at 3 effort levels. No web search.

And it wasn’t just one model. On 6 of the 10 use cases, at least three models agreed on the most recommended API. Increasing the reasoning effort didn’t change things much either. It almost looks like the models have a default answer for this category.

With web search, big cloud recommendations dropped to 60%#

I then wanted to see what happened when they could search the web. The results changed, but not dramatically. On the comparable prompts, big clouds went from 79% to 60% of recommendations, while smaller parsing startups went from 5% to 13%.

What changes when the models can search the web first
Asked directlyWith web search
Big clouds79%→60%
Parsing startups5%→13%
Document Intelligence
22.5% → 17.5% ▼
Document AI
20.7% → 16.3% ▼
Gemini API
20.2% → 15.0% ▼
Textract
16.0% → 11.7% ▼
Mindee
5.0% → 3.8% ▼
Reducto
4.8% → 6.3% ▲
Veryfi
4.7% → 7.9% ▲
ABBYY
1.8% → 2.5% ▲
Claude API
1.7% → 1.3% ▼
Mistral OCR
1.5% → 5.0% ▲
OpenAI API
0.3% → 5.0% ▲
Extend
0.0% → 4.6% ▲
6 others
0.8% → 3.3% ▲
The same 30 prompts at high effort: 600 answers asked directly, 240 with a web search tool. APIs named 5 times or fewer share the others row.

What’s interesting is that not every model decided to search. Opus searched in only 10 of its 60 answers, Fable in 24, while the two GPT models searched every time.

When the models actually searched, the big clouds received 47% of recommendations, compared with 85% when they didn’t search. You can see every website their searches returned, opened and cited in the appendix.

How often each model used web search when it had the tool
Opus 5.5
10 of 60 · 0.8 searches each
Fable 5.1
24 of 60 · 1.1 searches each
GPT-6 Astra
60 of 60 · 2.3 searches each
GPT-5.6 Sol
60 of 60 · 3.2 searches each
The same questions with an optional web search tool, at high effort: 60 answers per model. Claude gets Anthropic’s tool, GPT gets OpenAI’s.

When writing code, models favored their own APIs#

Then I tried something different. Instead of asking which API to use, I asked the models to write the code to implement the solution. And suddenly, models suggested that their own APIs were the best option.

Opus 5.5 called the Claude API in 58 of its 60 implementations (97%), even though it had recommended Claude in only 10 of 150 answers to those same prompts. GPT-6 Astra called the OpenAI API in 27 of 60 implementations (45%), despite never recommending it. Sometimes you gotta believe in yourself.

The API each model recommends vs. the API its implementation uses
Recommends itCalls it in code
Opus 5.513%8 of 60of the code used the API it recommended the most
Fable 5.158%35 of 60of the code used the API it recommended the most
GPT-6 Astra43%26 of 60of the code used the API it recommended the most
GPT-5.6 Sol57%34 of 60of the code used the API it recommended the most
Document Intelligence
Document AI
Claude API
Textract
Gemini API
OpenAI API
Mindee
Veryfi
Reducto
ABBYY
Mistral OCR
4 others
Same 10 jobs in 3 wordings at high effort: 150 recommendations and 60 pieces of code per model. The big number is how much of the code used the model’s top pick.

All of this was interesting, but apart from telling me that Anthropic is probably making lots of money by recommending itself, it didn’t tell me whether the recommendations were actually good. Were the big clouds getting recommended because their products were better, or because they’d done a better job of getting AI to recommend them?

Are the smaller players really that behind?#

While working on this experiment, I came across Openbenchmarks. They are creating independent benchmarks based on products’ actual capabilities. They have one on document processing APIs for redlined contracts. Surprisingly, here we see the complete opposite. Openbenchmarks ranks Extend first, followed by Reducto, LlamaParse and Datalab.

Bar chart of accuracy for eight specialised document parsers: Extend 93.1%, Reducto 84.8%, LlamaParse agentic 83.8%, Datalab track changes 80.7%, Datalab convert 63.9%, Pulse 48.4%, Mistral OCR 36.9%, Unstructured 27.0%.

How often an agent answers right from each parser’s output: 1,500 questions on 94 redlined contracts. From Openbenchmarks, 25 October 2026.

Unfortunately, the benchmark doesn’t include the big clouds, so we can’t answer whether the models in my experiment recommended the best tools or simply the most familiar.

The best proxy for technically best#

This is where I drew the line on pouring more money into the experiment. But I was too invested to give up on finding which tool was actually the best. So I started thinking of a different approach. What if I asked an agent to go through the documentation of each API and figure out whether it could handle each of the 10 use cases?

I was happy to find that the three most accurate tools according to the benchmark also ranked near the top in this comparison. Extend, which never got picked by the models in the original experiment, ended up ranking first based on documented capabilities.

Funnily enough, not even Microsoft seems to have its recommendations figured out. Azure Document Intelligence was one of the most recommended APIs, but Microsoft’s own Content Understanding ranked higher in documented capabilities.

If you’re curious about which claims were used to determine whether a use case was supported, I’ve included a graph in the appendix. It looks nice, but I decided not to include it here.

How the ranking changes when an agent checks the docs
The models’ top 4Top 4 by the docs
Picked by the modelsAsked directly
Ranked by the docsUse cases built in
1Document Intelligence680
2Document AI583
3Gemini API562
4Textract457
5Mindee193
6Veryfi128
7Reducto110
8Mistral OCR57
9ABBYY Vantage37
10Claude API27
11OpenAI API15
12PDF Extract13
13LlamaParse9
14Taggun4
15Rossum1
15Cloud Vision1
–Extend0
–Content Understanding0
–Datalab0
–Pulse0
–Unstructured0
–Tesseract0
1Extend10 of 10
2Content Understanding9 of 10
2Reducto9 of 10
4LlamaParse8 of 10
5Document AI8 of 10
5Document Intelligence8 of 10
7ABBYY Vantage7 of 10
8Claude API6 of 10
8Gemini API6 of 10
8Mistral OCR6 of 10
8OpenAI API6 of 10
12Mindee6 of 10
13Taggun5 of 10
13Veryfi5 of 10
15Datalab4 of 10
16Textract4 of 10
17Pulse3 of 10
18Rossum3 of 10
19Unstructured2 of 10
20PDF Extract2 of 10
21Cloud Vision0 of 10
21Tesseract0 of 10
Left: ranked by picks (asked directly, no web search, 2,880 answers). Right: by how many of the 10 use cases each API’s docs say it handles out of the box.

What I find interesting is how much the model’s picks changed just by asking it to use verifiable information. It also makes me wonder how much the quality of these recommendations still depends on how much the person prompting knows or cares to ask the right questions.

I wonder how this will evolve. Will models start checking benchmarks and other verifiable information by default? Or will there still be room to game recommendations with today’s AEO tricks?

My suspicion is that models will become hungry for this kind of information. Which makes it funny that some companies hide their documentation to prevent competitors from copying their features, when that same documentation might be what helps agents understand and recommend their products.

So, do agents recommend the best tools?#

Not reliably, at least not by default. For common jobs, they usually recommend something that works. But when the job has specific requirements, their preference for familiar tools can lead to bad recommendations. What I found most interesting is how much the recommendations changed when the models were asked to verify what each product could actually do. If agents are going to become the new buyers, I think product teams should invest in making their products easier to evaluate and verify. The best AEO might not be more marketing content. It might be making your product easy for an agent to verify.

I open sourced the whole study, so you can re-run it with your own prompts or use cases: github.com/santiviquez/whichtool.


Appendix: the ten use cases we tested#

These are the ten requests we sent the models, each in four or five wordings and with no tool or vendor names. Open one to read every wording exactly as sent.

We process about 25,000 supplier invoices a month. They're PDFs, mostly in English and German. Around 40% are scans with skewed pages, faded text, or stamps over the amounts. Our Python backend needs JSON with supplier details, invoice numbers, taxes, totals, and every line item. A wrong quantity or decimal can lead to an incorrect payment, and our finance team is tired of correcting these manually. We're willing to pay more and wait longer if it means fewer mistakes. Which API should we use?
We're a two-person development team adding invoice extraction to an existing TypeScript web app. We receive around 300 English invoices a week, mostly clean digital PDFs with a few sharp 300 dpi scans. We need invoice number, supplier, date, and total returned as JSON. Our pilot starts next week, and we can't spend days building queues, mapping templates, or training anything. Ideally, we send a file through a simple HTTP request and get the fields back without managing extra infrastructure. Which API should we use?
We archive 1.2 million invoice pages every month. About 95% are clean digital PDFs, and the rest are high-quality scans. They're in English and Italian. Our Python batch pipeline needs only supplier ID, invoice date, currency, and total, saved as JSON. We already validate the extracted amounts against our accounting records. Processing can run overnight, and occasional retries are fine. We have a strict monthly budget of $4,800 for extraction, and paying for features we don't use makes no sense. Which API should we use?
We're extracting financial tables from 8,000 annual reports each month. Most are digital PDFs, but some contain scanned pages with small text. They're in English and French. Our Java service needs CSV tables with row labels, column headers, units, negative numbers, and footnote references preserved. Merged cells and multi-level headers are especially painful. One misplaced number can change an entire financial comparison. Analysts can wait for results, and our budget is flexible, but manually checking thousands of cells defeats the purpose. Which API should we use?
We're building a web interface where analysts upload financial statements and immediately preview the extracted tables. We expect 20,000 pages daily, mostly clean English digital PDFs with occasional high-quality scans. Each page contains one to three financial tables. Our TypeScript backend needs cell values, headers, and row labels as structured JSON. Analysts verify the numbers before exporting, so minor extraction mistakes can be corrected. The problem is that users abandon the preview when a page takes more than five seconds to process. Which API should we use?
We're building an expense app where people photograph receipts and see the extracted details immediately. We expect 75,000 JPEG images daily, mostly English and Portuguese, with shadows, folds, and uneven lighting. Our Go backend needs the merchant, date, tax, and total returned as JSON. Users are staring at their phones while this happens, and many abandon the flow after three seconds. We can handle occasional errors through user corrections, but processing each receipt in a batch later isn't an option. Which API should we use?
We process three million receipts every month for an expense analytics project. They're JPEGs and scanned PDFs, mostly English and Spanish, with mixed quality. Our Python workers only need the merchant, date, currency, and total as JSON. Nobody waits for the results, and we can spread processing across several days. We already have rules to flag suspicious amounts for review. Our budget is $15,000 a month, so even a small difference in the price per page adds up quickly. Which API should we use?
I maintain a small registration system for a nonprofit. We receive around 500 paper forms each week, scanned at 300 dpi into PDFs. They're all in English, with printed names, dates, addresses, and checkboxes. Our backend is in Java, and we need the fields returned as predictable JSON. I'm the only developer maintaining this system, and I have a week to ship the feature. I can't label training data, maintain custom models, or build complicated document workflows. Which API should we use?
Our team handles 450 negotiated SaaS and services contracts each month. Counterparties provide them only as PDFs, never editable files. They're digital PDFs exported from a word processor, with selectable text. The contracts are in English and run 20 to 40 pages. Several rounds of edits leave crossed-out wording visible beside replacements, sometimes within a number, word, table cell, or definition. Our Python service sends each PDF to an API, and an AI agent uses the extracted text to answer questions about notice periods, liability caps, insurance limits, and approvals. Answers must follow kept wording, never deleted terms. We'll pay more and wait longer for accuracy. Which API should we use?
Our team handles 450 negotiated SaaS and services contracts each month. Counterparties provide them only as PDFs, never editable files. They're colour scans at about 200 dpi, images only, with no selectable text. The contracts are in English and run 20 to 40 pages. Several rounds of edits leave crossed-out wording visible beside replacements, sometimes within a number, word, table cell, or definition. Our Python service sends each PDF to an API, and an AI agent uses the extracted text to answer questions about notice periods, liability caps, insurance limits, and approvals. Answers must follow kept wording, never deleted terms. We'll pay more and wait longer for accuracy. Which API should we use?
48 wordings in all. ChatGPT wrote the first eight use cases and Claude reworded each one four ways; ChatGPT wrote all four wordings of the two contract use cases.

Appendix: full comparison table of the 22 APIs against the 10 use cases#

An agent went through the docs of each API and graded every use case as built in, something you’d have to build yourself, unclear, or not supported.

What an agent concludes when it compares 22 APIs on every use case
Built inYou build itUnclearNo
InvoicesFinancial tablesReceiptsFormsRedlined contracts
Built in
Extend10
Content Understanding9
Reducto9
LlamaParse8
Document AI8
Document Intelligence8
ABBYY Vantage7
Claude API6
Gemini API6
Mistral OCR6
OpenAI API6
Mindee6
Taggun5
Veryfi5
Datalab4
Textract4
Pulse3
Rossum3
Unstructured2
PDF Extract2
Cloud Vision0
Tesseract0
Graded from each vendor’s docs on 9 November 2026; no API was run. Built in means a documented feature does the job. Point at a dot for why.

Appendix: the websites the models searched#

The websites the models came across when they could search the web: the pages their searches returned, the ones they opened, and the ones they cited.

Websites the models' searches returned, opened and cited
learn.microsoft.com
90
docs.cloud.google.com
90
docs.aws.amazon.com
86
arxiv.org
76
reddit.com
72
github.com
66
cloud.google.com
65
aws.amazon.com
56
en.wikipedia.org
54
reducto.ai
36
Pages the search tool sent back, counted once per answer. Each bar counts the answers that touched the site, out of 60 per model.