How to Judge an AI Vendor Demo: 10 Tests (India 2026)

To judge an AI vendor demo, ask the vendor to run it live on 20 of your own real documents or messages that they have never seen, then score every output yourself against an answer key you wrote beforehand. A demo on examples the vendor chose proves only that the software works on those examples. The 10 tests below take about 2 hours and tell you whether the vendor deserves a paid pilot.

This guide is for Indian business owners and operations heads who are shortlisting AI vendors for a task such as reading invoices, answering customer queries or sorting enquiries. You will leave with a scorecard, a list of red flags and a clear idea of what to ask for in writing.

Key takeaways

• Never judge a vendor on their own sample data. Bring 20 to 30 of your own real examples.

• Write the correct answers down before the demo, so you are not swayed by confident-sounding output.

• Watch the failure cases. A vendor who shows you where the AI is wrong is more trustworthy than one who shows only wins.

• Ask for accuracy as a number measured on your data, not as a promise.

• A good outcome of a demo is a paid pilot with a fixed price and a pass mark, not a long contract.

Why can you not trust a polished AI demo?

You cannot trust it because the vendor controls the inputs. They pick the cleanest invoice, the easiest question and the best day for the internet connection.

AI tools also fail in a particular way. A language model (the type of AI behind ChatGPT) can state a wrong answer in the same confident tone as a right one. This is called hallucination. In a demo, a hallucination looks like success unless you know the correct answer.

When we run a first call with a new client, the part that usually surprises them is how

different their real data looks from the sample data. Blurred scans, mixed Hindi and English, WhatsApp photos of handwritten bills and half-filled forms are normal in Indian businesses. A demo on tidy samples hides all of it.

How to judge an AI vendor demo: what are the 10 tests?

Run these 10 tests in one sitting, with one person from your side keeping score.

Score each test 0, 1 or 2. Below 12 out of 20, do not move to a pilot with that vendor. Between 12 and 16, ask for a second demo that fixes the weak tests. Above 16, move to a pilot.

Not sure which fits your project? Send us 3–4 lines on WhatsApp and we'll tell you honestly, even if the answer is a tool you can set up yourself.

What should you prepare before the demo?

You should prepare an answer key, a pass mark and a named person who decides what correct means. Without these, every demo looks fine.

Here is a simple preparation list, taking about half a day.

• Pick one task, for example reading 20 supplier invoices or answering 20 real customer questions.

• Collect 20 to 30 real examples, including at least 5 difficult ones.

• Write the correct output for each in a spreadsheet. Do this yourself, not the vendor.

• Decide the pass mark, for example 9 out of 10 invoices read correctly.

• Remove or mask anything sensitive that the vendor does not need, such as Aadhaar numbers or phone numbers. Under India's Digital Personal Data Protection Act, 2023, you are responsible for personal data you share, so share only what the test needs.

If you have not yet checked whether your task and data suit AI at all, start with our AI readiness checklist for small businesses in India. It covers the six checks before any vendor call.

What does a good vendor do when the demo goes wrong?

A good vendor stays calm, shows you the failure and tells you what would fix it. A weak vendor changes the subject. Some of the best signs are small. They say "I don't know yet, we would need to test that. "They ask which mistakes cost you the most. They tell you which part of your task should stay with a person.

Some also tell you that you do not need AI for this task. A spreadsheet formula, a fixed rule or a ready tool is sometimes the right answer. If you are choosing between a ready tool and a custom system, read ChatGPT vs custom AI for business in India before you pay for anything.

What should be in writing before you pay for a pilot?

Six things should be in writing: what the pilot tests, the pass mark, the price, who owns the outputs, what happens to your data, and what happens if the pilot fails.

A pilot is a small, time-limited trial on your real data that shows whether the full build is worth it. The longer cost question is covered in our guide to AI development cost in India.

Who is the right AI vendor for you: us, a freelancer or a ready tool?

The right choice depends on the size of the task and how much support you need after launch. We are not the best fit for every case.

If you think you fit the Ureka row, here is how we work. Every AI project starts with a fixed-price pilot of about 4 weeks on your own data, with accuracy measured on your examples. You own the code, data and models from day one. The pilot price is You can read the steps on our AI pilot page and the wider AI and ML development services.

FAQ

How do I evaluate an AI vendor's demo?

Bring 20 to 30 of your own real examples, including 5 difficult ones, and write the correct answers before the demo. Ask the vendor to run them live and score each output yourself. Judge the failure cases as closely as the successes. A demo on the vendor's own sample data proves very little.

What questions should I ask an AI development company in India?

Ask what accuracy they measure on your data and how, what happens when the AI is wrong, where your data is stored and deleted, who owns the code and the model, and what the pilot costs and proves. A vendor who answers in numbers and plain words is usually safer than one who answers in claims.

How accurate should an AI system be before I use it?

There is no single number. It depends on what a mistake costs you. A system that suggests invoice fields for a person to check can work at lower accuracy than one that posts payments on its own. Set the pass mark before the pilot starts, based on the cost of an error and the time saved.

What is an AI pilot and how long does it take?

An AI pilot is a small, time-limited trial on your real data that shows whether a full build is worth it. At Ureka a pilot takes about 4 weeks, has a fixed price, and measures accuracy on your own data. You decide on the full build only after you see the numbers.

Should I use ChatGPT or a custom AI system?

Use ChatGPT or a similar ready tool for one-off writing, summarising and brainstorming by one person. Consider a custom system when the same task repeats on your own documents or data, the output must plug into your software, or you need control over where data goes. A short call can usually settle which one you need.

Who owns the AI model and data after the project?

It should be you, and it should be written in the contract. Ask for ownership of the code, the data, any trained model and the accounts used. At Ureka the client owns the code, data and models from day one. Also ask how your data is deleted from the vendor's systems at the end.