How to choose an AI vendor or LLM provider

There are two kinds of AI purchasing decisions most businesses face. The first is choosing an AI-powered product, like a meeting assistant, a support tool or a document processor. The second is choosing a model provider, the company whose large language model (LLM) powers a custom tool or your team's general-purpose assistant. The questions overlap heavily, and in both cases the demo is the least informative part of the process.

Start with the job, not the vendor

Before talking to vendors, write down in a few sentences what you need the tool to do, who will use it, which systems it must connect to, and what data it will touch. This keeps you from being steered toward features you don't need and makes comparisons fair.

Also decide what a successful trial looks like. For example, "correctly extracts the key fields from at least 45 of 50 real invoices" or "answers our staff's top 30 policy questions accurately with a link to the source document."

Data handling: read the actual terms

This is where vendors differ most, and where assumptions cause problems. Get answers in writing, ideally in the contract or data processing agreement, to these questions:

• Is our data used to train or improve models? Many providers offer business plans where it isn't by default, but settings and terms vary between products and plan levels.
• How long is our data retained, and can we have it deleted?
• Where is it stored and processed? This matters if you have customers or contracts with location requirements.
• Which subprocessors are involved? Many AI products are built on another company's model, so your data may pass through more than one provider.
• Will you sign the agreements we need? For example, a data processing agreement, or a business associate agreement if you're handling protected health information.

Consumer versions of AI tools often have different terms than business versions of the same product. Make sure the plan you're evaluating is the plan you'll actually buy.

Security basics

You don't need to run a full security audit for every tool, but a few basics are reasonable to expect from any vendor handling business data:

• Single sign-on with your existing company accounts, so access ends when someone leaves.
• Role-based permissions, so people only see what they should.
• Audit logs showing who did what.
• Encryption in transit and at rest.
• Independent security attestations where appropriate for the data involved. Ask what they have and request the report.

For tools that take actions or connect to other systems, ask how they defend against prompt injection, where hidden instructions in an email or document try to hijack the AI. It's the first item in the OWASP Top 10 for LLM applications, a useful free reference for AI security risks.¹

Test accuracy with your own examples

Vendor demos use examples chosen to look good. Your business has messier data. Insist on a trial with your own material, including the awkward cases: poorly scanned documents, unusual requests, incomplete records.

Build a small test set of real examples, perhaps 30 to 100, where you know the correct answer. Run each candidate against the same set and score the results. This takes a few hours and is by far the most reliable way to compare vendors. Keep the test set; you can reuse it later to check whether a vendor's updates made things better or worse.

Pay attention to how the tool fails, not just how often. A tool that says "I'm not sure" is safer than one that confidently makes things up.

Choosing a model provider

If you're building a custom tool or choosing a general-purpose assistant for your team, you'll pick among the major model providers, and possibly open-weight models you host yourself. A few practical points:

• Capability differences are real but shift often. Leading models leapfrog each other regularly. Test on your task rather than relying on published rankings.
• Bigger isn't always better. Smaller, faster models are often good enough for classification, extraction and routing, and cost much less per use.
• Consider availability through your existing cloud. Some models are offered through the major cloud platforms, which may simplify contracts and security reviews if you already use one.
• Avoid unnecessary lock-in. Design custom tools so the model can be swapped without a rewrite. That lets you move as prices and capabilities change.
• Open-weight models give you more control over where data goes but shift hosting, security and maintenance work onto you.

Understand the real cost

AI pricing comes in several forms: per seat, per use, per token (a unit of text processed) or a combination. Ask the vendor to estimate your monthly cost using your expected volume, and ask what happens if usage grows. Include the costs around the tool too: setup, integration, training and the staff time spent reviewing output. Our guide to measuring ROI honestly has a fuller list.

Plan your exit before you enter

The AI market is young and changing fast. Vendors may be acquired, change pricing or shut down. Before signing, ask:

• Can we export our data, configurations and history in a usable format?
• What notice do we get before price or terms changes?
• What happens to our data if we cancel?
• How long is the commitment, and can we start with a shorter term?

A vendor scorecard

Score each finalist from 1 to 5 on these criteria, weighted by what matters most to you:

For a quick list of questions to bring to vendor calls, see our resources page. If you'd like an independent second opinion on a shortlist, book a free call. Our recommendation will be based on what fits your business.