AI evaluation for e-commerce
Shopping is where AI mistakes cost real money: a wrong price, an invented product, a budget ignored. I evaluate AI systems on that ground, with the judgement of someone who has built stores, catalogs and brands for sixteen years.
What I evaluate
- Shopping assistant answers: is the product real, is the price current, does the answer respect the user's budget, size and intent.
- Product recommendations and comparisons: are the attributes right, are the trade-offs honest, are the claims supported.
- Catalog outputs: attribute extraction, normalization and categorisation against canonical values.
- Agent tasks: online purchases, cancellations and account workflows, completed safely and documented step by step.
How I evaluate
Rubrics with subscores and gates. Each response is scored on separate dimensions, such as attribute accuracy, completeness, relevance to intent and format. Hard gates override the score: a product that doesn't exist, a wrong price or a wrong category fails the whole response.
Strict where it should be, flexible where it can be. Normalization is checked strictly against canonical values. Recommendations allow more than one good answer, because two different products can both be right for a shopper.
Labelled edge-case suites. Every case has an ID, a label, the input and the expected behaviour, so pass rates can be tracked per label.
Error slicing and triage. Failures are grouped by category, attribute and source, ranked by impact, and each group gets a decision: fix the data, change the rule, add training scenarios or accept it as a documented limitation.
Verification. Every factual claim is checked against a primary source: the manufacturer's page, the retailer's live listing, the regulator's records.
An example
A model describes a bag as genuine leather because one reseller listing says so. The manufacturer's page says faux leather. Fluent, confident and wrong. In my rubric that is an attribute-accuracy failure and, for a material claim, a gate failure.
| ID | Label |
|---|---|
| EC-01 | missing attribute |
| EC-02 | variant split |
| EC-03 | unit mismatch |
| EC-04 | brand typo |
| EC-05 | budget conflict |
Where this comes from
I run my own store, I've restructured catalogs for brands in Europe and the US, and I build and use AI agents in my daily work. I know what a correct answer looks like because I've had to produce one for real customers.
Discuss a project