How to tell an evidence product from a dashboard

A category has grown up around AI governance, and much of it is genuinely useful for running a programme: workflow, sign-off, coordination across teams who otherwise never speak. The problem is what happens when someone external asks a hard question, and the answer turns out to be a screenshot.

Recording versus proving

Most tools in this space store assertions. Someone confirms a bias assessment was performed, and the system records that a person confirmed it, at a time, with their name attached.

This is real information and it has a ceiling: the record is only as good as the system holding it and the honesty of the person who ticked the box. It establishes that a process was followed, which is a different claim from establishing that a thing is true.

Note the last row. This is not a hierarchy where one replaces the other. They do different jobs, and the failure is buying the left column expecting the right one.

A tick box records that someone was willing to tick it. That is worth something. It is not worth what it is often sold as.

Four questions

1. What artefact comes out? Ask to see it. A dashboard view is not an artefact; a signed statement or a hash you can recompute is. If the answer is a report the platform generates on demand, the platform is the evidence, and platforms are commercial relationships with end dates.

2. Can it be verified without you? The single most informative question in this or any adjacent category. Verification that requires logging in is verification with a commercial dependency, and obligations covering AI systems routinely outlast procurement cycles.

3. What happens if we cancel? A satisfactory answer includes exportable artefacts that remain checkable afterwards. An unsatisfactory one is read-only access for ninety days, which is a countdown on your evidence.

4. What can you not prove?The best question, because the honest answer is long and the dishonest answer is short. A vendor who volunteers that they cannot assess representativeness, cannot establish that a bias evaluation was competent, and cannot verify anything about data they never saw has understood the problem. A vendor whose answer is “we cover the full framework” has told you they are selling coverage of a checklist.

The half nobody can sell you

As set out in the Article 10 piece, the obligations split into claims about bytes and judgements about the world. The first can be made provable. The second requires a competent person forming a defensible view, and no product substitutes for that.

Tooling that implies otherwise creates a specific risk: an organisation with a green dashboard and no actual assessment, which is worse than an organisation that knows it has a gap. The dashboard has removed the discomfort that would have prompted someone to do the work.

What good looks like

• Artefacts that leave the platform and remain checkable.
• A clear line between what is proved and what is recorded.
• Verification that works with the vendor switched off.
• A vendor who tells you what they cannot do before you ask.

None of that is exotic, and very little of it is standard. Which is mostly a statement about how young this market is, and partly a statement about how rarely buyers have asked.