An AI implementation needs more than a convincing demonstration. Evaluate the workflow it supports, the information it can access and the decisions people must still make. Ask for evidence that matches the claimed outcome and the actual operating conditions.
Start with the delivered capability
The BRD project presents an assistant connected to a banking knowledge base. Retrieval provides material for an answer; it does not guarantee that the answer is complete or correct. Relevant evaluation questions include whether the right document was retrieved, whether the answer is supported by it and how the system behaves when information is missing.
Define what a result means
A resolution rate needs a denominator, a time period and a definition of a resolved request. Time savings require a comparable baseline. Review who measured the result and which users and tasks were included. An implementation description and a measured business outcome are different types of evidence; request the one needed for your decision.
Test failures and boundaries
- Questions for which no reliable source exists.
- Conflicting or outdated documents.
- Requests that exceed the user's access rights.
- External service failures and unexpected costs.
- Cases that require escalation to a person.
Use questions from the actual workflow and agree acceptance criteria before the pilot. Re-test after changes to the model, sources or retrieval process.
Agree ownership after launch
Specify who updates sources, reviews errors, monitors usage and approves changes. Integrations, access controls and support are part of the operating system, not details to defer indefinitely.
Explore AI development or when a simpler solution may be more appropriate. A useful first step is a bounded evaluation with a documented decision about the next investment.
