From NewGen Consulting
Which business problem is worth testing?
Describe the task, its current cost or effort, the people affected and the acceptable error rate. Compare AI with simpler alternatives such as rules, search or a workflow change. Choose a pilot with a clear owner and a measurable outcome rather than selecting a model first.
Is the data usable and permitted?
Check completeness, accuracy, freshness and whether the examples represent real operating conditions. Confirm who owns the data and who may access it. Document restrictions on personal or confidential information, retention and third-party processing before submitting data to a provider.
How will quality and safety be evaluated?
Create a representative evaluation set separate from development examples. Include difficult cases, missing information and attempts to obtain unauthorized data. Measure task-specific errors and the cost of human review. Set boundaries for what the system may suggest, what it may execute and when a person must approve the result.
Can the pilot be operated after the demonstration?
Assign responsibility for access control, monitoring, incident handling and changes to prompts, data or models. Estimate recurring usage and support costs. Provide a fallback process when the service is unavailable or confidence is insufficient, and test that people can use it.
Example: extracting orders from documents
For a document-processing pilot, compare extracted product codes and quantities with manually checked examples. Test ambiguous descriptions, duplicates and unreadable files. Route uncertain matches to a person before an order is released. NewGen's logistics case study describes a related delivery context; it does not establish an expected accuracy rate for a new project.
When should an AI pilot stop?
Pause when data permissions are unresolved, the task cannot be evaluated reliably, or the cost of errors outweighs the benefit. Continue only when the pilot meets agreed quality, cost and operating criteria. A decision to improve data or retain a simpler process is also a useful outcome.
Explore the next step
Further reading
NIST AI Risk Management Framework. This guide offers general planning considerations; the scope of an engagement depends on your organization.