On AI · Essay
Randy McGraw · 2025
As the rate of AI adoption across enterprises hastens, and agentic AI integration to workflows becomes a competitive necessity, more is at stake for leaders to get AI and data-related deployments right the first time.
I have been engaged in no fewer than three major rescue projects over the last fourteen months, and several more are on the horizon. I believe that even more projects than reported are failing to reach the expected pre-deployment ROI and projections for the value unlock that AI can deliver. This happens across industry verticals; none are immune to the root causes.
As reported recently in Information Week, the number-one reason that nearly two out of five AI projects in the USA end in disappointment is bad initial expectations and the consequent decision-making that flows into ROI projections. There are clearly better and worse ways of originating and deploying projects that I notice happening in the market.
Even within the same enterprise, there can be substantially different experiences and levels of outcome with AI and data projects — pointing to intra-organizational differences in overall approach, and the lack of a unified AI framework for the business that is championed across business units.
There are several reasons that "post-deployment disappointment" is not uncommon. Some frequently co-inhere in a single project, and most gravitate around: poor initial definition of objectives, lack of understanding of the technology, archaic decision-making processes, and poor vendor selection. Here, I focus on vendor selection — and how to avoid its pitfalls.
The AI vendor landscape has recently become crowded. Where I would see an average of two to three competitors in a bidding situation five years ago, that number is now three times higher. Parsing through this is difficult, and many enterprises are stuck with thinking and decision-making processes that are anathema to the AI space — leftovers from the hardware-centric IT era, or simply lacking a fundamental understanding of how data and AI projects differ from web and mobile projects — all leading to a susceptibility to patently unrealistic proposals.
Moreover, the prohibitively high cost structures of many reputable global brand-name firms push payback periods out further than they need be. All of this highlights how rightsizing AI initiatives, and adopting the right ways to evaluate proposals and vendors, are mission critical.
Too often, organizations fall into the trap of selecting vendors based on presentations that are compelling on the surface, or on prior relationships, rather than on the actual performance of the AI itself. To avoid this, businesses must adopt a rigorous, measurable proof-of-concept (POC) process that ensures fair evaluation and optimal selection.
A well-structured POC allows organizations both to think ultra-clearly through what they want to accomplish with their deployment and to properly compare AI vendors on a level playing field. A step-by-step framework for scaled enterprises:
To ensure fairness, all vendors should receive the same dataset. It is also advisable to hold back a portion of the data for blind testing, which helps assess the AI's ability to generalize rather than just memorize patterns. If a vendor does not insist on this, do not use them.
Before vendors start working on the POC, establish key performance indicators for the application, ensuring they are fully tied to the business's real needs:
By setting these parameters upfront, organizations can objectively measure each vendor's AI capabilities — eliminating charlatanism and the tendency of many newer vendors to exploit information asymmetry with outrageous performance claims.
Require all vendors to submit their first round of results within a defined period. This ensures all vendors progress at a similar pace and provides an early indication of approach and methodology. Vendors with real experience — the kind that will de-risk your eventual deployment — will shine by delivering objectively more performance in a shorter timeframe.
Once initial results are in, rank each vendor against the predefined criteria. Share the rankings and performance breakdown with the vendors so they understand where they stand and what needs improvement. This lets you double down on what the enterprise wants, and enjoin any midcourse correction that may be needed.
By offering vendors multiple rounds — a second chance to refine their models based on feedback — organizations can identify which vendors are experienced without exaggeration, and which are truly adaptable and responsive: the hallmarks of a good AI and data vendor. A vendor that understands how AI models improve over time, and commits to continuous improvement round over round, matters. This approach simulates real-world AI deployment, where — unlike buying web or mobile software — continuous improvement is necessary and a proper hallmark of long-term success.
Throughout the POC, evaluate not just the AI's performance but the vendor's ability to work collaboratively with your team; to incorporate feedback and adjust; to provide technical support and domain expertise, including commercial understanding; and to explain their activity in a manner all your teams can understand, in the context of your business.
At the end of the measurable POC period, cross-functional decision-makers will understand which solution performs best on real-world data — how and why; which vendor is easiest to work with and most responsive; which can best customize to the business's unique needs; which embraces the dance-off versus eschews it; and which behaves like a partner who ensures the intelligent deployment gets more intelligent over time.
The AI and data world is not the hardware-centric IT world, nor the fixed-point web software world. Evaluating directions and deciding among vendors requires an enterprise to think differently. Running a comprehensive, pre-structured, data-driven evaluation process minimizes the risk of choosing a vendor based on marketing materials, previous relationships, polished sales pitches, unsubstantiated claims — or even graft.
In the AI and data world, the negative impacts of a poor decision can be far greater than in the software world. A measurable process ensures that the natural asymmetry of information between enterprise and vendor does not gravitate toward charlatanism, or even simply unsubstantiated claims. It ensures that the selected vendor is not only technologically superior but the best fit for the organization's operational and strategic needs — approaching the relationship more like a partner than a one-and-done vendor. It also ensures the enterprise does not overspend for results.
By adopting a measurable POC process, organizations can confidently invest in AI solutions that drive real business value — a strong foundation for long-term AI success.
© Randy McGraw. All rights reserved. This article may not be reproduced, copied, or used in any form without the written permission of the author.
← All On AI