“How you gather, manage, and use information will determine whether you win or lose.” – Bill Gates
The road to successful GenAI adoption will be full of many twists and turns, and your ability to understand what’s working, what’s not, and where you need to pivot will play a large role in determining your overall impact. Unfortunately, the target state for success with GenAI is far less clear than with most strategic initiatives, like a merger integration or expanding an existing business model into a new market. As a result, you’ll need to be clear about your objectives, while being adaptive and flexible along the journey. You’ll also need to solve a range of new measurement challenges that are still a bit of work in progress. Here are some things to keep in mind as you work to accelerate your impact with GenAI. We call it your GenAI GPS.
Start with ‘Strategic Intent’ and embrace adaptability
Though GenAI’s potential value is compelling, exactly what ‘great is going to look like’ in 2025 and 2026 is difficult, if not impossible to forecast. Given that reality, leaders will need to define a clear ‘Strategic Intent’ that defines what types of opportunities they’d like to pursue and what types of challenges they’re hoping to solve, and then be comfortable with both evolving significantly over the next 12 – 18 months.
Define clear strategic Outcomes for your GenAI efforts
With the pace of GenAI innovation, leaders should focus on defining clear and measurable business Outcomes for their adoption efforts, leaving significant flexibility around the ‘how’ they’re delivered. Leading indicators that focus on experimentation and targeted learnings are particularly impactful during early-stage innovation efforts like GenAI adoption in 2024.
You’ll need to solve for a range of new measurement challenges
GenAI solutions, particularly those that leverage Large Language Models (LLMs), present a wide range of new monitoring and measurement challenges that need to be systematically addressed. As an example, we’ve historically measured if a system is available and how long it takes to complete a defined task. That’s because traditional systems are programmed to take specific actions in response to defined inputs. LLMs, in contrast, are asked to generate custom responses to user prompts, based on the context of the interaction and the desired communication style and range of ‘creativity’. It turns out that the very thing that enables LLMs to replicate more natural, human-like interactions, is much, much harder to monitor for quality.
Leverage integrated insights to inform your GenAI journey…
To generate a comprehensive-view of the effectiveness of your GenAI efforts, you’ll need to leverage a wide-range of integrated measures of success, including:
- Security
- Customer Satisfaction & Loyalty
- Efficiency Impacts
- Revenue Impacts (for customer facing solutions)
- Adoption
- Performance
- Costs
- Responsible AI Measures (e.g. Bias, Fairness, Explainability, Inclusivity, etc.)
- Supportability & Scalability
- Brand Impacts
Refine your GenAI strategy and plans quarterly, and execute relentlessly in-between…
GenAI may accelerate adoption of Quarterly adaptability best practices. While evaluating and optimizing your strategy and plans is becoming increasingly important in general, it’s absolutely critical when adopting GenAI. As best practices are rapidly evolving, it’s important to define and align your strategy, plans and tactics on a quarterly basis, and then execute relentlessly against committed plans. It’s likely that new industry innovation is going to take place intra-quarter across some or many areas of your GenAI efforts. When it does, unless the changes are absolutely disruptive, it’s important to remain disciplined, execute your committed roadmap, and then reassess as inputs into the next quarter. Otherwise, teams can spend the majority of their time reacting to new announcements instead of delivering meaningful increments of value every quarter.
What We’re Seeing in Practice
- GenAI Measurement is really hard…: It’s difficult to overstate the challenges that LLM-based solutions present from a measurement perspective. Innovations in the space are occurring on a continual basis, and this will become less of an issue in 2024, but as of the beginning of the year we still have a long way to go.
- System latency remains a challenge for enterprise-scale LLM-based systems: Latency is a measure of a system’s responsiveness to user requests, and it’s a key challenge for LLM-based solutions in the enterprise space. Foundational models, like OpenAI’s ChatGPT and Google’s Bard have been trained on vast quantities of data, and respond to user prompts based on their existing training. When LLM-based solutions are asked to provide nuanced responses to company-specific topics, they frequently need to access data they weren’t trained on. As of early 2024, this can drive significant delays in response time and can be a very real customer dissatisfier. There’s likely to be significant innovation in this space in coming quarters, but it needs to be addressed as part of your adoption efforts.
- Solving for hallucination and Responsible AI needs is critical, but can add to system performance challenges: Who knew computers could ‘hallucinate’? It turns out that LLMs are the worst type of liars – they do it frequently, and extremely convincingly… We’ve collectively come a long way in solving for hallucination, and many organizations even leverage a second LLM to ‘fact check’ the primary LLMs responses prior to sharing them with users. These efforts aren’t foolproof yet, though, and can further contribute to the latency issues described above.
- GenAI measurement adds complexity to a space where most organizations already struggle: While GenAI presents a wide-range of new measurement challenges, let’s face it – many organizations already struggle to deliver the integrated insights leaders are looking for. Ideally, GenAI will add momentum to in-flight efforts to mature integrated system measurement, but the additional complexity won’t make it any easier…
- Companies can’t afford to ‘fly blind’ when it comes to GenAI insights: The complexity, pace, and inherent risks of GenAI solutions place a premium on an organization’s ability to understand where things are working, where they’re not, and where a different approach may be needed.
“The Goal is to turn data into information, and information into actionable insights” – Jeanne Harris, Columbia University
If 2024 is the year we begin scaling customer-facing GenAI solutions, it will need to be the year we solve for a range of measurement challenges. While there’s a lot of momentum in this space, significant work remains and it needs to be a core focus of your adoption efforts if you’re looking to accelerate your GenAI journey.
This is the fifth blog of an integrated ‘Keys to Winning with GenAI’ series.
Previous Blogs in the series:
- Accelerated Innovations ‘Keys to Winning’ with GenAI
- Defining a Compelling GenAI ‘True North’ for Your Business
- Ensuring You Have the Skills & Capabilities to Win with GenAI
- Building & Sustaining Momentum for your GenAI Adoption Efforts
- Streamlining GenAI Innovation & Solution Delivery
To learn more about Accelerated Innovation’s end-to-end GenAI enablement services, or specific topics of interest, schedule a call and get started on our GenAI Journey.
