Transcript: Are leaders deploying AI faster than they can effectively govern it?

DAVID ELLIMAN

Hello and welcome to Tech Tomorrow. I'm David Elliman, Chief of Software Engineering at Zühlke. Each episode we tackle a big question to help you make sense of the fast-changing world of emerging tech.

And today I'm joined by Zahra Shah, an expert in responsible AI and frontier technology. Zahra serves on the board of NexaQuanta, an AI transformation consultancy firm, and is also chair and a founding member of UKAI's Women in AI Working Group. She has extensive experience advising companies on AI governance and ethics and is passionate about creating responsible AI policy. So, who better to help me answer today's question: are leaders deploying AI faster than they can effectively govern it?

ZAHRA SHAH

What we have really noticed is that if you're using AI tools to deploy code, they're literally changing on a daily basis, whereas governance takes time to really keep up with the rapid changes. But one of our key approaches to governance is to make sure that we focus on a responsible AI framework. So, we include looking at ethics by design, safety by design. Actually, a lot of companies we notice do not even realise that they're already using AI. A lot of their employees are already using what we call shadow AI without their knowledge.

DAVID ELLIMAN

Shadow AI. Yeah. Emerging as a term, isn't it?

ZAHRA SHAH

Yes. And then the other thing that we also experience is that companies don’t really know what data is being used, which is, I would say, the most crucial aspect of an effective AI system. So it's almost helping and guiding companies to find and select the right use case because it's almost like if you select the wrong problem to solve, you're never going to get a return on investment.

That is something, I suppose for some organisations it is painful, but it is better to do that at the beginning, at the outset, where you do the due diligence, you do what we call requirements analysis. You come up with the proper use case. You look at your strategy, you ensure that you have, you know, responsible AI principles that you are adopting, things like explainability, transparency.

Are you complying with data privacy regulations? You know, you're making sure that whether you're building it in-house or you're going with an external vendor... You have to look at things like bias. Do you have a strategy for addressing that? How do you manage, if that particular system is hallucinating, what will you do? How do you mitigate that? So, you're thinking about all of that in the beginning.

DAVID ELLIMAN

I mean, that makes perfect sense. To be able to understand and prioritise the risk means that you have to understand what those risks actually are. So there's a degree of understanding. And one of the problems with AI machine learning over the last, since it's become more corporately adopted maybe over the last 20, 25 years, but particularly just in the last few years since the sort of the transformer models and LLMs and that stuff, is that there's a certain degree of black-box-ness around it.

And that's not just exclusive to those, but it’s been before: explainable AI, transparency, repeatability, all those things are kind of hard to do. And if a board wants to adopt a particular AI approach because they've been sold on an idea, it kind of suggests that they've got to actually understand. There's got to be some initial movement to actually say, well, okay, here's the promise. And let's now build in all of the understanding into that in order to do what you just said.

ZAHRA SHAH

And also I know for some people this might be overwhelming. So another approach that we suggest is to do a pilot first, perhaps in the case of a knowledge assistant or an AI agent that helps your employees with, let's say, HR policies.

Something that is low risk and you can try out the pilot so it builds confidence, so you can look at all of the risk mitigating factors when you're developing the pilot. You ensure that you know, you're following your AI strategy, you have a responsible AI policy in place, you have somebody who is overseeing it. You know, if you're not big enough to have a proper board, or an AI ethics committee.

So at least there should be somebody on your existing board who has that responsibility. And if you don't have somebody with the expertise, either you can get external expertise or you can train somebody.

And then as you're deploying the pilot, you also will need to train your staff members. And what we find is actually often there's a big gap between what the system can do and how employees use it. And if they're not trained on how to use a system correctly, I still see a lot of people, they're using AI systems like Google 2.0. So they need education and training. How should I ask the right kind of question, which we call or refer to as prompt engineering.

But you need to provide the right context. You need to provide the right level of guardrails when you are questioning the system. When you're doing a pilot, it gives you the opportunity to kind of test that and then you learn from it.

And you can give yourself as an organisation some leeway. You know, instead of the KPIs for the pilot being, ‘oh we must get a return on investment straight away’. Perhaps the KPI should be, ‘did our X, Y, Z guardrails work well?’ or ‘did we mitigate specific biases?’. You can have realistic KPIs at the pilot stage to give everybody the breathing room to come up to speed.

DAVID ELLIMAN

And I was just thinking for your example there, I mean you gave the example of an HR system and obviously that's a great example because there's a lot of personal information in there.

For anybody listening to this that is going to be thinking, ‘okay, so I hear a lot about responsible AI or ethics, what does it mean on a daily basis?’, and you mentioned guardrails and bias in terms of, there's a project starting next Monday, which is, as you say, is going to be a pilot to do something, maybe some sort of chat assistant within the HR system. What would a sort of a responsible tech on that team try to inject and want to expect from it?

ZAHRA SHAH

You first would look at the data, what data is being used in that organisation. Because a lot of organisations have multiple databases. And if you are using data, which is what is classified as personal data. Then do you have permission or consent of your employees too, because there might be, let's say, medical information, which is very sensitive. There might be information that some employees do not wish to share with other employees in the organisation, for whatever reason.

When you are designing your data sets, then you have to make sure that you take all of that into account. You make sure that the data is accurate, there's no missing information. When you're labelling data, you also make sure that it reflects diversity. You'd have to take that into account when you're designing your dataset.

And then of course, now we have the EU AI Act. So if the organisation has a footprint across Europe, then they will need to also comply with the EU AI Act. So then when you're designing it, you have to make sure that you are complying with all the necessary regulations.

And then when you're selecting the AI model, or whether you're developing it in-house, or for example, if you're using something like Claude. So, then you have to also figure out where will I store, for example, the information. You obviously have cost constraints. Then you would also look at energy efficiency, because some models might be, let's say, more cost effective than others, but they're very expensive to run.

You'd have to look at all of those considerations. You don't want to be stuck in a situation where you ended up going with a certain vendor because you had a good price, but you didn't realise you were tied to that vendor and now you cannot go with any other vendor. And then you realise that it's taking up so much of your memory space, it's very expensive to run for the company. And then you also realise that you're not even using all of the capabilities of the model.

DAVID ELLIMAN

Vendor lock-in is one of the topics we come back to on this show. And yes, it's far more common than most leaders realise. It usually happens not through a single dramatic decision, but through a series of small, sensible sounding ones.

A team picks a model that solves an immediate problem. Data starts flowing into a proprietary fine-tuning pipeline. APIs get baked into the product. Internal tooling prompts, evaluation harnesses, all of it gradually shapes itself around one provider's quirks. By the time anybody steps back and asks the question, switching costs can quietly have become substantial.

When a leader finds themselves locked into a specific vendor, the route back to a more neutral position starts with abstraction. Anytime you call a model in production, that call should go through a thin internal layer of your own, one that hides the specifics of which provider you are using underneath. It also means owning your prompts, your evaluations, and especially your data, so you can repoint the same workflow at a different model and measure honestly what you lose or gain.

And it means routinely running parallel benchmarks against at least one alternative. So you always have a credible plan B. It's important to remember that every company has unique needs and there are many vendors and solutions available. The key is to find the best fit for each specific context.

ZAHRA SHAH

Not every company needs a large language model. So, when you are recommending a solution, then you have to look at, okay, if they're solving an HR problem, perhaps they only need like a small model. They don't need something that does X, Y, and Z, because all the different models are suitable for different tasks. Then you'd make sure that you know the person who's recommending the model to you, and that they're not just recommending it to you for monetary concerns, you know, that the recommendation is based on what is good for the company.

And it might be that they don't need a very fancy model. Perhaps they need only a small, targeted model, and then they can fine tune it for their requirements. And then the other thing you'll also have to look at, there's a technique that we use referred to as Retrieval-Augmented Generation, or RAG.

In very simple terms, what that means is RAG can actually help to reduce or mitigate hallucinations. So for example, in some models where there's more risk of hallucinating, you can use this RAG methodology where you use real time data to mitigate that risk. So there are various things you have to take into account to make sure that you are recommending the right solution to that company, and you're not just recommending the solution that's gonna make you the most money. So you have to understand your risks and then prioritise the key risks, and then have a mitigation process, and then monitor and track it.

DAVID ELLIMAN

Many things about the regulatory compliance or putting the right guardrails in place, checking for biases. It's such a fundamental job that it's hard to see that you would do this without being intrinsic to the project, and yet we see some sort of ethics theatre in some companies where here'll be something where there's a sort of a checkbox of things that are applied after the fact. Maybe a system might be checked way down the line.

Everything that you've just said is like, okay, sleeves up day one, this is how we're going to work together on doing this. And you know, there are a number of challenges like a system in terms of its fairness, accessibility, and maybe even its sustainability footprint are all things that seem to me in this day and age to be essential to put in right at the beginning. And you know, where people think, well we have an ethics officer. It's like, what is that? You know, this is intrinsic to the actual work.

ZAHRA SHAH

Definitely. And also I think because of the nature and the speed of advancement in AI, people are afraid as well of the accountability.

What we have seen is that nobody is willing to take accountability. The legal person thinks it's the technology person's responsibility. The technology person thinks it's, you know, the product person's responsibility.

To be very honest, AI is a tool and now it's being used in every part of the company. So if it's used in finance, the responsibility still remains with the director of finance. If it's used within the legal department, then the responsibility sits with whoever is in charge of the legal department. Similarly with marketing or sales or HR, you can't say that things went bad because of the AI system, because the AI system is there to help you.

So, another way of dealing with that accountability issue is to have a committee of people within the different departments working together. They can share that responsibility as well. And then do, as I suggested, almost like a pre-mortem look at what can go wrong.

And often the person who understands that business function is best placed to figure out the various risks of what could go wrong from a business point of view. And perhaps a person who has expertise in AI can then figure out what can go wrong in terms of the system point of view. And the legal expert can figure out, oh my god, where you can be fined or we can go to jail. You would need people from the different, I would say, areas of expertise, within a committee.

DAVID ELLIMAN

Like a multidisciplinary team?

ZAHRA SHAH

Exactly. And in this way, you know, it can also help with the fear, because then you're working together. By addressing risks at the outset, you prevent the risks from becoming issues and you also take this view that it is a challenging thing, it's not easy, because it's changing so quickly. So you give people a bit of leeway.

DAVID ELLIMAN

You touched on something earlier that I think is worth diving into a little bit. You talked about the EU AI Act and their relationship to having to satisfy that there's a host of different, sort of ethical frameworks that span from homegrown through to maybe the OECD have one, NIST have one.

But just coming back to it, there is, I think, quite a confusing sort of plethora of either regulatory standards that one might have to build against or guidance in terms of how you are supposed to behave and operate, and it's better than none. But I think that some people might be inundated with potential opportunity, and I think it may change the way people react in the light of how they respond to governments.

We're in this kind of unique situation in the UK where we're seeing the regulatory drift that's happening in the US and the tightening up of more regulation in the EU. And I just wondered what your feelings were, you know, a C-level person within a company in the UK, say as an example. Because we're in that sort of middle ground.

ZAHRA SHAH

It is a very interesting kind of global situation, actually. We have the US which is sort of almost with very little or minimal regulation when it comes to AI.

And then you have the EU, which some people would argue as overregulation because the EU AI Act also follows a risk-based approach. It's not a principles-based approach. And that approach is very hard to enforce when it comes to AI systems because AI systems are very dynamic. So the risk changes depending on how the system is being used. A low-risk system, if it's not used correctly, can become a high-risk system very quickly.

And I feel that the UK is in a very interesting situation, UK can utilise this opportunity to come up with its own responsible AI regulation, which is balanced, which is, I would say, taking the middle way between the US and Europe and come up with our own regulation, which is principles based, which is innovation friendly, which protects minorities and vulnerable people in this country. Because you've seen what happened with Grok AI and Ofcom had to take a decision to investigate X, and then there were changes to the Online Safety Act and now currently there's a consultation going on about whether we should ban social media for young people.

And a lot of these issues will keep coming up because of the advancement in AI. And obviously now because we look at what's happening in terms of the geopolitical situation in the Middle East, it makes you realise that we also have to look at sovereign AI, that the UK must ensure that our sensitive aspects of our country are literally, our sovereignty relies on making sure that our sensitive information, data, AI infrastructure, it should really be supported by UK companies. So that's something that they have to think about.

And then you have to also think about making sure that the benefits of AI are equally divided across Britain. That we don't just focus on London, and also that we really focus on supporting the public to upskill people. That's a huge challenge. We'll need to be able to upskill people across the UK, and particularly in underserved parts of the UK, regions in the north, and make sure that when we are coming up with data centres or other AI related projects under the UK government's AI Opportunities Action Plan.

And I know that the action plan does take into account regional hubs, which is good, but this is something that they also have to make sure that we have to be careful about causing an AI divide within the country that might cause division.

DAVID ELLIMAN

There are a lot of polarising predictions about how AI will change the job market and the reality is we're not seeing wholesale displacement, but we are seeing a quieter, more uneven shift. Tasks within roles are being absorbed by AI rather than entire jobs being eliminated. And the people who learn to work alongside these tools are pulling steadily ahead of those who don't.

The risk is a slow widening of the gap between the AI-fluent and everybody else. When it comes to what leaders could be doing to upskill staff, both technical and non-technical, the mistake I see most often is treating upskilling as a training course rather than a capability programme.

A two-hour prompt engineering session does very little on its own. What works is giving people meaningful problems to solve with these tools in their working context with proper coaching and time to experiment safely. And critically, that has to extend well beyond the engineering organisation: finance, legal, HR, marketing, operations even.

Anyone whose job touches information is touching AI now, whether they know it or not. Training and upskilling are of course important, but another key factor in successful AI adoption is trust.

ZAHRA SHAH

The other trend that's happening globally is that people are also seeing this kind of erosion of trust in AI because there are a lot of companies who are not using ethical AI principles. And as people discover that and they've had unfortunately bad experiences, there will come a time where people will want to look for companies that they trust. And in that scenario, the UK can take the leading position.

This can be a country where people can, companies where they feel, let's say in the US you feel that there are certain companies, you can no longer trust those companies because they don't follow any regulation. But then if the UK has this regulation, perhaps even what UKAI is working on, like a trust mark or almost like a sort of a trademark that if companies are developed or trained or built in the UK, that means you can trust them, because they comply with XYZ regulation.

So we are uniquely positioned actually to lead in responsible AI globally and be the light actually for other countries. UK can be the country that takes that, I would say, brave step to make sure that we do things the right way.

DAVID ELLIMAN

Which leads us perfectly to the central question of the episode. So, in your opinion, Zahra, do you think that leaders are deploying AI faster than they can effectively govern it?

ZAHRA SHAH

I would say in some situations, yes.

And my aim is not to blame anybody or criticise anybody, but it's just the nature of the situation. It's changing so rapidly that a lot of leaders, they feel that they have no option. And there are companies or nations that are basing decisions on fear, literally they're not making decisions based on rational objectivity. They're kind of driven by this fear, ‘oh God, if I don't do it, then I'll be left behind’.

So it's a very difficult situation, but actually I was very impressed by what I recently saw with the government of Singapore. They recently launched in January under the World Economic Forum, the world's first governance framework for agentic AI. I kind of analysed that framework, and it's a very good framework because it's based on an agile model, and it also looks into kind of an iterative approach.

Some of the principles that I mentioned, it does take that into account, and I was thinking that perhaps, because the UK is part of the Commonwealth, perhaps the UK could work with other Commonwealth countries like Singapore, and we can share best practise. And the UK has always taken sort of a leading position in terms of legislation and regulation, because that's a strong point.

We can do that in terms of responsible AI and work with other countries, and some other countries have expertise in perhaps helping us in the successful implementation of UK's AI Opportunities Action Plan.

Because we need skills, we need people who can sort of help us in terms of using sustainable sources of energy, which has become, as you know, with the geopolitical situation, most countries realise that they have to look at alternative sources of energy.

So, a lot of these things, these challenges, we can kind of address them by working together with other countries.

DAVID ELLIMAN

Thank you for listening to season three of Tech Tomorrow, brought to you by Zühlke. If you'd like to learn more about what we do, you can find links to our website and more resources in this episode's show notes. You can also listen back to all previous episodes right now in your podcast app of choice. Until next time.