Transcript: Is irrational AI making our decision-making worse?

DAVID ELLIMAN

Hello, and welcome to Tech Tomorrow. I'm David Elliman, Chief of Software Engineering at Zühlke. Each episode, we tackle a big question to help you make sense of the fast-changing world of emerging tech.

Today, I'm joined by Stephanie Antonian. She's the founder and CEO of Aestora, where she writes essays exploring the moral and philosophical implications of artificial intelligence, while also helping organisations think more strategically and responsibly about how they use it.

Before turning her focus to these deeper questions, she worked as a consultant for major organisations like Google and Accenture. Her essays examine topics such as the links between AI and religion, the ways it shapes family life, and our evolving relationship with this often misunderstood technology.

Today, she's here to help me tackle the question: Is irrational AI making our decision-making worse? 

You've written about this idea of AI being irrational. Do you want to introduce us to your thoughts on that and what your central points are?

STEPHANIE ANTONIAN

I mean, it's really what it says on the tin, which is most of the technology that we are using under the banner of AI fails all of the principles of logic and rationality. And so, when we use words like intelligence, we have to remember that they do have definitions.

Humanity's biggest dilemma is not really being able to differentiate between the type of thinking that helps us or that harms us. And so, we have collectively spent millennia really defining what the best of human thinking is, and that is logic and rationality. And we have put those into rules and made them feel the discipline. And, when you follow the history of technology, you see that initially it was really aligned to the ideas of the scientific ideal of rationality, and it was really about bringing this great thinking to the world at large and scaling it.

But what happened is that with AI, we've already lived through the time where machines outsmarted the best of human thinking, when IBM's Deep Blue beat Garry Kasparov. That was more than 20 years ago. And actually, the truth is that no one cared, because intelligence isn't that profitable.

And so, what we are seeing now is a resurgence of the AI banner, but with a new technology called machine learning coming in. And that's not focused on logic any more. It's focused on what's profitable, and the financial markets are irrational. And so it's really a tech suited to that. So, we've lost that principle of logic, and we're struggling because we think it's there, but it isn't.

DAVID ELLIMAN

AI is a big umbrella and machine learning is part of that. And people in the post-LLM age talk about AI, and that's what they mean.

Do you think that this sort of route through, I guess, classifications, almost philosophical classifications of intelligence and reasoning and so forth, and as we have developed systems that model that, do you think that we've kind of lost something along the way in how we think about it? Or is it in the way that we use it?

I guess, is it the tool, or is it the human interaction with the tool that's really in question here?

STEPHANIE ANTONIAN

It's the human interaction. It's always the human interaction because we are sovereign, whether we like it or not. And it's really the stories we tell about this technology and the expectations that we set, because there's lots of really good use cases for machine learning.

Rationality isn't the only type of intelligence. There are use cases for it, but it's set within a certain bound. It is a bit like in mathematics with numbers: you have rational numbers and irrational numbers. Rational numbers can be defined, and irrational numbers just go on and on and on, like pi. And it's not that they're not real, it's just that they can't be defined.

And so, machine learning is a bit more like the irrational numbers. It can't be defined. And that's okay if you understand it. But when I speak to a lot of business leaders, they're waiting for the final number of pi to be revealed. They're investing in it thinking it's coming. And that type of thinking leads to catastrophe.

So, technology's great, technology's fine, as long as you know what it is and how to use it. But it's where people get a bit confused and put in too many hopes and aspirations that it falls apart.

DAVID ELLIMAN

I guess I was thinking as you were speaking, you've kind of got the big and the small promises. The big promise of AGI and this sort of intelligence, and I can see that all those ideas are going to be distorted.

But I guess in the small, you've got somebody who may have access to a tool that may help them do something with their job. That series of these small optimisations I think feels like it's within reach more for people. I'm a software engineer, so I'm used to talking about or using some of the tooling to write, review and understand code, for example.

There are use cases where it's like, yeah, this is fine. I can have a bounded context around this. I can establish safeguards and guardrails around the use of this thing. Whereas when the story gets bigger, we seem to lose that sense of ethical guardrails.

STEPHANIE ANTONIAN

Yeah, we lose our minds. We just become absolutely hysterical.

But there are lots of really incredible use cases for machine learning. So, it's like, though I might criticise some things about the AI industry, I'm not anti the tech. I think there are lots of really good use cases. I just don't think it does well, I know it doesn't do what people think it's going to do.

And I think when it comes to the AI conversation, it's always about the future. What's the future going to look like? Future of work, future of fun, future of childcare. Everything's about the future of. But if you want to understand the future, you've got to understand the present.

People haven't done enough work on what the actual present state of play is like. Where's the money going? How does it flow? What's the economics behind this? What would this do to the broader economy? What are the limitations of the technology? What exactly can it actually deliver? What are the outputs? Do the outputs meet the requirements? What's their accuracy rate? That is where the work needs to be done in the present and in the here and now. But it's quite difficult work.

DAVID ELLIMAN

What I would say are the main topics and trends that describe the current state of play in AI and machine learning right now? Honestly, the one that stands out for me isn't a headline grabber. It's a quieter shift. The move from asking what can this technology do to what can an organisation actually deliver with it?

It's a very different conversation and it forces you to confront the economics, the accuracy, the integration, and the accountability, and exactly what Stephanie was arguing. There's a lot of work there for organisations and they still haven't done that.

So, do I think we've got the right balance between the present and future thinking? No, I don't think we have. Willing too far into the future, and not particularly in a useful way. The present is where the engineering actually lives. It's where you measure cost, where you measure output, where you prove value.

The future is where strategy sits, but strategy has to be grounded in what you can actually observe today. Otherwise, it's just science fiction with a PowerPoint template.

Stephanie ends her essay, AI is Irrational, with a striking headline from an MIT report: in 2025, 95% of generative AI pilot programmes were reported as failing. That's contested by some people, but it's certainly a high number most people experience.

So why don't we just look at why that might happen?

STEPHANIE ANTONIAN

Machine learning is probabilistic and classical AI is more about possibility. And so I think a lot of businesses don't first start by questioning: are they looking for probability or for possibility? Because, say for example, in the world at large, the probability is that we will head for climate catastrophe, and the probability is that we will head for nuclear war.

And so probabilistic machines aren't that useful because what we need are possibilities. We need the anomalies. We need the absolute outliers. We need the black swans. We do not want the average with the past before. We want something different. We want change.

And so for a lot of businesses, unless the business is thriving organically, they're going to struggle. Because if this is a business with really tight margins, with not a lot to play with, then the probability is that it will get worse. The route that would take you to is that it will escalate that faster. And so I think a lot of businesses are expecting to see possibility from systems that can only deliver probability.

DAVID ELLIMAN

I think in some industries the probabilistic suggestion becomes a possibility in the sense that, in the financial markets, all the future prices are probabilistically assessed and are placed into the market, and they'll compete with each other and one will win out. One will be more attractive as a price for some purchaser to buy or sell.

So that prediction became the reality. So I think there are some cases where, I guess if I was to use the word, the opinion or the probability might end up driving reality in a sense. So we might see an outcome and we might end up going towards it because that's what it told us, as opposed to what the potential was.

STEPHANIE ANTONIAN

Yeah, absolutely. Absolutely. And that's what we're heading towards. And so we have to understand that machine learning works by aggregating the past and coming up with the most statistically likely answer to whatever prompt you give it.

The bigger question is, do we want to continue on this past trajectory? Because if we do, then it's fine, then it's great. But if we don't, then what are we expecting? Because it is obviously not progression, it's regression, because we're going back to the average of the past. That's where I think we don't quite understand it, or want to have that difficult conversation.

DAVID ELLIMAN

Bringing it back to some sort of use cases where people are using AI for various purposes, there's a sort of snowball. You can imagine, as this builds up with the greater and greater use of these things, the bigger the risk or the lack of accountability. How do you feel about that human-in-the-loop aspect to this?

STEPHANIE ANTONIAN

There always has to be meaningful human control because, like I was saying before, humans are sovereign, and there's nothing we can do about that. You can argue against reality, but reality is reality. And so, I think a lot of what we're doing is trying to build these systems that are going to make decisions for themselves so that there's no accountability.

But as a society, we can't actually cope with that for very long because when something goes wrong, we need someone's head to roll. We want to see blood, basically. And I think what we've seen with the court ruling on social media companies is that the time has ended where they get to pretend that they're not accountable. And so, there's always a human accountable.

Even when it's a machine, there's still the creator. There is always a human accountable, and we will always have to go back to that because we need to have a vision of justice for there to be order in society. If we are told that fundamental decisions about our lives are determined by a machine that has no accountability, we just won't put up with that.

DAVID ELLIMAN

It's interesting, isn't it? I was thinking about the moving away of personal agency from yourself. In some senses governments and large institutions sort of do that for us anyway. And now we might be moving towards an automated equivalent of that, in a sense, where the technology might be making decisions on our behalf. And I think the one worrying aspect of that might be finding out that it's being done.

The second is the transparency of not knowing how it's been done or how those decisions were arrived at. Being an engineer, I tend to think about these things in terms of building blocks.

I think if the blocks stay relatively small and have enough kind of self-contained guardrail around them, then assembling blocks of those should be safer, as long as there is some sort of interject point where there is a review and approval of people involved in that. And that's a bit like how we build software.

We build a little bit of software, we associate it with a test, and if it proves the test, that sort of pairing goes off into a production pipeline. And if something else down the track breaks the original test, then you know that you've created a side effect. So having that sort of quality-baked-in idea is almost like something we need to do with ethical frameworks around AI. The problem I have is that nobody ever defines what an ethical framework meaningfully actually is.

STEPHANIE ANTONIAN

There's so many things there. But firstly, that's because you've got a sensible approach. That's the sensible way to do it. And the reason why nobody defines what an ethical principle is because the problem with ethics is you can argue for anything. You can just debate. You just debate all day. You just debate and debate and debate. And that's ultimately why we have religions, because someone said, hey, okay, these are the rules for this one, draw a line in the sand.

And so I don't think there's an ethics issue in AI. I think there's a self-esteem problem in AI, but there's not an ethics issue in AI because ethics is just intellectual fodder until you come up with rules and beliefs and determinations. And what drove the Enlightenment and the time of rationality rising was the philosophy that all humans were equal, that all humans had this innate ability to reason and this innate worth.

And we have just totally got rid of that. And we're getting a bit confused as to why this is not going so well and what the future holds. But we are basically saying humans are irrelevant now. Actually, machines are more important, and that's the way we are going to do it. And everyone needs to suck it up. And that, of course, just won't happen.

DAVID ELLIMAN

If Stephanie is right and it's impossible to establish a set of rigid policies on AI ethics, what should we focus on?

I think the honest answer is stop arguing where ethics as a philosophy and start treating this as safety engineering because we know how to do safety engineering. Aviation does it. Medical devices do it. Civil engineering does it.

Traceability and a test regime catches all the things that would otherwise harm people. Translate that into AI. Who is accountable for this decision? Is there a human somewhere in the chain whose name is actually on the outcome? Can we audit how the system reached its output?

Do we have the equivalent of a test suite? The things the system must never do, are we running it continuously? The ethics then shows up baked into the constraints rather than hovering above everything as a philosophical problem we can never quite resolve.

If AI ethics and governance policies aren't one size fits all, what should business leaders be doing first thing Monday morning when they sit down and try to tackle these issues?

STEPHANIE ANTONIAN

So I go back to being in the present and understanding what's happening today. And so by that, I would first do a costing exercise, because we seem to have forgotten about cost benefit.

And so you can already see the amount of money that these AI companies are losing to be able to provide the tool for free and you can estimate how much that's going to cost you then in terms of the real cost of this, and then you can create the benchmark for that so that when you do experiments internally... Because there's no harm in doing experiments, you can do experiments, you don't have to be an AI-first company to use AI.

In fact, you shouldn't be. Once you do those experiments, you can actually then calculate what the actual value is compared to what the cost is and whether that's something you would want to put in as a policy. So that was a bit of a long answer, but I'd start by saying that you need to come up with some real benchmarks, some axioms of the future for which you can compare everything to.

That's the first starting point. Like, how much do I think this product is going to cost me? What do I think the capabilities of this product are going to be like today? From that then in the next five years, like, where in my organisation would this be valuable? Where does probability provide value?

Because there's a lot of areas where probabilistic models are really good, so where exactly there would that provide value? And then now when I do experiments, how much value is it? Is it worth? Should I make this a proper policy? Should I roll it out? There's a lot of very serious analysis that is required before you commit something.

Otherwise, you end up like Klarna, who's going to be AI first and then lost hundreds of millions. And you also don't want to do nothing because it's a cool tool. It's silly not to explore it, but you want to be in control and so. I think it also goes back to one of the threads of this conversation, which is on the human of the human accountability.

It's like you are a business leader because you are the business leader, not AI. You still have to stay the centre of creation and apply all the things you already know that need to be done and need to be determined to this because you are still the boss. This absolutely cannot replace you in that regard.

DAVID ELLIMAN

So Stephanie, coming back to the key question of the episode, do you think that irrational AI is making our decision making worse?

STEPHANIE ANTONIAN

It's making us not make decisions because... It's classic analysis paralysis. The more information you have, the harder it is to make a decision. We've created these tools that give us way more data, but that actually cannot make any meaningful decisions.

Like, I can't make a decision on any of the important questions, and so we've just frozen, we're just not making decisions. You know, even in the advice I'd give to business leaders like that advice was get your ducks in a row to be able to make a decision. Because we are in a paralysis now. Like, we can't see the woods from the trees. We can't move forward.

So, we are not making decisions. That's what AI is doing to us.

DAVID ELLIMAN

I always think that, you know, innovation seems to come about with some degree of real focus. You do something because you've got a very specific goal in mind and there's a certain maybe frugality to the amount of data that you need, which I think definitely supports your argument about maybe having too much data leads to indecision.

I have this impression, and I'd just love you to react to it if you would. That when you see AI being used in middle management and stuff is being sloshed about, maybe reports are, are generated quicker, maybe this, maybe that, fine if it helps people, but it's almost like, yeah, it's helping you because a lot of that stuff is probably unnecessary.

If you look at the frugality of innovation exactly about what you actually need to do, what does this business actually need, and then you unravel everything that actually happens within an organisation, within its political and power structures... You wonder what's actually necessary, you know, to ship the product or whatever it is that they're doing.

STEPHANIE ANTONIAN

Exactly. I think what we forget is that AI is not a tool for simplicity. It's an incredibly complex tool, like machine learning is incredibly complex and it adds a layer of complexity to everything that you're doing. Even take the grant making process, you know, you use AI to help you write your grant applications, but then so does everybody else. So then you need to come up with another person who's going to make it a bit more unique.

And so it's like you just, what you've done, you just made a complex process where the problem was that it's overly complex. Even more complicated. And it was just like newer and newer levels of abstraction.

But we already know everything we need to know. ChatGPT can give me answers to loads of questions that are actually irrelevant to my life. But if there's anything deeply relevant to my life, I actually already know the answers. I know how to have my perfect figure. I just don't know why I don't do it.

Like, you know, we have enough food to end hunger. We have enough money to end poverty. We have the tech to solve the climate crisis. It's a tree, but we don't want to do it. And that's the bigger problem.

So, bringing it back to business where I speak to some people who asked me, how do I bring in AI into my business? And I was like, well, your fundamental challenge, especially for consumer goods, is that the consumer is getting poorer and poorer and they can't afford your products.

So how's AI going to fix that? It's not. I think what you are saying is like a lot of people just being like, ‘Oh, to be honest, I don't want to do the work. I don't want to have to deal with the bigger issues and the bigger reality. And so it's just easier for me to just jump on AI.’

DAVID ELLIMAN

Thank you for listening to Tech Tomorrow, brought to you by Zühlke. If you'd like to learn more about what we do, you can find links to our website and more resources in this episode's show notes. Until next time.