Illustration: Natalie Peeples/Axios
The leaders of America's largest banks are convinced that AI will transform their businesses, but they're mostly struggling to implement more than a handful of AI products, even internally, and can't agree on what counts as a win.
Why it matters: The massive banking industry is a crucial testbed for transforming the AI industry's promises into real-world results.
Driving the news: A small group of banks is pulling away from the majority on AI, thanks to frequent experimentation and investment, per new research from Evident Insights, an AI benchmarking and intelligence platform.
- JPMorgan Chase, Capital One and Royal Bank of Canada are "ahead" of their competitors and "accelerating away from the pack," per Evident's latest report (PDF).
- "Successful banks continuously reinvent themselves" through innovation, Evident co-founder and CEO Alexandra Mousavizadeh told Axios.
JPMorgan Chase CEO Jamie Dimon, in his annual letter to shareholders April 8, said he is "completely convinced" the consequences of AI will be "extraordinary."
- JPMorgan Chase now employs more than 2,000 AI specialists.
- The bank also dominates AI research, per Evident's report, with 45% of all papers published by banks in 2023, while Bank of America and Capital One account for 67% of all AI-focused patent filings.
Yes, but: Research is only one piece of the AI puzzle for banks.
Case in point: Jeff McMillan, the new head of firmwide AI at Morgan Stanley, which ranks 17th overall in Evident's index, tells Axios that relationships and staff training matter just as much.
- McMillan says he and his team have been talking to senior OpenAI engineers and executives "weekly, if not daily" since early 2022, applying those insights into a new AI assistant used by the bank's advisors.
- "To have direct connectivity to the senior leaders of Open AI — the people who invented this — I can't tell you what an advantage that has been," he says.
- The partnership is mutually beneficial: OpenAI gets the stamp of approval of an 88-year-old bank that operates in a strict regulatory environment.
- But McMillan warns companies not to focus too much on which AI models they use: "You will not fail because your LLMs don't do what you need. You're going to fail because you don't educate and empower your employee base and you don't put the right controls in place."
Reality check: McMillan, like other AI leaders at banks, is delivering only a handful of AI tools — which suggests no bank has figured out how to use AI anywhere near its potential.
- "We've deployed two use cases. But what about a world where we might have 100,000?" he says.
How it works: Morgan Stanley's AI assistant is built on the bank's internal data, sorted to separate what can only be shared internally and what clients can see. Answers to prompts come with supporting citations.
- Axios received a demonstration of the tool and spoke to financial adviser Patrick Biggs, one of its frequent users, who described how it helps him give advice to clients around the world on topics — like minimizing taxes when getting married — where rules vary by country.
- Biggs uses the assistant "several times a day" and says it has "improved dramatically over last 12 months."
- He feels the biggest win is having the expertise of 80,000 staff at his fingertips without having to track them down via email or phone.
- In the past, Biggs would have to tell a client he'd research answers he didn't know on the spot. "Now I say, 'Just give me 15 seconds,'" because the tool provides real-time information in a shareable format, he says.
It took months of fine-tuning to get the assistant to this standard, McMillan says: "We went through over 20,000 pieces of feedback" from advisers saying they didn't get what they needed from it.
- "This idea that you can take 100,000 documents, throw them into a system and somehow expect generative AI to make sense of that: It's a fantasy."
- Content often needs retagging, he says, and retrieval-augmented generation — which helps chatbots base answers on real data and avoid making things up — is a key ingredient for getting "highly accurate, consistent answers."
What they're saying: "There's not a single thing that we have produced or probably will produce for some time that is not going to go to a human being for ultimate final checking," McMillan tells Axios.
What's next: Morgan Stanley will go live with a product called Debrief before summer, McMillan said. It allows advisers to record client calls (with consent) to create summaries and recommended actions.
Editor's note: This story has been corrected to show Dimon's annual letter was posted April 8 (not April 9) and to say that Morgan Stanley is 88 years old (not 89).