Solving the data challenge: a marketplace of endless possibilities

In the first post of his series, guest blogger Fernando Lucini examines how retail banks can realise the potential of their data and the role cloud has to play.

It’s no secret that retail banks struggle to realise the potential of their data. Long-standing legacy infrastructures and growth through M&A make IT complexity a fact of life. Add multiple customer accounts with separate systems of record, limited coordination between siloes and oceans of structured and unstructured data, and the enormity of the challenge is obvious.

In this blog series, I’ll look at how retail banks can make the leap to realising the promise of data and AI solutions at scale. I’m not pretending it’s at all straightforward. Quite the opposite. But it’s essential if all the hard work currently being put into pilots and proofs of concept is to bear fruit.

Technology’s evolved to the point where it’s economically possible to scale AI and data solutions across the enterprise, spanning business units and thousands of employees. That way, it’s possible to transform how people and smart machines collaborate—and the results they can achieve.

First, a history lesson. There’s been a steady evolution in how banks manage and exploit data. When Hadoop arrived, it offered a comparatively cheap way to store and use siloed data in one place. But it’s proved ineffective as a dumping ground. As banks discovered, Hadoop can’t solve the challenge of dirty, unprepared data. And although it’s good at certain tasks, it’s no Swiss army knife. Hadoop can’t do everything on the same data: if data’s used for reporting, one use case is needed. And that’ll be completely different from what’s required for analytics.

In response, banks have introduced more governance around how they exploit their data. That’s a positive development. Often, where Hadoop’s being used, it’s not actually getting data from the business. The issues involved in extracting data from a system of record and landing it in Hadoop have gotten in the way.

If data’s going to be accessible to other teams for other tasks, it must be tagged so it’s easily identified. Complicating matters further, depending on its planned use, data will have to be stored in different ways. And of course, all of this must happen at lightning speed. There needs to be absolute clarity over who has authority to use data, and for what purpose. If data’s being enriched, it should be stored in a hybrid data hub, not in Hadoop. And so on.

As the cloud’s become more trusted, there’s been a shift to leverage what it has to offer—through hybrid on-premise/cloud approaches, or via all-out cloud migrations. It makes sense. The cloud provides infinitely more flexibility and scalability for data. Storage is cheap, there’s no need for hardware, and it’s easy to turn nodes on and off.

Now we’ve reached a fork in the road: whether or not to go cloud-native. In other words, should banks use cloud providers’ own applications, or install their own software in the cloud? Although AWS, Microsoft, Google and others offer excellent proprietary applications, many banks distrust their maturity. Increasingly, however, that’s looking like a miscalculation. By moving their own software into the cloud, these banks are missing out on huge benefits in terms of speed, automation and scalability that cloud-native approaches provide.

The cloud solves a lot of the data challenges that banks have grappled with—but it also raises some new ones. There’s mounting pressure from regulators to prevent banks storing all their data with one vendor. But there’s no advice on where to direct data. The speed of public-cloud evolution also creates problems. With vendors offering multiple new features, it’s hard for banks to keep up. They’re unaccustomed to real-time decision-making, and used to planning longer term.

To develop intelligent products—like optical character recognition (OCR)—combine them with data, and launch them at speed, banks need a completely new way of working. For now, it makes more sense to go to AWS Marketplace, for instance, and pull out the product they need. And provided they’re sourced from the same vendor, they know these products will slot together and scale rapidly.

Hadoop worked well for focused use cases. But it couldn’t be all things to all people. Now banks are waking up to the potential of a marketplace with endless possibilities. It doesn’t solve the data-curation challenge (banks still spend 80 percent of the time curating data vs 20 percent exploiting it). But the cloud does provide a great incentive to re-examine data and decide what should be stored on-premise and what should be migrated.

Next time I’ll introduce the four key areas banks need to focus on in their journey to becoming intelligent businesses.