CalypsoAI has accelerated a global financial firm’s use of AI models from months to days by modernizing its model risk management (MRM) process using VESPR, our flagship software solution. In this case study, we discuss modern MRM solutions for financial services.
Recently, a customer came to us with an interesting observation. “My model risk management (MRM) teams and tools are stuck in the past,” she said. “They are not prepared to assess machine learning and AI models”.
This sentiment is not unique to this firm. In fact, many of our commercial clients observe that their teams lack the ability to quantitatively asses the quality and compliance of AI models. But for this client in particular, the lack of tooling was slowing their ability to use a model down so much that it almost made their data science investments meaningless. “It takes us 6 weeks to even get an initial set of questions,” our client informed us. “After that, we are looking at 4-6 months of back and forth before something can be used in production.
The inability to rapidly deploy models has sadly become commonplace with our interactions with clients. As organizations are expanding their use of AI, they are running into the underlying risks of the technology.
Within the financial services industry, business operations as disparate as cybersecurity, fraud, risk management, customer trading recommendations, and customer portfolio rebalancing are all being transformed by AI. In fact, 85% of financial services businesses believe AI will allow their companies to obtain or sustain a competitive advantage, due in no part to the promises that come from AI implementation, speed, precision, scale, and new insights. We believe their predictions to not go far enough. in the winner-take-all environment of AI, it is possible that AI will not only be a financial services competitive differentiator, but it will also be a requirement to a firm’s survival.
But the acceleration of these technologies is not without risk. AI systems can fail, they are extremely difficult to understand and explain, they can be difficult to validate, and they can be hacked. Companies implementing AI systems face validation, monitoring, and security challenges, creating uncertainty when considering AI adoption and deployment. This is especially true in heavily regulated industries such as financial services. These challenges are preventing the adoption and benefits of AI because organizations lack AI model risk management processes and tools to validate model performance, define / score model risk, and monitor AI systems for ongoing risks, such as model drift.
Organizations are aware of these risks but lack the tooling to solve them. For our client, this meant going months between when a model was built to when it was finally deployed. Such lags in data science effort create massive inefficiencies, not to mention damaged morale to data scientists leading to increased turnover.
In order to best mitigate the risks of AI systems, organizations require 3rd party validation, testing, and model risk management capabilities to sit alongside their AI development process. CalypsoAI is not alone in our assessment (we sell these solutions, so we are a bit biased). Regulators (e.g. FDIC), legacy technology companies (e.g. IBM) and technology experts (e.g. MIT Tech), and even the enterprise AI platforms themselves (e.g. – Data Robot), all recommend the use of 3rd party validation of AI models. As expected, governments are implementing compliance standards for the use of AI, as well as for the rules that AI must live by.
Calypso was able to successfully work along two separate paths this our client. First, we took our clients models and used Vespr internally to validate their performance, providing written and verbal reports to their model risk and global compliance teams. At the same time, we configured Vespr to serve as an ongoing model risk management solution for ongoing validation, compliance, and active security monitoring.
As a result, our client brought their deployment of AI models down from months to days, without sacrificing quality or compliance. Get in touch with us to learn more about CalypsoAI’s Modern MRM solutions for financial services here.
