Building more and more AI-dependent security features into AI models’ operations is a good thing for everyone who will use them. Relying on the companies developing GenAI systems like ChatGPT, Gemini, BloombergGPT, and others to conduct thorough red-teaming or, ideally, purple-teaming engagements to test those tools against potential threats is crucial. However, it is only the beginning. Cybercriminals have their own versions of red-teaming, table-top exercises, and planning activities.
Significant players in cybercrime have followed the standard path for large businesses, merging, acquiring, vertically integrating, and establishing organizational structures with teams reporting to top and mid-level management in operations and other critical business units. Their business model is “cybercrime as a service” and they actively market products, such as malicious code. Jailbreak attacks are just one type of weapon in their arsenal, but it is an especially insidious one, as the code used in the attack can be both the crime and the perpetrator.
Successful jailbreak attacks via large language models (LLMs) in the financial sector can do more than just override system safeguards. In the hands of a highly skilled hacker, a compromised model could allow:
- Access to and control of critical financial systems or proprietary source code
- Execution of arbitrary commands to modify machine learning (ML) algorithms, systems, or other functionalities
- Escalation or downgrade of privileges to gain more power and root access to internal and confidential systems
- Introduction of malware
- Spying on network activity
- Access to or extraction of private or sensitive data, such as customer records, employee data, legal documents, business strategies, and intellectual property
- General chaos within the targeted institution’s ecosystem
These issues can result in long-term consequences, including substantial financial losses, regulatory fines for failing to adequately secure personal data, damage to the organization’s reputation and competitive advantage, and legal repercussions. Proactively deploying safeguards that address these concerns is the only reasonable approach to creating a secure environment for financial and other organizations to implement LLMs at scale and across the enterprise.
CalypsoAI’s SaaS-enabled security and enablement platform applies a rigorous review of prompts being sent, scanning for semantic patterns and categories, such as role-playing, reverse psychology, hypothetical situations, or world-building, which indicate a user is instructing the LLM to ignore or override system controls, and for content misaligned with user-established criteria based on the organization’s acceptable use policy and values. It stops prompts that instruct an LLM to execute actions resulting in dangerous, illegal, unethical, immoral, or amoral content from leaving the organization’s system.
Our platform is the first-of-its-kind tool that can provide the peace of mind decision-makers need to greenlight the safe, secure, and ethical use of LLMs across the financial enterprise.
Click here to schedule a demonstration of our GenAI security and enablement platform.
Try our product for free here.
Going to RSA? Click here to book a meeting with us.
