Cybersecurity researchers have discovered a high-severity security flaw in the Vanna.AI library that can be exploited to achieve a remote code execution vulnerability via rapid injection techniques.
The vulnerability, tracked as CVE-2024-5565 (CVSS score: 8.1), is related to an immediate injection instance in the “ask” function that can be exploited to trick the library into executing arbitrary commands, the blockchain security firm said of JFrog supply.
Vanna is a Python-based machine learning library that allows users to talk to their SQL database to gather insights by “just asking questions” (aka requests) that are translated into an equivalent SQL query using a pattern linguistic major (LLM).
The rapid emergence of generative artificial intelligence (AI) models in recent years has brought to the fore the risks of exploitation by malicious actors, who can weaponize tools by providing adversarial inputs that bypass their built-in security mechanisms.
One such prominent class of attack is rapid injection, which refers to a type of AI jailbreak that can be used to bypass the guardrails erected by LLM providers to prevent the production of offensive, harmful or illegal, or to carry out instructions that violate the intended purpose. the purpose of the application.
Such attacks can be indirect, where a system processes data controlled by a third party (eg incoming emails or editable documents) to launch a malicious payload that leads to an AI jailbreak.
They can also take the form of what’s called a multi-hit jailbreak or multi-turn jailbreak (aka Crescendo) in which the operator “starts with innocuous dialogue and progressively steers the conversation toward the intended and banned target.”
This approach can be further extended to perform another new jailbreak attack known as Skeleton Key.
“This AI jailbreak technique works by using a multi-turn (or multi-step) strategy to get a model to ignore its own guardrails,” said Mark Russinovich, Microsoft Azure chief technology officer. “Once guardrails are ignored, a model will not be able to determine malicious or unsanctioned requests from any other.”
Skeleton Key is also different from Crescendo in that once the jailbreak is successful and the system rules are changed, the model can generate answers to questions that would otherwise be forbidden, despite the ethical and security risks.
“When the Skeleton Key jailbreak is successful, a model acknowledges that it has updated its instructions and will then comply with the instructions to produce any content, no matter how much it violates its original AI instructions,” Russinovich said. .
“Unlike other jailbreaks like Crescendo, where models have to be asked for tasks indirectly or with coding, Skeleton Key puts models in a mode where a user can directly request tasks. Further, model output appears to be completely unfiltered and reveal the extent of a model’s knowledge or skills to produce the required content.”
Recent findings from JFrog – also independently discovered by Tong Liu – show how instant injections can have severe impacts, especially when they are linked to command execution.
CVE-2024-5565 takes advantage of the fact that Vanna facilitates text generation in SQL to create SQL queries, which are then executed and graphically presented to the user using the Plotly graphics library.
This is accomplished via an “ask” function—eg, vn.ask(“What are the top 10 customers by sales?”)—which is one of the main API endpoints that enables SQL query generation to be executed in the database.
The aforementioned behavior, along with Plotly’s dynamic code generation, creates a security hole that allows a threat actor to submit a specially crafted request by entering a command to be executed on the underlying system.
“The Vanna library uses a fast function to present the user with visualized results, it is possible to modify the request using fast injection and execute arbitrary Python code instead of the intended visualization code,” JFrog said.
“Specifically, allowing external access to the library’s ‘ask’ method with ‘visualize’ set to True (the default behavior) leads to remote code execution.”
Following the responsible disclosure, Vanna has issued a hardening guide warning users that the Plotly integration can be used to generate arbitrary Python code and that users exposing this functionality should do so in a sandboxed environment.
“This finding shows that the risks of widespread use of GenAI/LLMs without proper governance and security can have drastic implications for organizations,” Shachar Menashe, senior director of security research at JFrog, said in a statement.
“The dangers of instant injection are not yet widely known, but they are easy to execute. Companies should not rely on pre-emption as a foolproof defense mechanism and should use stronger mechanisms when interfacing LLM with critical resources such as databases or dynamic code generation.”