Assessing GenAI Security Solutions in the Wild with PINT
Join Václav Volhejn (Senior ML Scientist at Lakera) and Julia Bazińska (ML Engineer at Lakera) for a live webinar discussion exploring the newly released PINT (Prompt Injection Test) benchmark.
Goodhart’s law states that when a measure becomes a target, it ceases to be a good measure. This is an acute issue in the prompt injection defense field where all public benchmarks have been overfit on to the point of becoming uninformative: having a 99% accuracy on a Hugging Face prompt injection dataset does not translate into being an effective defense in practice.
Our new PINT benchmark was designed to combat this issue, and includes a mechanism to create updated versions of the dataset if it too starts being affected by Goodhart’s law.
It was designed to provide a neutral way to evaluate the performance of prompt defense solutions, without relying on known public datasets. It is a critical piece to ensure that prompt defense solutions can be evaluated objectively and translate to operational attacks (rather than being optimized purely to perform on public benchmarks).
This webinar is a must-attend for anyone looking to learn more about how to benchmark AI security solutions and how to contribute to a more secure GenAI future.
- Learn the origin story of PINT: Discover why we developed a new benchmark for evaluating prompt injection detection systems and why it's open-source.
- Understand what PINT is & how to use it: Gain insight into its design, methodology, and how to effectively use it for evaluating prompt injection detection systems.
- Explore PINT use cases: See the examples of models evaluated using PINT.
- Learn how you can contribute to PINT: Find out how you can play a role in the ongoing development of PINT and contribute to a more secure future for Generative AI.
%201.webp)
Assessing GenAI Security Solutions in the Wild with PINT
Join Václav Volhejn (Senior ML Scientist at Lakera) and Julia Bazińska (ML Engineer at Lakera) for a live webinar discussion exploring the newly released PINT (Prompt Injection Test) benchmark.
Goodhart’s law states that when a measure becomes a target, it ceases to be a good measure. This is an acute issue in the prompt injection defense field where all public benchmarks have been overfit on to the point of becoming uninformative: having a 99% accuracy on a Hugging Face prompt injection dataset does not translate into being an effective defense in practice.
Our new PINT benchmark was designed to combat this issue, and includes a mechanism to create updated versions of the dataset if it too starts being affected by Goodhart’s law.
It was designed to provide a neutral way to evaluate the performance of prompt defense solutions, without relying on known public datasets. It is a critical piece to ensure that prompt defense solutions can be evaluated objectively and translate to operational attacks (rather than being optimized purely to perform on public benchmarks).
This webinar is a must-attend for anyone looking to learn more about how to benchmark AI security solutions and how to contribute to a more secure GenAI future.
- Learn the origin story of PINT: Discover why we developed a new benchmark for evaluating prompt injection detection systems and why it's open-source.
- Understand what PINT is & how to use it: Gain insight into its design, methodology, and how to effectively use it for evaluating prompt injection detection systems.
- Explore PINT use cases: See the examples of models evaluated using PINT.
- Learn how you can contribute to PINT: Find out how you can play a role in the ongoing development of PINT and contribute to a more secure future for Generative AI.
untouchable mode.
Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.
Several people are typing about AI/ML security. Come join us and 1000+ others in a chat that’s thoroughly SFW.
