Will my system work if image quality starts to drop significantly? If my system works at a given occlusion level, how much stronger can occlusion get before the system starts to underperform? I have faced such issues repeatedly in the past, all related to an overarching question: How robust is my model and when does it break?
As developers, we often face daunting questions as we build a model toward production. For instance:
I have faced such issues repeatedly in the past, all related to an overarching question: How robust is my model and when does it break?
We all know that building production-level computer vision systems is challenging. The number of factors that can influence input images or videos in the real world is daunting. As such, gathering evidence that the systems we built will work under these varying factors is key and a major challenge for that final push to production.
Ultimately, we as developers carry the burden of proof.
A major tool at our disposal to bring transparency to how a computer vision system will perform in the real world and provide answers to the questions above is stress-testing. Systematically answering such questions (for example, how occluded do objects have to be before my systems break) provides two major advantages:
1. If the system’s robustness is unacceptably low (for example, the system is highly affected by increased occlusion), gathering the right data to increase robustness becomes easier to strategize (by prioritizing images with highly occluded objects). This includes generating the right synthetic data and augmentations.
2. It helps communicate to the product’s users its limitations, ultimately ensuring transparency and safer use.
The fundamental insight relies on the idea of metamorphic relations. Some visual changes result in clear, predictable effects in the image’s annotations. For example, a rotated dog is still a dog, or a horizontal flip on a gauge reading should result in a symmetric reading. This allows us to measure performance while exploring such visual changes using techniques from the fuzz testing world. To learn more about these, take a look at our blog post on metamorphic relations and fuzz testing.
Stress-testing is a key functionality of Lakera’s MLTest. Our stress-testing capabilities push the boundary and bring you answers to the questions discussed above. If you are interested to see what MLTest has to say about the breaking points of your systems, do reach out!
Download this guide to delve into the most common LLM security risks and ways to mitigate them.
Get the first-of-its-kind report on how organizations are preparing for GenAI-specific threats.
Compare the EU AI Act and the White House’s AI Bill of Rights.
Get Lakera's AI Security Guide for an overview of threats and protection strategies.
Explore real-world LLM exploits, case studies, and mitigation strategies with Lakera.
Use our checklist to evaluate and select the best LLM security tools for your enterprise.
Discover risks and solutions with the Lakera LLM Security Playbook.
Discover risks and solutions with the Lakera LLM Security Playbook.
Subscribe to our newsletter to get the recent updates on Lakera product and other news in the AI LLM world. Be sure you’re on track!
Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.
Several people are typing about AI/ML security. Come join us and 1000+ others in a chat that’s thoroughly SFW.
