Cookie Consent
Hi, this website uses essential cookies to ensure its proper operation and tracking cookies to understand how you interact with it. The latter will be set only after consent.
Read our Privacy Policy
Back

The computer vision bias trilogy: Shortcut learning.

Nobel Prize-winning economist, Daniel Kahneman once remarked “by their very nature, heuristic shortcuts will produce biases, and that is true for both humans and artificial intelligence, but their heuristics of AI are not necessarily the human ones”. This is certainly the case when we talk about “shortcut learning”.

Lakera Team
December 1, 2023
March 29, 2022
Learn how to protect against the most common LLM vulnerabilities

Download this guide to delve into the most common LLM security risks and ways to mitigate them.

In-context learning

As users increasingly rely on Large Language Models (LLMs) to accomplish their daily tasks, their concerns about the potential leakage of private data by these models have surged.

[Provide the input text here]

[Provide the input text here]

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Lorem ipsum dolor sit amet, Q: I had 10 cookies. I ate 2 of them, and then I gave 5 of them to my friend. My grandma gave me another 2boxes of cookies, with 2 cookies inside each box. How many cookies do I have now?

Title italic

A: At the beginning there was 10 cookies, then 2 of them were eaten, so 8 cookies were left. Then 5 cookieswere given toa friend, so 3 cookies were left. 3 cookies + 2 boxes of 2 cookies (4 cookies) = 7 cookies. Youhave 7 cookies.

English to French Translation:

Q: A bartender had 20 pints. One customer has broken one pint, another has broken 5 pints. A bartender boughtthree boxes, 4 pints in each. How many pints does bartender have now?

Lorem ipsum dolor sit amet, line first
line second
line third

Lorem ipsum dolor sit amet, Q: I had 10 cookies. I ate 2 of them, and then I gave 5 of them to my friend. My grandma gave me another 2boxes of cookies, with 2 cookies inside each box. How many cookies do I have now?

Title italic Title italicTitle italicTitle italicTitle italicTitle italicTitle italic

A: At the beginning there was 10 cookies, then 2 of them were eaten, so 8 cookies were left. Then 5 cookieswere given toa friend, so 3 cookies were left. 3 cookies + 2 boxes of 2 cookies (4 cookies) = 7 cookies. Youhave 7 cookies.

English to French Translation:

Q: A bartender had 20 pints. One customer has broken one pint, another has broken 5 pints. A bartender boughtthree boxes, 4 pints in each. How many pints does bartender have now?

Hide table of contents
Show table of contents

Nobel Prize-winning economist, Daniel Kahneman once remarked:

“By their very nature, heuristic shortcuts will produce biases, and that is true for both humans and artificial intelligence, but their heuristics of AI are not necessarily the human ones”. This is certainly the case when we talk about “shortcut learning”.

Despite careful testing on the data side, model bias can reveal itself more directly in what the computer vision system learns. This issue of a computer vision model using the wrong visual features for prediction is referred to as shortcut learning.

Looking in the wrong places.

The black-box nature of many computer vision models renders such shortcuts difficult to find, and as a result, trained models tend not to generalize well to unknown environments. In the paper Recognition in Terra Incognita, Caltech researchers showcase a classification model that does well at finding cows on an evaluation set but fails when asked to classify cows by the beach or other unusual environments. For a computer vision models, visual features indicating grass and mountains may contribute to detecting a cow in the image, while beach or indoor features may heavily weigh against it. It is expected that the model uses such features, but their impact should be understood before deploying such models in production. A company building a cow detector unaware of this fact would disappoint some coastal clients, creating reputational risk.

Cows on a beach, shortcut learning. This cow is in the mountains and  the AI can identify her.
This image of a cow on a beach. Illustrating shortcut learning and how an artificial intelligence might be looking at the wrong characteristics .
The environments frequent in the training data can create biases and may lead to invalid predictions when the test image is found in completely different settings.

How to detect shortcuts.

In this paper, the authors show that face detection benchmarks achieve above-random performance even after removing the hair, face, and clothes of subjects. This indicates that irrelevant background features are being used for prediction. Another piece of research identifies an initial list of such biases that can appear in practice for medical applications. Similar ablation experiments, where the parts of the image relevant for prediction are masked out, can be useful in identifying such shortcuts. Metadata can be a powerful tool to detect and test for some of these shortcuts as well. Statistical dependence between metadata dimensions and the performance of the model can surface concerning shortcuts: if the demographic of a patient is highly correlated with performance then further investigation is needed!

To summarize, shortcut learning happens when your computer vision system is looking at the wrong visual features to make predictions. Such shortcuts can be detected from image data alone, for instance, by measuring reasonable performance despite masking out the regions of the image that matter for prediction. They can also be detected by referring back to your metadata: if there is a strong link between metadata parameters and the performance of the model, then it’s worth taking a closer look. Having practices in place during the machine learning model evaluation process to detect these shortcuts is key to a high-performing model.

Get started with Lakera today.

Get in touch with mateo@lakera.ai to find out more about what Lakera can do for your team, or get started right away.

Lakera LLM Security Playbook
Learn how to protect against the most common LLM vulnerabilities

Download this guide to delve into the most common LLM security risks and ways to mitigate them.

Unlock Free AI Security Guide.

Discover risks and solutions with the Lakera LLM Security Playbook.

Download Free

Master Prompt Injection Attacks.

Learn LLM security, attack strategies, and protection tools. Includes bonus datasets.

Unlock Free Guide

Learn AI Security Basics.

Join our 10-lesson course on core concepts and issues in AI security.

Enroll Now

Optimize LLM Security Solutions.

Use our checklist to evaluate and select the best LLM security tools for your enterprise.

Download Free

Uncover LLM Vulnerabilities.

Explore real-world LLM exploits, case studies, and mitigation strategies with Lakera.

Download Free

Understand AI Security Basics.

Get Lakera's AI Security Guide for an overview of threats and protection strategies.

Download Free

Explore AI Regulations.

Compare the EU AI Act and the White House’s AI Bill of Rights.

Download Free
Lakera Team
Read LLM Security Playbook

Learn about the most common LLM threats and how to prevent them.

Download
You might be interested
min read
Computer Vision

Not All mAPs are Equal and How to Test Model Robustness

Model selection is a fundamental challenge for teams deploying to production: how do you choose the model that is most likely to generalize to an ever-changing world?
Mateo Rojas-Carulla
December 1, 2023
min read
Computer Vision

Case study: How Privately accelerated computer vision certification with Lakera.

Case Study: Find out how Privately was able to increase their SDO performance measures, such as 80% reduction in real-world failures and 10x faster development cycles — from roughly 2 weeks to 2 days.
David Haber
December 1, 2023
Activate
untouchable mode.
Get started for free.

Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.

Join our Slack Community.

Several people are typing about AI/ML security. 
Come join us and 1000+ others in a chat that’s thoroughly SFW.