Why testing should be at the core of machine learning development.

AI (artificial intelligence) is capable of helping the world scale solutions to our biggest challenges but if you haven’t experienced or heard about AI’s mishaps then you’ve been living under a rock. Coded bias, unreliable hospital systems and dangerous robots have littered headlines over the past few years.

Lakera Team
December 1, 2023
January 18, 2022

We now live in a world where AI (artificial intelligence) is used in mission-critical systems yet still developed like consumer technology.
As ML (machine learning) engineers, all we want is to build our systems to work without harm to others or getting stuck on a merry-go-round of prototyping. So, how can we get there?

We can take inspiration from traditional software engineering practices!

Why? Well, do you remember the last time your online stock broker purchased the wrong shares? Or Twitter failed to retrieve your latest tweets? I don’t. Major software malfunctioning is so unexpected that when it happens – it makes headline news.

How has software become so reliable?

As C. A. R. Hoare’s classic 1996 article “How did software get so reliable without proof?” points out, the answer is likely around rigorous development processes, continuous improvement of existing software, and extensive testing.

Traditional software goes through well-defined testing and release processes.

Software engineers are more than familiar with concepts such as:

– Test-driven development

– Unit tests

– Regression tests

– Integration tests

Tests are a part of CI/CD (continuous integration/continuous development) pipelines. Engineers don’t merge code unless all tests have passed. By the time they go to production, they are confident that the software works as expected. They follow a “test-to-ship” strategy.

Software has become so reliable through development processes, continuous improvement of existing software, and extensive testing.

What is our strategy when it comes to ML-driven software? As it turns out, things look a bit different.

Let’s take a look at how we typically develop ML systems today. It’s common practice to split our dataset into training, validation, and testing subsets. The first two become part of the model training loop, whereas the testing subset is used separately, outside of the training loop, to assess performance on unseen data. A typical evaluation strategy would include calculating various metrics over these data subsets and using them as an indication of real-world system performance.

It turns out that this strategy is often insufficient. Many teams find that their ML systems end up performing ‘well enough’ on their carefully selected datasets but are too brittle to be used in the real world.

At the same time, creating more complete quantitative testing and release processes is often seen as too time-consuming, especially within smaller teams. We have observed many who instead spend a lot of time on qualitative testing – which tends to fall short of constructing a thorough understanding of performance. As a result, computer vision development follows a “ship-to-test’” strategy.

This graphic illustrates the development process of computer vision systems. Collect data, train, evaluate, release, deploy and then find vulnerabilities.
Computer vision development mostly follows a “ship-to-test” strategy. This impacts customer experience at best and ends in fatal accidents at worst.

The fact that ML systems are only really tested during operation has obvious and major implications. These systems tend to operate with significant risk as vulnerabilities only tend to surface during operation: pedestrians are not detected at night, COVID diagnostics are fundamentally flawed, or systems exhibit undesired biases.  At best, this leads to low customer satisfaction or products that never make it in the market, and at worst, it puts people and society at risk.

The good news is that we can bring some of the concepts from traditional software development to ML development. We need to ensure that vulnerabilities are found during development. So, we need to bring back ‘test-to-ship’.

Lakera’s MLTest provides the quality gate which automatically surfaces vulnerabilities as part of existing processes.

Putting systematic testing at the core of your development processes is a great way to build better AI products faster. Our ML testing series provides a few simple strategies that any development team can use to prevent failure during operation.

Lakera’s MLTest equips every computer vision development team with a world-class testing infrastructure. Our product finds critical vulnerabilities and flaws in computer vision systems–automatically as part of existing development processes and before they can impact operation. We want to enable every team, small to large, to ship AI products quickly and reliably. Get in touch to schedule a demo!

Lakera LLM Security Playbook
Learn how to protect against the most common LLM vulnerabilities

Download this guide to delve into the most common LLM security risks and ways to mitigate them.

Lakera Team
Read LLM Security Playbook
Learn about the most common LLM threats and how to prevent them.
You might be interested
min read
Machine Learning

Generative AI: An In-Depth Introduction

Explore the latest in Generative AI, including groundbreaking advances in image and text creation, neural networks, and the impact of technologies like GANs, LLMs, and more on various industries and future applications.
Deval Shah
December 1, 2023
min read
Machine Learning

Test machine learning the right way: Detecting data bugs.

In this second instance of the testing blog series, we deep dive into data bugs: what do they look like, and how can you use specification and testing to ensure you have the right data for the job?
Mateo Rojas-Carulla
December 1, 2023
untouchable mode.
Get started for free.

Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.

Join our Slack Community.

Several people are typing about AI/ML security. 
Come join us and 1000+ others in a chat that’s thoroughly SFW.