Cookie Consent
Hi, this website uses essential cookies to ensure its proper operation and tracking cookies to understand how you interact with it. The latter will be set only after consent.
Read our Privacy Policy
Back

OpenAI’s CLIP in production

We have released an implementation of OpenAI’s CLIP model that completely removes the need for PyTorch, enabling you to quickly and seamlessly install this fantastic model in production and even possibly on edge devices.

Daniel Timbrell
October 20, 2023
Last updated: 
November 13, 2024
On this page
Table of Contents
Hide table of contents
Show table of contents

Deploying state-of-the-art machine learning models can often lead to a myriad of issues due to the dependencies of the more salient packages - most commonly PyTorch and TensorFlow. At Lakera, we have released an implementation of OpenAI’s CLIP model that completely removes the need for PyTorch, enabling you to quickly and seamlessly install this fantastic model in production and on edge devices.

Source: OpenAI Clip Architecture

CLIP (Contrastive Language-Image Pre-Training) is powering some of the most exciting image to text applications out there right now. It’s a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. There are three main components that comprise this model:

  1. The text tokeniser, which converts the given natural language into tokens (embeddings).
  2. The image preprocessor, which converts the given image into embeddings.
  3. The CLIP model itself, which outputs the cosine similarities of the text and image embeddings generated above.

The main issue we have found is that all three of these pieces utilise PyTorch - so we decided to simplify things for you!

We achieved this with the following steps:

  1. The text tokeniser was rewritten in NumPy.
  2. We wrote our own image preprocessor, which mimics the functionality of CLIP’s preprocessor.
  3. We exported the CLIP model to an .onnx format, meaning that we have essentially swapped the PyTorch dependency for the lightweight onnxruntime.

Try it out! Don’t forget to give it a star and reach out if you have any feedback!

Lakera LLM Security Playbook
Learn how to protect against the most common LLM vulnerabilities

Download this guide to delve into the most common LLM security risks and ways to mitigate them.

Unlock Free AI Security Guide.

Discover risks and solutions with the Lakera LLM Security Playbook.

Download Free

Explore Prompt Injection Attacks.

Learn LLM security, attack strategies, and protection tools. Includes bonus datasets.

Unlock Free Guide

Learn AI Security Basics.

Join our 10-lesson course on core concepts and issues in AI security.

Enroll Now

Evaluate LLM Security Solutions.

Use our checklist to evaluate and select the best LLM security tools for your enterprise.

Download Free

Uncover LLM Vulnerabilities.

Explore real-world LLM exploits, case studies, and mitigation strategies with Lakera.

Download Free

The CISO's Guide to AI Security

Get Lakera's AI Security Guide for an overview of threats and protection strategies.

Download Free

Explore AI Regulations.

Compare the EU AI Act and the White House’s AI Bill of Rights.

Download Free
Daniel Timbrell

GenAI Security Preparedness
Report 2024

Get the first-of-its-kind report on how organizations are preparing for GenAI-specific threats.

Free Download
Read LLM Security Playbook

Learn about the most common LLM threats and how to prevent them.

Download

Explore AI Regulations.

Compare the EU AI Act and the White House’s AI Bill of Rights.

Understand AI Security Basics.

Get Lakera's AI Security Guide for an overview of threats and protection strategies.

Uncover LLM Vulnerabilities.

Explore real-world LLM exploits, case studies, and mitigation strategies with Lakera.

Optimize LLM Security Solutions.

Use our checklist to evaluate and select the best LLM security tools for your enterprise.

Master Prompt Injection Attacks.

Discover risks and solutions with the Lakera LLM Security Playbook.

Unlock Free AI Security Guide.

Discover risks and solutions with the Lakera LLM Security Playbook.

You might be interested

Jailbreaking Large Language Models: Techniques, Examples, Prevention Methods

What does LLM jailbreaking really means, and what are its consequences? Explore different jailbreaking techniques, real-world examples, and learn how to secure your AI applications against this vulnerability.
Blessin Varkey
March 25, 2025

Reinforcement Learning: The Path to Advanced AI Solutions

Reinforcement Learning (RL) solves complex problems where traditional AI fails. Learn how RL agents optimize decisions through trial-and-error, revolutionizing industries.
Deval Shah
March 11, 2025
Activate
untouchable mode.
Get started for free.

Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.

Join our Slack Community.

Several people are typing about AI/ML security. 
Come join us and 1000+ others in a chat that’s thoroughly SFW.