Deploying state-of-the-art machine learning models can often lead to a myriad of issues due to the dependencies of the more salient packages - most commonly PyTorch and TensorFlow. At Lakera, we have released an implementation of OpenAI’s CLIP model that completely removes the need for PyTorch, enabling you to quickly and seamlessly install this fantastic model in production and on edge devices.
CLIP (Contrastive Language-Image Pre-Training) is powering some of the most exciting image to text applications out there right now. It’s a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. There are three main components that comprise this model:
- The text tokeniser, which converts the given natural language into tokens (embeddings).
- The image preprocessor, which converts the given image into embeddings.
- The CLIP model itself, which outputs the cosine similarities of the text and image embeddings generated above.
The main issue we have found is that all three of these pieces utilise PyTorch - so we decided to simplify things for you!
We achieved this with the following steps:
- The text tokeniser was rewritten in NumPy.
- We wrote our own image preprocessor, which mimics the functionality of CLIP’s preprocessor.
- We exported the CLIP model to an .onnx format, meaning that we have essentially swapped the PyTorch dependency for the lightweight onnxruntime.
Try it out! Don’t forget to give it a star and reach out if you have any feedback!
Learn how to protect against the most common LLM vulnerabilities
Download this guide to delve into the most common LLM security risks and ways to mitigate them.
Don’t miss the updates!
Subscribe to our newsletter to get the recent updates on Lakera product and other news in the AI LLM world. Be sure you’re on track!
You might be interested
Jailbreaking Large Language Models: Techniques, Examples, Prevention Methods
What does LLM jailbreaking really means, and what are its consequences? Explore different jailbreaking techniques, real-world examples, and learn how to secure your AI applications against this vulnerability.
.png)
The Beginner’s Guide to Hallucinations in Large Language Models
As LLMs gain traction across domains, hallucinations—distortions in LLM output—pose risks of misinformation and exposure of confidential data. Delve into the causes of hallucinations and explore best practices for their mitigation.
Activate
untouchable mode.
untouchable mode.
Get started for free.
Lakera Guard protects your LLM applications from cybersecurity risks with a single line of code. Get started in minutes. Become stronger every day.
Join our Slack Community.
Several people are typing about AI/ML security. Come join us and 1000+ others in a chat that’s thoroughly SFW.
