視覺語言模型的運作方式與類型

VLM:Vision Language Transformers, is an artifiacl intelligence that can process visual information, such as images, and language, such as text, and learn how they relate to each other.
A VLM is an AI model designed to connect visual information, such as images, with language, such as captions, questions, and answers. Understanding how these models learn requires exploring their training data, their learning objectives, and the loss functions that guide their improvement.

Vision-Language model can learn relationships between what it sees in an image and what people say about
for example:

Input: an image of dog with a ball
Question: what is happening
Output:a dog running on the grass after the ball.

How does a VLM work?

1. Vision encoder:Turns the image into numerical features,
2. Multimodal connection:Helps the model combine visual and language information,
3. Language model:Can use that information to answer questions or describe the image.

Types of vision language models
Vision language models can be categorised based on how they process and connect visual and textual information.

1. Vision-to-text Models this models focus on generationg textual descriptions or answering questions based on visual inputs.

2. Text-to-Vision Models
   text to vision models generate or modify visual content based on textual descriptions.

3. Cross-modal retrieval Models
   Cross-modal retrieval models use one type of data, such as text or images to search and retrieve information from another modality.

Image search using text
alows users to retrieve relevant images using textual queries.
Query: mountaiin view
Output:images of mountain
Text search using images
uses an imge as input to retrieve related text, descriptions or documents .
Input:Image of a car
Output: different descriptions about the car