Top Multimodal Vision Models

Grounding DINO is a zero-shot object detection model made by combining a Transformer-based DINO detector and grounded pre-training.

CLIP (Contrastive Language-Image Pre-Training) is an impressive multimodal zero-shot image classifier that achieves impressive results in a wide range of domains with no fine-tuning. It applies the recent advancements in large-scale transformers like GPT-3 to the vision arena.

LLaVA is an open source multimodal language model that you can use for visual question answering and has limited support for object detection.

CogVLM shows strong performance in Visual Question Answering (VQA) and other vision tasks.

Qwen-VL is an LMM developed by Alibaba Cloud. Qwen-VL accepts images, text, and bounding boxes as inputs. The model can output text and bounding boxes. Qwen-VL naturally supports English, Chinese, and multilingual conversation.

MetaCLIP is a zero-shot classification and embedding model developed by Meta AI.

BakLLaVA is an LMM developed by LAION, Ontocord, and Skunkworks AI. BakLLaVA uses a Mistral 7B base augmented with the LLaVA 1.5 architecture.

Kosmos-2 is a multimodal language model capable of object detection and grounding text in images.

BLIPv2 is a multimodal model developed by Salesforce Research.

GPT-4 with Vision is a multimodal language model developed by OpenAI.

SigLIP is an image embedding model defined in the "Sigmoid Loss for Language Image Pre-Training" paper.

Gemini is a family of Large Multimodal Models (LMMs) developed by Google Deepmind focused specifically on multimodality.