paperlist.ai

CVPR 2021 Papers

Conference on Computer Vision and Pattern Recognition

202620252024202320222021202020192018201720162015

Decision breakdown

1,661papers

Counted from the official paper list; may differ slightly from the number the conference announced.

  • Poster
    1,661100.0%

Top 10 topics (click to filter)

Trends for all 207 research topics

Browse all papers

All1,661Poster1,661

Topic: Multimodal LLMs · 2 papers · Reset filters

VinVL: Revisiting Visual Representations in Vision-Language Models

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao

Expand abstract

This paper presents a detailed study of improving vision features and develops an improved object detection model for vision language (VL) tasks. Compared to the most widely used bottom-up and top-down model [2], the new model is bigger, pre-trained on much larger training corpora that combine multiple public annotated object detection datasets, and thus can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses solely on improving the vision-language fusion model and leaves the object detection model improvement untouched, we present an empirical study to show that vision features matter significantly in VL models. In our experiments we feed the vision features generated by the new object detection model into a pre-trained transformer-based VL fusion model Oscar+, and fine-tune Oscar+ on a wide range of downstream VL tasks. Our results show that the new vision features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

Poster
Multimodal LLMs
Causal Attention for Vision-Language Tasks

Xu Yang, Hanwang Zhang, Guojun Qi, Jianfei Cai

Expand abstract

We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the Q-K-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT [??], which uses fewer data and less computational power, comparable to the heavier UNITER [??]. Code is published in https://github.com/yangxuntu/lxmertcatt.

Poster
Multimodal LLMs