Skip to content

Commit

Permalink
add news
Browse files Browse the repository at this point in the history
  • Loading branch information
zhengjinaling committed Jul 23, 2024
1 parent 58df791 commit 81f4875
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 2 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

[[paper]](https://arxiv.org/abs/2405.19783) [[project page]](https://2toinf.github.io/IVM/)

### 🔥 IVM has been selected as outstanding paper at MFM-EAI workshop @ICML2024

## Introduction

We introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model. By constructing visual masks for instruction-irrelevant regions, IVM-enhanced multimodal models can effectively focus on task-relevant image regions to better align with complex instructions. Specifically, we design a visual masking data generation pipeline and create an IVM-Mix-1M dataset with 1 million image-instruction pairs. We further introduce a new learning technique, Discriminator Weighted Supervised Learning (DWSL) for preferential IVM training that prioritizes high-quality data samples. Experimental results on generic multimodal tasks such as VQA and embodied robotic control demonstrate the versatility of IVM, which as a plug-and-play tool, significantly boosts the performance of diverse multimodal models.
Expand Down Expand Up @@ -95,7 +97,6 @@ Robot Infrastructure: [https://github.com/rail-berkeley/bridge_data_robot](https

This work is built upon the [LLaVA](https://github.com/haotian-liu/LLaVA) and [SAM](https://github.com/facebookresearch/segment-anything). And we borrow ideas from [LISA](https://github.com/dvlab-research/LISA)


## Citation

```
Expand Down
21 changes: 20 additions & 1 deletion index.html
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,26 @@ <h1 class="title is-1 publication-title is-bold">
<span class="author-block">✉Corresponding author:</span>
<span class="author-block"><a href="mailto:[email protected]">[email protected]</a></span>
</div>

<style>
.accepted {
text-align: center;
color: #0a283d; /* Dark blue color */
font-size: 24px;
background-color: #e8f0fe79; /* Light blue background */
border: 1px solid #B6D4FE;
border-radius: 10px;
padding: 20px;
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
}
</style>
<div class="accepted">
<i class="fas fa-fire icon" style="color: red;"></i>
Exciting News!
<div>Our paper has been selected as
<span style="color: red; font-weight: bold;"> outstanding paper </span>

at MFM-EAI workshop@ICML2024</div>
</div>

<div class="column has-text-centered">
<div class="publication-links">
Expand Down

0 comments on commit 81f4875

Please sign in to comment.