MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective

Hailang Huang^1,2*,  Yong Wang²,  Zixuan Huang^1,2*,   Huaqiu Li^2,3*,   Tongwen Huang²,
Xiangxiang Chu^2†,   Richong Zhang^1‡

¹Beihang University,  ²Alibaba Group,  ³Tsinghua University

^*Work done during an internship at Alibaba Group     ^†Project Leader     ^‡Corresponding Author

📖Paper | 🏠Homepage | 🤗Huggingface

Large Multimodal Models (LMMs) have demonstrated remarkable capabilities. While existing benchmarks for evaluating LMMs mainly focus on image comprehension, few works evaluate them from the image generation perspective. To address this issue, we propose a straightforward automated evaluation pipeline. Specifically, this pipeline requires LMMs to generate an image-prompt from a given input image. Subsequently, it employs text-to-image generative models to create a new image based on these generated prompts. Finally, we evaluate the performance of LMMs by comparing the original image with the generated one. Furthermore, we introduce MMGenBench-Test, a comprehensive benchmark developed to evaluate LMMs across 13 distinct image patterns, and MMGenBench-Domain, targeting the performance evaluation of LMMs within the generative image domain. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability in both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks, related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, our pipeline facilitates the efficient assessment of LMMs performance across diverse domains by using solely image inputs.

---

Usage

Getting Started

Environment Installation

Clone this repository

git clone [email protected]:lerogo/MMGenBench.git

cd MMGenBench

Download dataset

huggingface-cli download --repo-type dataset lerogo/MMGenBench --local-dir MMGenBench-data

Install the relevant environment, including torch, transformers, diffusers and unicom (used to extract image representation).

Preliminary

We use the InternVL2-2B as an example. The structure of the code and data is as follows.

.
├── MMGenBench-data # The MMGenBench-Test/Domain dataset we downloaded from huggingface
│   ├── MMGenBench-Domain.json
│   ├── MMGenBench-Domain.tsv
│   ├── MMGenBench-Test-label-count.json
│   ├── MMGenBench-Test-label-index.json
│   ├── MMGenBench-Test.json
│   ├── MMGenBench-Test.tsv
│   ├── README.md
│   └── check.py
├── README.md # This file
├── evalimg # For extracting features and calculating metrics using the image representation model
│   ├── metric_fid.py
│   ├── output
│   │   ├── InternVL2-2B_MMGenBench-Domain.json
│   │   └── InternVL2-2B_MMGenBench-Test.json
│   ├── requirements.txt
│   ├── run.py
│   └── run.sh
├── generate # For processing LMMs' output with the text-to-image models
│   ├── flux.py
│   ├── input
│   │   ├── InternVL2-2B_MMGenBench-Domain.xlsx
│   │   └── InternVL2-2B_MMGenBench-Test.xlsx
│   ├── kolors.py
│   ├── lumina.py
│   ├── output
│   │   ├── InternVL2-2B_MMGenBench-Domain.tsv
│   │   └── InternVL2-2B_MMGenBench-Test.tsv
│   ├── requirements.txt
│   ├── run.py
│   ├── run.sh
│   ├── sd.py
│   └── tools.py
└── visual # For visualization
    ├── outputs
    │   ├── InternVL2-2B_MMGenBench-Domain.json
    │   ├── InternVL2-2B_MMGenBench-Domain.xlsx
    │   ├── InternVL2-2B_MMGenBench-Test.json
    │   └── InternVL2-2B_MMGenBench-Test.xlsx
    ├── run.py
    └── run.sh

Evaluation Pipeline

Stage 1

Adapt your model in VLMEvalKit and use MMGenBench for inference.

Run command:

torchrun --nproc-per-node=4 run.py --model <YOUR LMM> --data MMGenBench-Test MMGenBench-Domain --mode infer --verbose

We use the InternVL2-2B as an example. Then you can get two files: InternVL2-2B_MMGenBench-Test.xlsx, InternVL2-2B_MMGenBench-Domain.xlsx. Put them in folder ./generate/input

Stage 2

Modify ./generate/run.sh to select the text-to-image model and to select the number of GPUs you need to use.

And run:

cd generate
bash run.sh

Then you can get two files: ./generate/output/InternVL2-2B_MMGenBench-Test.tsv, ./generate/output/InternVL2-2B_MMGenBench-Domain.tsv

Stage 3

We will use the unicom model to extract features from the original images and generated images, you need to install unicom (https://github.com/deepglint/unicom).

Modify ./evalimg/run.sh to evaluate the performance on MMGenBench-Test and MMGenBench-Domain respectively.

And run:

cd evalimg
bash run.sh

Then you can get two files: evalimg/output/InternVL2-2B_MMGenBench-Test.json, ./evalimg/output/InternVL2-2B_MMGenBench-Domain.json.

Visual

Run command:

cd visual
bash run.sh

You can see the relevant results in the output folder, including metrics and visualization results.

Q&A

If you have any questions, please submit an issue or contact lerogohl<AT>gmail.com.

Citation

If you find MMGenBench or code useful, please cite

@misc{huang2024MMGenBench,
      title={MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective},
      author={Hailang Huang and Yong Wang and Zixuan Huang and Huaqiu Li and Tongwen Huang and Xiangxiang Chu and Richong Zhang},
      year={2024},
      eprint={2411.14062},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.14062}, 
}