[飞桨多模态大模型套件PaddleMIX开发大赛] RFC: enhance paddlemix llava data processing

[GreatV]

This pull request introduces three new classes (DataAnalyzer, DataAugmentor, and DataCleaner) to the paddlemix/datacopilot/example/data_insight module, enhancing the functionalities for data analysis, augmentation, and cleaning. The most important changes include the implementation of methods for analyzing datasets, augmenting data, and cleaning datasets by handling anomalies, detecting duplicates, and filtering by quality.

Data Analysis Enhancements:

• paddlemix/datacopilot/example/data_insight/data_analysis.py: Introduced the DataAnalyzer class, which includes methods for analyzing dataset statistics, image quality, text quality, and image-text matching using the CLIP model.

Data Augmentation Enhancements:

• paddlemix/datacopilot/example/data_insight/data_augmentation.py: Added the DataAugmentor class, which provides methods for augmenting images and text, as well as combining original and augmented samples.

Data Cleaning Enhancements:

• paddlemix/datacopilot/example/data_insight/data_cleaning.py: Introduced the DataCleaner class, which includes methods for handling anomalies, detecting duplicates, and filtering samples based on quality metrics.

[paddle-bot]

Thanks for your contribution!