A Vision-Language Model for Multitask Classification of Memes (Neural Networks)

The current study tackles the challenges of multimodal hate speech identification in internet memes, which frequently combine text and images to express cultural biases. The authors describe ViT-BERT CAMT, a cross-attention multitask model that uses a linear self-attentive fusion mechanism to combine vision transformer (ViT) data with BERT-based textual representations. The model performs exceptionally well in identifying sentiment, sarcasm, offensiveness, and discriminatory content when tested on the SemEval 2020 Memotion and MIMIC datasets. The results demonstrate how well joint image-text modeling captures subtle semantic and spatial links, promoting automatic moderation of online debate that is sensitive to cultural differences.

https://www.sciencedirect.com/science/article/abs/pii/S0893608025009694