# FloodLense: A Framework for ChatGPT-based Real-time Flood Detection

Pranath Reddy Kumbam and Kshitij Maruti Vejre  University of Florida,
Gainesville, USA

###### Abstract

This study addresses the vital issue of real-time flood detection and
management. It innovatively combines advanced deep learning models with Large
language models (LLM), enhancing flood monitoring and response capabilities.
This approach addresses the limitations of current methods by offering a more
accurate, versatile, user-friendly and accessible solution. The integration of
UNet, RDN, and ViT models with natural language processing significantly
improves flood area detection in diverse environments, including using aerial
and satellite imagery. The experimental evaluation demonstrates the models’
efficacy in accurately identifying and mapping flood zones, showcasing the
project’s potential in transforming environmental monitoring and disaster
management fields.

###### Index Terms:

Machine Learning, Large language models, Flood Detection, Real-time
Monitoring, Disaster Management

##  I Introduction

###  I-A Societal Applications

The increasing prevalence and severity of flood events, exacerbated by climate
change and rapid urbanization, underscore the critical need for effective
flood detection and management strategies. Climate change contributes to more
frequent and intense weather phenomena, leading to unpredictable and
devastating floods. Urbanization further aggravates this issue by creating
impermeable surfaces that hinder natural water absorption and drainage,
elevating flood risks in densely populated areas. These challenges are
compounded by the substantial impact of floods on societies, economies, and
the environment. Flooding disasters result in significant loss of life,
displacement of communities, extensive damage to infrastructure, and long-
lasting economic disruptions. Additionally, they pose severe environmental
threats, including ecosystem destruction and water pollution. Current flood
management practices often struggle to keep pace with these challenges.
Traditional methods face limitations in real-time data acquisition, coverage,
and predictive accuracy, highlighting the urgent need for more advanced,
integrated, and scalable solutions. This research aims to address these gaps
by introducing a novel flood detection system, leveraging the latest
advancements in deep learning and natural language processing to offer a more
effective approach to flood monitoring and response. For example, during
sudden flash floods, users in a vulnerable area could query their locality,
receive a real-time visual update, and make informed decisions regarding
evacuation or safety precautions.

###  I-B Existing Methods

Image processing techniques have played a significant role in flood detection.
Techniques such as edge detection, pixel analysis, and the utilization of
remote sensing technologies have been successful [1] [2] [3]. These approaches
require manual feature engineering, and their inherent lack of scalability,
adaptability, and robustness have motivated a shift towards data-driven
machine learning models.

Some of the earlier data-driven approaches relied heavily on using
thresholding-based approaches on hand-crafted features to discriminate between
floodwater and confusers like vegetation and shadows [4]. These traditional
methodologies typically required domain experts to hand-craft features, which
could be a time-consuming and expertise-driven process. The authors in [5]
employed traditional machine learning algorithms like Random Forest (RF),
Support Vector Machine (SVM), and Maximum Likelihood Classifier (MLC) to
process satellite imagery for detecting flooded areas. Regression analysis and
tree-based approaches have also been employed for predicting heavy rain damage
[6].

The authors in [5] have also implemented a novel unsupervised machine learning
framework called the change detection approach based on a combination of the
Otsu algorithm, fuzzy rules, and iso-clustering methods for urban flood
detection and reported a superior performance compared to traditional
supervised methods. The use of change detection has been extensively explored
in some of the other works [7]. While an unsupervised machine learning
algorithm such as change detection is more robust due to its higher speed,
lesser requirements for training data, and computation runtime, some of the
more traditional change detection implementations such as the one used by [7]
are highly prone to excess false positives.

Traditional supervised methods can achieve high accuracy given a sufficiently
large and well-labeled training dataset. However, They require a significant
amount of labeled data, which can be time-consuming and expensive to obtain.
The automatic identification of features through deep learning algorithms can
reduce the reliance on manual feature engineering and domain expertise, thus
potentially improving the accuracy, efficiency, and timeliness of flood
detection from satellite imagery.

In recent years, deep learning models have significantly impacted the realm of
flood segmentation and detection [8], often outperforming the traditional
methods outlined in section 1.1. Models such as U-Net and Convolutional Neural
Networks (CNNs) [9] [10] [11] [12] have risen to prominence due to their
ability to learn complex patterns from large datasets without the necessity
for manual feature engineering. Furthermore, the application of modified U-Net
architectures and other tailored deep learning models like FAPNET have also
been explored for enhancing flood-water detection in satellite remote sensing
applications [13].

LSTM, a type of recurrent neural network which has shown promise in time-
series prediction, has been utilized for time-series prediction which is
relevant for flood prediction [14] [15]. The intersection of spatial and
temporal data dynamics is crucial for a comprehensive understanding and
prediction of flood events. To that extent, Spatial-Temporal Graph Deep
Learning Models [16] have also been explored for flood nowcasting, focusing on
spatial and temporal data dynamics. Furthermore, unsupervised and semi
supervised approaches [17] [18] have been harnessed to overcome the challenges
associated with data labeling, showcasing potential for automating flood
detection and segmentation from satellite imagery without the necessity for
extensive labeled data.

The increased application of deep learning models in flood segmentation and
detection underscores the transition towards more data-driven, automated, and
precise approaches in flood management, leveraging the availability of large
datasets and advanced computational techniques to augment the identification
and analysis of flood events from satellite imagery. Deep learning methods can
handle complex data structures and are capable of learning features
automatically from the data, potentially achieving higher accuracy compared to
traditional machine learning methods. However, they require a large amount of
labeled data and computational resources. The availability of high-quality
flood datasets is rather limited, which can hinder the training of deep
learning models, limiting their ability to generalize to new terrains,
geographical, and climatic and other real-world scenarios. While the
computational complexity of deep learning algorithms is an inherent problem,
we aim to solve the later shortcoming with our approach that leverages the
generalization ability [19] of a generative conditional diffusion model.

Diffusion models are an emerging class of deep generative models that
reconstruct high-fidelity samples through iterative denoising [20]. Recent
works have applied conditional diffusion models to diverse image-to-image
translation tasks like segmentation and inpainting [21] [22], demonstrating
photorealistic output. Diffusion models have also been used for medical image
segmentation [23]. Such conditional generative modeling is promising for flood
detection and visualization but remains unexplored.

This section focuses on the review of some of the existing interfaces for
remote sensing and flood detection. While flood detection has benefited
enormously from remote sensing and computer vision advances, advances in flood
detection algorithms must be accompanied by progress in public accessibility.
Below are some of the organizations that offer static interfaces for remote
sensing and flood detection.

The USGS employs WaterWatch [24], collects flood data through the USGS’s
extensive network of streamgages. It doesn’t harness satellite imagery hence
it can’t be used for post-flood analysis. Limitation is that it can’t be used
to find the regions actually affected by the flood.

The Global Flood Detection System from JRC [25] uses the capabilities of
AMSR-E’s 36 GHz H-polarization data to detect daily riverine inundation
events. The methodology, however, struggles in areas with intricate land use
patterns or topography, occasionally making it difficult to differentiate
between wet soils, water bodies, and genuine floodwaters. Another limitation
is its reliance on historical baselines that, if not updated, can sometimes
lead to inaccuracies in flood predictions.

MODIS NRT Global Flood Product [26] from NASA LANCE represents another
significant stride in flood detection. This technology capitalizes on the
Modified Normalized Difference Water Index (MNDWI) to ascertain flood-affected
regions. While effective, it faces challenges comparable to JRC’s system,
especially in regions with complex topographies and diverse land use patterns
[27].

On the algorithmic front, GEE has championed a unique approach to identify
flood-affected terrains. By using the threshold detection technique grounded
in Otsu’s Algorithm, it seeks to provide reliable flood data. However, its
results sometimes grapple with the issue of misclassification, thereby
limiting its accuracy in certain scenarios [28].

Lastly, MAXAR’s Open Data Program [29] endeavors to provide a visual
representation of flood-affected zones. By furnishing high-resolution before-
and-after images, it presents a stark visual comparison. However, it restricts
itself to imagery without delving into deeper analytical aspects, thus serving
more as a visualization tool than a comprehensive flood analysis interface.

Static interfaces including the traditional graphical user interface (GUI)
based systems for flood detection and monitoring pose significant challenges
around limited accessibility and usability for non-experts including the
general public. Traditional interfaces can be difficult to use since they
require the input query to be formulated precisely through parameters,
coordinates, etc. The lack of interactivity and contextual feedback in static
interfaces makes it difficult for users to explore data interactively or get
clarifications on model outputs. Moreover, non-conversational systems cannot
handle follow-up questions. Integrating a flood detection backend with
conversational interfaces such as GPT [30] could help overcome these
limitations by enabling natural language conversations that make flood
detection/monitoring more contextual and accessible.

###  I-C Technical Contributions and Project Novelty

This project introduces significant technical advancements in flood detection
by integrating state-of-the-art deep learning models with the ChatGPT
platform. Key contributions include:

  * •

Integration: The combination of UNet, Residual Dense Network (RDN), and Vision
Transformer (ViT) models with ChatGPT represents a novel approach, enhancing
the system’s capability to process and interpret complex environmental data
effectively.

  * •

Advanced Image Processing: The application of these deep learning models for
processing aerial and satellite imagery significantly improves flood area
detection and mapping accuracy, outperforming traditional methods.

  * •

User Interaction: By leveraging ChatGPT’s natural language processing
capabilities, the system offers an intuitive and user-friendly interface,
allowing users to interact and receive real-time flood data in a
conversational manner.

  * •

Contribution to Disaster Management: This project represents a significant
step forward in environmental monitoring and disaster response, providing a
scalable and efficient tool for timely flood detection and management.

These contributions not only demonstrate the project’s technical prowess but
also highlight its potential to revolutionize flood management practices,
offering a more responsive, accurate, and user-centric solution.

###  I-D Limitations of Current Approaches

Current state-of-the-art flood detection approaches [8] [9] [10] [11] [12] are
hindered by several machine learning challenges that limit their practical
application. Traditional models are heavily data-dependent, necessitating
extensive, well-labeled datasets that are both costly and time-consuming to
assemble, which impedes their ability to generalize across diverse flooding
events. These models often rely on manual feature engineering, requiring
significant domain expertise, and may still fail to capture the multifaceted
nature of floods, leading to inaccuracies. Furthermore, the models’
applicability is constrained by computational demands and a lack of
interpretability, which is crucial in emergency situations where understanding
the rationale behind predictions is vital. Compounding these issues is the
absence of practical, user-friendly applications that can translate complex
model outputs into actionable insights for disaster response. Most existing
solutions do not cater to non-expert users and lack real-time processing
capabilities, which are essential during time-critical events. Additionally,
there is a notable gap in integrating these models with existing emergency
management systems to provide a seamless end-to-end response mechanism. Our
project seeks to bridge these gaps by creating an accessible, real-time flood
detection tool that non-technical users can easily interact with, thus
enhancing the practical utility of flood detection in urgent scenarios.

The introduction of a conditional diffusion model represents a significant
innovation in the field. This model type utilizes an iterative generative
process to refine predictions, providing a powerful mechanism for capturing
the complex patterns associated with flood delineation. Unlike traditional
deep learning models, the diffusion model has the potential to produce high-
fidelity segmentation maps with fewer artifacts and more precise boundaries.
Additionally, the project explores the potential of Vision Transformers (ViT),
which use self-attention mechanisms to process satellite images. This allows
the model to focus on the most relevant parts of an image, improving the
accuracy of flood detection, particularly in complex environments where
traditional CNNs may falter. By integrating these models with ChatGPT,
FloodLense provides an intuitive, natural language-based interface that allows
users to request flood information simply by asking questions. This ease of
use represents a significant step forward in making advanced flood detection
technology accessible to a broader audience, including those without technical
expertise.

##  II Problem definition

The central problem addressed in this project is the real-time detection and
mapping of flood areas using advanced deep learning techniques integrated with
a conversational AI platform. The primary components of this problem include:

  1. 1.

Inputs: The system inputs are high-resolution aerial and satellite images,
which capture diverse landscapes under various weather conditions and urban
developments. These images are critical in identifying potential flood-
affected areas.

  2. 2.

Deep Learning Models: The system utilizes a combination of UNet, Residual
Dense Network (RDN), and Vision Transformer (ViT) for image analysis. UNet is
pivotal for precise localization in the images, essential for delineating
flood boundaries accurately. RDN and ViT are integrated to enhance image
details and pattern recognition. RDN focuses on improving image resolution and
clarity, crucial in differentiating flooded areas from other water bodies or
wet surfaces. ViT, on the other hand, processes images as sequences of
patches, providing a comprehensive view of large-scale environmental patterns,
enhancing the detection of widespread flooding.

  3. 3.

Output: Accurate identification and mapping of flooded regions, presented in a
format that is both informative and accessible to a range of users, from
disaster management professionals to the general public.

  4. 4.

Objective: The main goal is to develop a robust, scalable, and efficient flood
detection system capable of rapidly analyzing complex environmental imagery
and providing actionable, real-time insights.

Consider a scenario where a Coastal city is facing imminent flooding due to
prolonged heavy rain or rise in the sea levels. For example we want to check
the food situation in Chennai, We can ask the GPT to finding the current flood
situation in Chennai and The system requests real-time satellite imagery from
SentinalHub and passes it as input. Utilizing the combined strengths of UNet,
RDN, and ViT, the system accurately processes these images to identify and
segment flood-affected areas. The processed data is then relayed through the
ChatGPT interface, offering users detailed information about the flood’s
impact. This facilitates prompt and efficient response actions by disaster
management teams, potentially saving lives and reducing property damage.

Figure 1: ChatGPT plugin Simulation

In Figure 1. We can see that even though the prompt is not currently providing
the city name ”What is the Flood Situation in Chhheennai”, yet it provides the
correct image url to view the flood situation in Chennai, India. Figure 2.
shows the original image and the flood affected highlighted areas.

Figure 2: Original vs Highlighted Flood Detected Image

##  III Proposed solution

The FloodLense project introduces a comprehensive solution to flood detection
that harnesses the power of advanced deep learning techniques and user-
friendly interfaces. The central proposition is to utilize a blend of neural
network architectures, including U-Net [31], Residual Dense Network (RDN)
[32], Vision Transformer (ViT) [33], and conditional diffusion models [20], to
analyze and segment satellite imagery for real-time flood monitoring.

The U-Net architecture, specifically designed for biomedical image
segmentation, features a symmetric encoder-decoder structure. The encoder path
follows the typical convolutional neural network (CNN) architecture,
systematically reducing the spatial dimensions while increasing the depth to
capture contextual information. Conversely, the decoder path up-samples the
feature maps and combines them with high-resolution features from the encoder
via skip connections, essential for precise localization and segmentation of
spatial structures in satellite imagery.

The Residual Dense Network (RDN) enhances this by employing densely connected
layers. Each layer in RDN receives inputs from all preceding layers, fostering
feature reuse and propagation, and improving the learning of detailed
representations. This is particularly effective in differentiating subtle
features in complex images, such as distinguishing water from land in flooded
areas.

Vision Transformers (ViTs) diverge from traditional CNNs by applying self-
attention mechanisms, allowing the model to weigh different parts of the input
image differently. This approach enables ViTs to capture long-range
dependencies and global context in the image, which is instrumental in
understanding extensive and diverse flood patterns. Unlike CNNs, which process
local receptive fields, ViTs can analyze the image in a more holistic manner,
making them well-suited for large-scale environmental pattern recognition.

Together, these models provide a comprehensive approach to flood detection,
combining local precision with global contextual understanding, which is
critical for accurate and efficient flood mapping. Moreover, we plan to
explore the capabilities of conditional diffusion models, a class of
generative models that have demonstrated success in creating high-fidelity
images through an iterative process of denoising. These models hold the
promise of generating highly detailed and accurate segmentation maps,
essential for identifying flood-affected areas. In the preliminary analysis,
we have explored a U-Net and Residual Dense Network models using the FloodNet
dataset [34] [35]. The results produced are visually promising as the
segmentation map presented in Figure 3 shows, indicative of the models’
potential in flood detection.

Figure 3: Sample Output Showing Flooded Buildings - UNet (Top), RDN (Bottom)

To make the system as shown in Figure 4 accessible to users without technical
expertise, we plan to integrate ChatGPT [36], which will act as an interface
between the user and the deep learning models. Users will be able to input
queries in natural language, which ChatGPT will interpret to extract relevant
geographical and temporal data. This interaction simplifies the user
experience, making the process of accessing flood information simple.

Figure 4: System Architecture

SentinelHub [37] will be used as the primary source of satellite data,
providing up-to-date and high-resolution imagery necessary for the models to
perform segmentation. We will develop an API that will fetch the required
images from SentinelHub in response to user queries processed by ChatGPT. The
API will be designed for efficiency.

We have developed a Flask-based web application designed to provide access to
satellite imagery via a simple user interface. The application, hosted on
Replit, allows users to retrieve and display Sentinel-2 satellite images based
on specified latitude and longitude coordinates. A key feature of this
application is the /download_image endpoint, which dynamically generates a
bounding box around the provided coordinates and retrieves the latest true-
color image from the Sentinel-2 data collection. This image is then saved in a
server directory, with its URL returned to the user for access.

###  III-A Psuedo-Code for Satellite Image Processing and Retrieval

  1. 1.

Initialize Configuration

     * •

Set instance ID for Sentinel Hub Configuration.

  2. 2.

Define Function to Retrieve Coordinates

     * •

Input: Request parameters.

     * •

Process: Retrieve latitude and longitude from request, with default values.

     * •

Output: Bounding box coordinates based on input latitude and longitude.

  3. 3.

Define Function to Retrieve and Process Satellite Data

     * •

Input: Bounding box coordinates.

     * •

Process:

       1. (a)

Request Sentinel-2 data for the bounding box.

       2. (b)

Select the latest image from the data.

       3. (c)

Process the image:

          * –

Crop, resize, and convert color format.

       4. (d)

Save the image with a timestamp in a designated directory.

       5. (e)

Log image information.

       6. (f)

Generate a URL for accessing the image.

     * •

Output: JSON response with the image URL.

###  III-B Psuedo-Code to Simulate ChatGPT Plugin

  1. 1.

Initialize OpenAI Client

     * •

Set API key.

  2. 2.

Define Custom Tool for Location Retrieval

     * •

Tool to extract location from user input.

  3. 3.

Define Function to Get Location Details

     * •

Input: Location query in natural language.

     * •

Process: Use OpenAI API to interpret the location query.

     * •

Output: Extracted location details.

  4. 4.

Define Function to Geocode Location

     * •

Input: Location name.

     * •

Process: Use Nominatim service to convert location name to coordinates.

     * •

Output: Latitude and longitude of the location.

  5. 5.

Define Function to Get Satellite Data

     * •

Input: Latitude and longitude.

     * •

Process:

       1. (a)

Send a request to a local server with coordinates.

       2. (b)

Retrieve the URL of the processed image.

     * •

Output: Display the URL or error message.

  6. 6.

Main Function

     * •

Process:

       1. (a)

Prompt user for location query.

       2. (b)

Extract location details using OpenAI’s chat completion.

       3. (c)

Geocode the location to get coordinates.

       4. (d)

If coordinates are found, display them and retrieve the satellite image.

       5. (e)

If not, display an error message.

##  IV Evaluation

This study embarks on a critical evaluation of various models for flood and
water detection. The study’s cornerstone lies in its exploration of the
capability of deep learning models such as UNet [31], Residual Dense Net (RDN)
[32], and Vision Transformer (ViT) [33] in accurately segmenting and
identifying water bodies in diverse imaging scenarios. The investigation
navigates through the intricacies of model performance, computational demands,
and the pragmatic viability of these models in real-world applications.
Additionally, we also assess the effectiveness of the interface application
designed to extract location details from user-input sentences related to
flood and water detection.

Specifically, we will be investigating the following questions:

  * •

How do different models like UNet, Residual Dense Net (RDN), and Vision
Transformer (ViT) compare in terms of segmentation accuracy?

  * •

What are the implications of model complexity on computational efficiency and
the interpretability of results?

  * •

How does sensitivity to threshold variations impact model performance,
particularly with respect to false positives?

  * •

How accurate is the extraction of locations from natural language inputs?

  * •

How reliable is the process of converting these extracted locations into
geographic coordinates?

  * •

What is the model’s capability in handling a variety of location-related
queries, including those that are ambiguous or refer to non-existent
locations?

###  IV-A Evaluation Data and Experimental Configuration

In this study, two carefully chosen datasets form the backbone of the
experimental evaluation: the FloodNet aerial drone dataset [34] [35] for
preliminary model assessment, and the Sentinel satellite dataset [37] for
training the final model selected for deployment. This strategic delineation
ensures a comprehensive analysis of model capabilities in diverse imaging
scenarios, crucial for flood detection applications.

The FloodNet dataset is used for initial comparative analysis of the deep
learning models. It consists of aerial drone images that capture a multitude
of flood situations, providing a rich tapestry of data for understanding the
challenges in aerial-based flood detection. The dataset, tailored for
supervised flood detection tasks, encompasses 2,108 training samples, ensuring
a varied learning environment for the models. For the purpose of evaluating
the models’ accuracy in segmentation, 235 testing samples are included. The
images have been downscaled to a resolution of 128x128 pixels using nearest-
neighbor interpolation. This method is specifically chosen for its capability
to resize images with minimal interpolation errors, thus preserving the
integrity of the original data. Each image is a 3-channel RGB image and has
undergone normalization to standardize the input for the models, eliminating
variations due to disparate lighting conditions or scales.

The Sentinel satellite dataset, with its focus on satellite imagery of water
bodies, serves as the real-world application context for the final model
selected post-preliminary analysis. This dataset is composed of 2,820 training
samples, offering a different perspective and scale for water body detection
from space. The dataset is rounded off with 41 testing samples, essential for
assessing the model’s practical deployment capabilities. The images are
reshaped and downscaled to a resolution of 256x256 pixels, again using
nearest-neighbor interpolation to maintain the fidelity of the satellite
imagery while making it amenable to deep learning processing. Like the
FloodNet dataset, these images are also 3-channel and normalized to maintain
preprocessing consistency.

The selection of UNet, RDN, and ViT models for this study is influenced by
their varied architectural designs, which facilitates a broad spectrum
analysis across different deep learning methodologies. An additional model, a
diffusion model, was initially considered. However, due to its substantial
computational complexity, the study was constrained to working with images of
merely 64x64 dimensions when using this model, a scale that is suboptimal for
effective flood detection analysis. Consequently, the focus shifted to the
other three models, which offered a more feasible balance between
computational demand and performance efficacy. The preprocessing steps,
including normalization and resizing using nearest-neighbor interpolation,
ensure that the data across both datasets is uniformly processed, setting a
standard baseline for model evaluation.

The test data for our study consisted of sentences designed to simulate real-
world queries concerning floods and water bodies, incorporating a variety of
geographical references such as specific cities, regions, rivers, and
landmarks. This dataset featured a range of location types, including names of
cities, rivers, and notable landmarks, to ensure a comprehensive evaluation.
Our model employed OpenAI’s GPT-3.5-turbo [36] for the natural language
processing aspect, effectively interpreting and extracting location
information from the queries. For the conversion of these textual location
descriptors into geographical coordinates, we integrated Nominatim’s geocoding
service, enhancing the model’s capability to accurately geocode the extracted
data.

###  IV-B Evaluation Metric

In assessing the performance of deep learning models for flood/water
detection, a comprehensive set of evaluation metrics is employed to quantify
their capabilities and practical applicability. Key metrics include the
Intersection over Union (IoU) and the Dice Coefficient, both of which are
essential in segmentation tasks. IoU, or the Jaccard Index, measures the
overlap between the predicted segmentation and the ground truth, indicating
the model’s accuracy in identifying flood-affected areas. The Dice
Coefficient, related to IoU, assesses segmentation efficiency, especially
useful in imbalanced classes typical in flood detection. These metrics provide
a nuanced view of the models’ precision in segmenting areas accurately, a
critical aspect in flood detection applications.

Complementing these are metrics like Precision, Recall, and the F1 Score.
Precision calculates the proportion of correctly predicted positive
observations, while Recall measures the proportion of actual positives
correctly identified, both crucial in evaluating the model’s effectiveness in
identifying water regions and flood-affected areas. The F1 Score, the harmonic
mean of Precision and Recall, offers a balanced measure, particularly vital in
scenarios where a trade-off between these two metrics might occur.
Additionally, Accuracy gives an overview of the model’s overall performance,
and Inference Time indicates the model’s computational efficiency, essential
in time-sensitive flood detection scenarios. Lastly, the Sensitivity to
Threshold Variations metric is significant for understanding the models’
robustness and their ability to maintain accuracy under different operational
conditions, making these metrics collectively pivotal in offering a
comprehensive evaluation of the models’ segmentation accuracy, overall
performance, and practical applicability in diverse real-world scenarios.

In evaluating the ChatGPT integration’s performance for location extraction
and geocoding, we are utilizing three key metrics. First, ’Accuracy’ gauges
the precision in identifying and extracting location names from textual input.
Secondly, the ’Geocoding Success Rate’ measures the proportion of accurately
geocoded locations, reflecting the system’s ability to accurately map
extracted names to geographical coordinates. Lastly, the ’Error Rate’ is
assessed, which captures the frequency of erroneous geocoding outcomes or
instances where the system fails to correctly identify a location name. These
metrics collectively provide a comprehensive overview of the system’s efficacy
in handling geocoding tasks within natural language processing contexts.

###  IV-C Performance Analysis and Results

The study’s exploration into the capabilities of UNet, Residual Dense Net
(RDN), and Vision Transformer (ViT) models across FloodNet and Sentinel
datasets provides an insightful dissection of their performance in flood
detection tasks. This section delves deeper into the individual and
comparative analyses of these models, focusing on their performance across
various metrics.

In the preliminary analysis using the FloodNet dataset, the UNet model’s
performance emerged as a key focal point. As seen in Figure 3, UNet’s
segmentation outputs were observed to be markedly superior, suggesting a
nuanced understanding and handling of the complex spatial features
characteristic of flood-affected areas. Despite this visual proficiency, there
was a noticeable disparity when compared to its quantitative metrics as seen
in I. This variance underscores a crucial aspect of model evaluation in
practical applications: the potential limitations of standard quantitative
metrics in capturing the true effectiveness of a model in complex real-world
scenarios. UNet’s ability to outperform in visual segmentation accuracy,
despite not leading in quantitative metrics, hints at its architectural
strengths in effectively processing and delineating complex aerial images, a
vital attribute for accurate flood detection.

Figure 5: Sample Outputs from FloodNet Data. Positive values indicate water and flooded regions. TABLE I: Performance Metrics Metric | RDN | ViT | UNet
---|---|---|---
IoU | 0.49412 | 0.45401 | 0.42273
Dice | 0.56414 | 0.52523 | 0.47872
Precision | 0.57131 | 0.68377 | 0.57269
Recall | 0.59470 | 0.51801 | 0.49207
F1 Score | 0.56414 | 0.52523 | 0.47872
Accuracy | 0.90969 | 0.89435 | 0.90674
TABLE II: Inference Time on FloodNet Data Model | RDN | ViT | UNet
---|---|---|---
Inference Time (ms) | 10.41157 | 66.04217 | 10.70199

The evaluation of the RDN model in the FloodNet dataset context brought to
light its capabilities and limitations in flood scenario segmentation. While
the RDN model exhibited high levels of segmentation accuracy, it slightly
trailed behind the UNet model. This finding is significant as it indicates
that although RDN is proficient in handling the diverse range of features
presented in flood scenarios, it may not be as adept as UNet in capturing
certain finer details. This slight difference in performance could be
attributed to the inherent architectural differences between the two models,
with RDN possibly requiring further tuning to achieve the same level of detail
recognition as UNet in complex flood imagery.

The performance of the ViT model on the FloodNet dataset revealed a notable
gap between its theoretical capabilities and practical output. Despite being
designed as an advanced architecture, ViT’s segmentation outputs were less
accurate when compared visually. This gap between its quantitative metrics and
visual performance raises questions about the model’s suitability for detailed
segmentation tasks like those required in flood detection. It highlights the
potential misalignment between ViT’s architecture and the specific
requirements of fine-grained, localized segmentation tasks, a critical
consideration for its application in flood scenario analysis.

In the ablation study conducted on the FloodNet dataset, significant insights
were gleaned into the internal structure and critical components of the
Residual Dense Net (RDN), Vision Transformer (ViT), and UNet models. For the
RDN model, the ablation of different blocks and layers led to varying degrees
of decline in performance metrics such as precision, recall, and F1 score.
This variability in the impact of ablation across different layers underscored
the nuanced role each component plays in the model’s overall performance,
highlighting the importance of certain layers more than others in accurate
flood detection.

The study revealed a stark dependency of the ViT model on its decoder layers,
with metrics dropping to zero upon their ablation, indicating these layers’
crucial role in segmentation tasks. In contrast, the UNet model exhibited
significant performance drops, even resulting in undefined values in some
cases when specific encoding layers were removed. This suggests a critical
reliance on these layers for the model’s functionality in flood detection,
pointing to their role in effective feature processing. These findings from
the ablation study are instrumental in understanding the architectural
strengths and vulnerabilities of these models, providing a foundation for
future enhancements in their design for improved flood detection capabilities.
The complete ablation study results have been presented in the Appendix.

Figure 6: Samples for Multiple Output Thresholds. The false positives
identified in the lower threshold have been highlighted by red squares.

The sensitivity of these models to threshold variations was another critical
aspect of this study. Specifically, the increase in false positives at lower
threshold levels, such as 0.3, pointed to a trade-off between sensitivity and
precision. This phenomenon was particularly observable in the UNet model,
where a lower threshold led to the model being overly sensitive, erroneously
identifying certain non-flooded regions as flooded. This sensitivity is a
crucial operational factor, as it necessitates the careful calibration of
threshold settings to achieve an optimal balance for accurate flood detection.
The complete matrics across various thresholds have been presented in the
Appendix.

The decision to focus on the UNet model for the Sentinel dataset analysis was
influenced by its superior performance in the preliminary FloodNet dataset
evaluation. UNet’s ability to accurately segment flood-affected areas, as
evidenced by its visual outputs, along with its computational efficiency as
seen in II, positioned it as a promising model for further testing and
application in the different context of satellite imagery provided by the
Sentinel dataset.

Figure 7: Sample Outputs from Sentinel Data. Positive values indicate water and flooded regions. TABLE III: Performance Metrics on Sentinel Data (UNet) Metric | Value
---|---
IoU | 0.65117
Dice | 0.76080
Precision | 0.77475
Recall | 0.78513
F1 Score | 0.76080
Accuracy | 0.87711

Upon analyzing the UNet model with the Sentinel dataset, a clear improvement
was noted across all quantitative metrics as seen in III. This enhancement
indicated not only the model’s adaptability but also its effectiveness in a
new imaging context. However, the slight increase in inference time
underscored the necessity for models to be optimized for varying data
resolutions and complexities, as encountered in satellite versus aerial
imagery. Furthermore, the ablation studies on the Sentinel dataset revealed
potential vulnerabilities in the UNet model’s architecture, specifically in
instances where undefined values were observed. This aspect of the analysis
highlighted areas within the model’s architecture that could be optimized for
enhanced performance and reliability in real-world applications. Lastly, the
interpretability analysis through activation maps provided deeper insights
into the model’s processing behavior and response to satellite imagery,
indicating potential areas for performance enhancement and refinement.

A notable observation was the sparsity of activations in the encoder layers
when processing the Sentinel satellite imagery. This sparsity indicates that
the model, in its current architecture, selectively focuses on specific
features or regions within the satellite images. While this focused approach
suggests efficiency in identifying key features relevant to flood detection,
it also raises the possibility that the model might be overlooking other
potentially significant information.

  1. 1.

Visualization: No visualization provided, as the output is textual.

  2. 2.

Quantitative Metrics: The model showed high accuracy in standard scenarios but
struggled with ambiguous and non-existent locations. The geocoding service
successfully converted most locations to coordinates, except for fictional or
ambiguous ones.

  3. 3.

Case Study: For the sentence ”Tsunami alerts for the coast of Japan”, the
model returned ”Japan” instead of ”Coast of Japan”, indicating a limitation in
handling descriptive locations. In the case of ”Weather forecast for Mount
Everest”, the model incorrectly identified ”Mount Everest” as a valid location
for flood detection, showing a need for better filtering of irrelevant
queries.

  4. 4.

Analysis: The model performs well with clear, specific locations but has
limitations in understanding more complex or descriptive queries. The
inability to handle non-existent locations like ”Atlantis” indicates a need
for a validation layer before geocoding. The high success rate in standard
scenarios shows promise for practical application, with scope for refinement
in handling edge cases.

In an extensive evaluation of deep learning models including UNet, Residual
Dense Net (RDN), and Vision Transformer (ViT) across FloodNet and Sentinel
datasets, key insights have been gleaned regarding their efficacy in flood
detection. UNet’s performance, in particular, stood out with its superior
visual segmentation accuracy across both datasets, affirming its effectiveness
in delineating flood-affected areas. This was notable despite some
discrepancies between visual and quantitative data, underscoring UNet’s
potential as a reliable tool for real-world flood detection. Conversely, ViT,
while advanced in design, did not meet expectations in segmentation accuracy,
indicating a need for alignment between model architecture and the specific
requirements of flood detection tasks.

The study also highlighted the sensitivity of these models to threshold
settings, where a higher rate of false positives at lower thresholds was
particularly evident in the UNet model. This finding emphasizes the importance
of fine-tuning threshold settings to balance sensitivity and precision, which
is crucial for practical applications. Recommendations from the study include
optimizing UNet for broader deployment, carefully calibrating thresholds to
suit environmental conditions, and improving real-time processing for time-
sensitive scenarios. Additionally, exploring hybrid models that combine
strengths of different architectures, diversifying datasets, and collaborating
with interdisciplinary experts are suggested to enhance the models’
applicability and relevance in disaster management.

Finally, the interface model demonstrated proficiency in extracting and
geocoding locations from straightforward natural language inputs, but faced
few challenges with descriptive or ambiguous queries and in filtering out
irrelevant or fictional locations. This underscores the need for future
development to focus on enhancing the models’ understanding of complex queries
and incorporating validation mechanisms. Such advancements are critical for
improving the models’ functionality in diverse real-world flood and water
detection scenarios, meeting the dynamic challenges in environmental
monitoring and disaster management. This comprehensive study thus provides a
roadmap for refining these deep learning models, ensuring they are more
attuned to the nuanced requirements of flood detection and response.

##  V Conclusion

In conclusion, this research represents a significant stride in flood
management technology. By leveraging advanced deep learning models in tandem
with ChatGPT’s conversational AI, this project has successfully addressed some
of the most pressing challenges in real-time flood detection and mapping. The
system’s ability to analyze complex environmental data through aerial and
satellite imagery and provide accurate, timely flood information is a
noteworthy accomplishment.

This research addresses a critical need in the context of increasing climate
change impacts and urbanization. The integration of deep learning models such
as UNet with ChatGPT enhances the accuracy and efficiency of flood detection
over traditional methods, which often struggle with real-time data processing
and broad coverage. Furthermore, the user-friendly interface of ChatGPT opens
up this technology to a wider range of users, from disaster management
professionals to the general public.

The project’s experimental results demonstrate not only the feasibility of
such an integrated approach but also its potential to revolutionize the field
of environmental monitoring and disaster response. The system’s scalability
and adaptability to different environments and conditions underscore its
practicality and applicability in various geographical settings.

Looking ahead, future research can explore further enhancements to the system,
such as integrating additional environmental factors, improving prediction
accuracy, and expanding geographical coverage. The potential to adapt this
technology for other environmental monitoring applications, such as wildfire
detection or air quality monitoring, also presents exciting avenues for
further development.

This research underscores the vital role of technological innovation in
addressing environmental challenges. By advancing flood detection
capabilities, this project contributes significantly to improving disaster
preparedness and response, ultimately aiding in safeguarding communities and
ecosystems against the growing threat of floods.

## Acknowledgment

We would like to thank Dr. Zhe Jiang for his guidance and the University of
Florida for providing access to educational resources.

## References

  * [1] Qianyu Zhang, Nattha Jindapetch, and Dujdow Buranapanichkit.  Investigation of image edge detection techniques based flood monitoring in real-time.  In 2019 16th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pages 927–930, 2019.
  * [2] Angga Ariawan, Dwi Pebrianti, Ronny, Yudha Maulana Akbar, Lestari Margatama, and Luhur Bayuaji.  Image processing-based flood detection.  In Zainah Md Zain, Hamzah Ahmad, Dwi Pebrianti, Mahfuzah Mustafa, Nor Rul Hasma Abdullah, Rosdiyana Samad, and Maziyah Mat Noh, editors, Proceedings of the 10th National Technical Seminar on Underwater System Technology 2018, pages 371–380, Singapore, 2019. Springer Singapore.
  * [3] T. V. Bijeesh and K. N. Narasimhamurthy.  Surface water detection and delineation using remote sensing images: a review of methods and algorithms.  Sustainable Water Resources Management, 6(4):68, Jul 2020.
  * [4] L. Pulvirenti, N. Pierdicca, M. Chini, and L. Guerriero.  An algorithm for operational flood mapping from synthetic aperture radar (sar) data using fuzzy logic.  Natural Hazards and Earth System Sciences, 11(2):529–540, 2011.
  * [5] Ahad Hasan Tanim, Callum Blake McRae, Hassan Tavakol-Davani, and Erfan Goharian.  Flood detection in urban areas using satellite imagery and machine learning.  Water, 14(7), 2022.
  * [6] Snehil and Ruchi Goel.  Flood damage analysis using machine learning techniques.  Procedia Computer Science, 173:78–85, 2020.  International Conference on Smart Sustainable Intelligent Computing and Applications under ICITETM2020.
  * [7] Laura Giustarini, Renaud Hostache, Patrick Matgen, Guy J.-P. Schumann, Paul D. Bates, and David C. Mason.  A change detection approach to flood mapping in urban areas using terrasar-x.  IEEE Transactions on Geoscience and Remote Sensing, 51(4):2417–2430, 2013.
  * [8] Miao Xie, Zhe Jiang, and Arpan Man Sainju.  Geographical Hidden Markov Tree for Flood Extent Mapping (With Proof Appendix).  arXiv e-prints, page arXiv:1805.09757, May 2018.
  * [9] Anushree Rambhad, Dhirendra Pratap Singh, and Jaytrilok Choudhary.  Detection of flood events from satellite images using deep learning.  In Vikrant Bhateja, Xin-She Yang, Jerry Chun-Wei Lin, and Ranjita Das, editors, Intelligent Data Engineering and Analytics, pages 259–268, Singapore, 2023. Springer Nature Singapore.
  * [10] Xuan Wu, Zhijie Zhang, Shengqing Xiong, Wanchang Zhang, Jiakui Tang, Zhenghao Li, Bangsheng An, and Rui Li.  A near-real-time flood detection method based on deep learning and sar images.  Remote Sensing, 15(8), 2023.
  * [11] S. M. Jaisakthi, P. R. Dhanya, and S. Jitesh Kumar.  Detection of flooded regions from satellite images using modified unet.  In Vallidevi Krishnamurthy, Suresh Jaganathan, Kanchana Rajaram, and Saraswathi Shunmuganathan, editors, Computational Intelligence in Data Science, pages 167–174, Cham, 2021. Springer International Publishing.
  * [12] Zhouyayan Li and Ibrahim Demir.  U-net-based semantic classification for flood extent extraction using sar imagery and gee platform: A case study for 2019 central us flooding.  Science of The Total Environment, 869:161757, 2023.
  * [13] MD Samiul Islam, Xinyao Sun, Zheng Wang, and Irene Cheng.  Fapnet: Feature fusion with adaptive patch for flood-water detection and monitoring.  Sensors, 22(21), 2022.
  * [14] Ming Zhong, Hongrui Zhang, Tao Jiang, Jun Guo, Jinxin Zhu, Dagang Wang, and Xiaohong Chen.  A hybrid model combining the cama-flood model and deep learning methods for streamflow prediction.  Water Resources Management, 37, Sep 2023.
  * [15] Zhice Fang, Yi Wang, Ling Peng, and Haoyuan Hong.  Predicting flood susceptibility using lstm neural networks.  Journal of Hydrology, 594:125734, 2021.
  * [16] Hamed Farahmand, Yuanchang Xu, and Ali Mostafavi.  A spatial–temporal graph deep learning model for urban flood nowcasting leveraging heterogeneous community features.  Scientific Reports, 13(1):6768, Apr 2023.
  * [17] Ritu Yadav, Andrea Nascetti, Hossein Azizpour, and Yifang Ban.  Unsupervised Flood Detection on SAR Time Series.  arXiv e-prints, page arXiv:2212.03675, December 2022.
  * [18] Sayak Paul and Siddha Ganju.  Flood Segmentation on Sentinel-1 SAR Imagery with Semi-Supervised Learning.  arXiv e-prints, page arXiv:2107.08369, July 2021.
  * [19] Mingyang Yi, Jiacheng Sun, and Zhenguo Li.  On the Generalization of Diffusion Model.  arXiv e-prints, page arXiv:2305.14712, May 2023.
  * [20] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  arXiv e-prints, page arXiv:2006.11239, June 2020.
  * [21] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf.  SegDiff: Image Segmentation with Diffusion Probabilistic Models.  arXiv e-prints, page arXiv:2112.00390, December 2021.
  * [22] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  RePaint: Inpainting using Denoising Diffusion Probabilistic Models.  arXiv e-prints, page arXiv:2201.09865, January 2022.
  * [23] Junde Wu, Rao Fu, Huihui Fang, Yu Zhang, Yehui Yang, Haoyi Xiong, Huiying Liu, and Yanwu Xu.  MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model.  arXiv e-prints, page arXiv:2211.00611, November 2022.
  * [24] U.S. Geological Survey.  Waterwatch – maps and graphs of current water resources conditions, 2023.  Accessed: 2023-12-15.
  * [25] Z. Kugler and T. De Groeve.  The global flood detection system.  EUR 23303 EN JRC44149, Office for Official Publications of the European Communities, Luxembourg, 2007.
  * [26] NASA Earthdata.  Modis nrt global flood product.  NASA Earthdata, 2023.  Accessed: 2023-12-15.
  * [27] Chengxiu Li, Jadunandan Dash, Moses Asamoah, Justin Sheffield, Mawuli Dzodzomenyo, Solomon Hailu Gebrechorkos, Daniela Anghileri, and Jim Wright.  Increased flooded area and exposure in the white volta river basin in western africa, identified from multi-source remote sensing data.  Scientific Reports, 12(1):3701, Mar 2022.
  * [28] Author’s Name.  Title of the project.  Google Earth Engine Code Editor, 2023.  Accessed: 2023-12-15.
  * [29] Maxar Technologies.  Maxar open data program.  Maxar Technologies, 2023.  Accessed: 2023-12-15.
  * [30] OpenAI.  Research overview.  OpenAI, 2023.  Accessed: 2023-12-15.
  * [31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  arXiv e-prints, page arXiv:1505.04597, May 2015.
  * [32] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual Dense Network for Image Restoration.  arXiv e-prints, page arXiv:1812.10477, December 2018.
  * [33] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv e-prints, page arXiv:2010.11929, October 2020.
  * [34] Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy.  Floodnet: A high resolution aerial imagery dataset for post flood scene understanding.  IEEE Access, 9:89644–89654, 2021.
  * [35] Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Murphy.  Floodnet: A high resolution aerial imagery dataset for post flood scene understanding.  arXiv preprint arXiv:2012.02951, 2020.
  * [36] OpenAI.  Chatbot interaction.  ChatGPT powered by OpenAI, 2023.  Accessed: 2023-04-30.
  * [37] Sinergise Ltd.  Sentinel hub satellite imagery engine.  Available online: https://www.sentinel-hub.com, Year of Access.  Accessed on: [Access Date].

##  VI Appendix

###  VI-A Preliminary Results :On Aerial Drone Flood Images

TABLE IV: UNet Performance Metrics Metric | Threshold 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---
IoU | 0.43301 | 0.42650 | 0.42273 | 0.41835 | 0.41302
Dice | 0.49098 | 0.48274 | 0.47872 | 0.47429 | 0.46890
Precision | 0.57501 | 0.56790 | 0.57269 | 0.57329 | 0.57311
Recall | 0.51038 | 0.49959 | 0.49207 | 0.48446 | 0.47623
F1 Score | 0.49098 | 0.48274 | 0.47872 | 0.47429 | 0.46890
Accuracy | 0.90683 | 0.90672 | 0.90674 | 0.90644 | 0.90575
TABLE V: Residual Dense Net Performance Metrics Metric | Threshold 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---
IoU | 0.49718 | 0.49627 | 0.49412 | 0.49215 | 0.48899
Dice | 0.56819 | 0.56676 | 0.56414 | 0.56174 | 0.55899
Precision | 0.56014 | 0.56697 | 0.57131 | 0.57582 | 0.58092
Recall | 0.61566 | 0.60475 | 0.59470 | 0.58495 | 0.57423
F1 Score | 0.56819 | 0.56676 | 0.56414 | 0.56174 | 0.55899
Accuracy | 0.90709 | 0.90868 | 0.90969 | 0.91063 | 0.91081
TABLE VI: Vision Transformer Performance Metrics Metric | Threshold 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---
IoU | 0.46295 | 0.45869 | 0.45401 | 0.44926 | 0.44506
Dice | 0.53512 | 0.53039 | 0.52523 | 0.51969 | 0.51530
Precision | 0.67567 | 0.67880 | 0.68377 | 0.68568 | 0.63724
Recall | 0.53205 | 0.52533 | 0.51801 | 0.51019 | 0.50293
F1 Score | 0.53512 | 0.53039 | 0.52523 | 0.51969 | 0.51530
Accuracy | 0.89621 | 0.89532 | 0.89435 | 0.89332 | 0.89238
TABLE VII: Ablation Study Results for RDN, ViT, and UNet Models on FloodNet Dataset Model | Ablated Part | Precision | Recall | F1-Score
---|---|---|---|---
RDN | Block 0 Layer 0 | 0.5113 | 0.5812 | 0.5239
RDN | Block 0 Layer 1 | 0.5090 | 0.5842 | 0.5276
RDN | Block 0 Layer 2 | 0.5136 | 0.5541 | 0.5120
RDN | Block 1 Layer 0 | 0.4476 | 0.5686 | 0.4639
RDN | Block 1 Layer 1 | 0.4438 | 0.5040 | 0.4382
RDN | Block 1 Layer 2 | 0.4220 | 0.5580 | 0.4425
ViT | Decoder Layers 0,1,2,3 | 0.0 | 0.0 | 0.0
UNet | enc1 | 0.1842 | 0.8 | 0.2719


# RubCSG at SemEval-2022 Task 5: Ensemble learning for identifying misogynous
MEMEs

Wentao Yu, Benedikt Boenninghoff, Jonas Roehrig, Dorothea Kolossa
Institute of Communication Acoustics, Ruhr University Bochum, Germany
{wentao.yu, benedikt.boenninghoff,
jonas.roehrig<EMAIL_ADDRESS>
###### Abstract

This work presents an ensemble system based on various uni-modal and bi-modal
model architectures developed for the SemEval 2022 Task 5: MAMI-Multimedia
Automatic Misogyny Identification. The challenge organizers provide an English
meme dataset to develop and train systems for identifying and classifying
misogynous memes. More precisely, the competition is separated into two sub-
tasks: sub-task A asks for a binary decision as to whether a meme expresses
misogyny, while sub-task B is to classify misogynous memes into the
potentially overlapping sub-categories of stereotype, shaming,
objectification, and violence. For our submission, we implement a new model
fusion network and employ an ensemble learning approach for better
performance. With this structure, we achieve a 0.755 macro-average F1-score
(11th) in sub-task A and a 0.709 weighted-average F1-score (10th) in sub-task
B.111Code available at: https://github.com/rub-ksv/SemEval-Task5-MAMI.

RubCSG at SemEval-2022 Task 5: Ensemble learning for identifying misogynous
MEMEs

Wentao Yu, Benedikt Boenninghoff, Jonas Roehrig, Dorothea Kolossa Institute of
Communication Acoustics, Ruhr University Bochum, Germany {wentao.yu,
benedikt.boenninghoff, jonas.roehrig<EMAIL_ADDRESS>

##  1 Introduction

Hate speech against women remains rampant despite many efforts at education,
prevention and blocking. Misogyny takes place online and offline. Especially
on social media platforms, misogyny appears in different forms and has serious
implications Chetty and Alathur (2018). Currently, automated detection and
filtering seem to be the most effective way to prevent hate speech online.
However, over the past few years, the rising popularity of memes brought
misogyny to a new multi-modal form, which may be more likely to go viral due
to their often surprising combinations of text and image that may strike
viewers as funny and hence, as eminently shareable.

The multi-modality of memes also makes automatic detection more challenging.
Some memes express their hatred implicitly or through juxtaposition, so they
may even appear harmless when considering the text or the image in isolation.
SemEval-5 2022 Multimedia Automatic Misogyny Identification (MAMI) Fersini et
al. (2022) aims to identify and classify English misogynous memes.

In recent years, the Transformer model Vaswani et al. (2017) has been widely
used in natural language processing (NLP) and image processing. Transfer
learning Torrey and Shavlik (2010) with a pre-trained Transformer model can
save training resources and increase efficiency with less training data Wang
et al. (2020).

Therefore, in this work, we consider transfer learning to customize three uni-
modal models based on the Transformer model: i) fine-tuning a pre-trained
RoBERTa model for classification (BERTC) Liu et al. (2019); ii) training a
graph convolutional attention network (GCAN) using the pre-trained RoBERTa
model for word embedding; iii) fine-tuning a pre-trained image model, the
Vision Transformer (ViT) Dosovitskiy et al. (2020). Based on these three uni-
modal models, four bi-modal models are trained through our proposed model
fusion network, namely BERTC-ViT, GCAN-ViT, BERTC-GCAN, and BERTC-GCAN-ViT.
All models are evaluated with 10-fold cross-validation. The macro-average and
weighted-average F1-scores are employed as the metrics for the sub-tasks.
Ultimately, the ensemble strategy is applied on both the dataset- and the
model-level (detailed in Section 3.3) for better performance.

The remainder of the paper is structured as follows: Section 2 introduces the
MAMI challenge and related solutions to the task. Our ensemble model is
described in Section 3, followed by the experimental setup in Section 4.
Finally, our results are shown and conclusions are drawn in Sections 5 and 6.

##  2 Background

The MAMI dataset contains 10,000 memes as the training and 1,000 memes as the
test set; all of these are given together with the text transcription as
obtained through optical character recognition (OCR). The reference labels are
obtained by manual annotation via a crowdsourcing platform.

The challenge is composed of two sub-tasks: Sub-task A represents a binary
classification task and focuses on the identification of misogynous memes, so
each meme should be classified as not misogynous (noMis) or misogynous (Mis).
Sub-task B, in contrast, presents a multi-label classification task, where the
misogynous memes should be grouped further, into four potentially overlapping
categories. The dataset class distribution is illustrated in Table 1.

Table 1: MAMI-22 dataset class distribution. Mis: misogynous; Shm: shaming; Ste: stereotype; Obj: objectification; Vio: violence. Sets | Mis | Shm | Ste | Obj | Vio
---|---|---|---|---|---
training set | 5000 | 1274 | 2810 | 2202 | 953
test set | 500 | 146 | 350 | 348 | 153

Since the provided dataset contains two modalities (namely, images and texts),
an automated approach requires integrating the information from the images
with the textual information. However, the OCR-based transcriptions are quite
error prone, while the images are often hard to recognize for automatic
systems, due, among other reasons, to overlaid text and to the popularity of
further changes, such as the composition of multiple sub-images. Consequently,
it is challenging to identify the pertinent information of the respective
modalities, in order to merge it into a joint classification decision.

Some researchers have already worked on meme datasets. For example, Sabat et
al. (2019) created a hateful memes database, using the BERT model to extract a
contextual text representation and the VGG-16 convolutional neural network
Simonyan and Zisserman (2014) for image features. Then, text and image
representations are concatenated to obtain a multi-modal representation.
Facebook also organized a challenge for the identification of hateful memes in
2020 Kiela et al. (2020). The winner of this challenge adopted an ensemble
system with four different visual-linguistic transformer architectures Zhu
(2020).

The Transformer model has shown excellent performance in many tasks, and it
also shows promising results in the above studies, based on its use of the
attention mechanism to extract the contextual information within a text.
However, its ability to capture global information about the vocabulary of a
language remains limited Lu et al. (2020), and we hypothesize that this is
even more of an issue in the task at hand, due to the very short texts in the
given challenge.

For this reason, we combine a Transformer model with a graph convolutional
network (GCN) Yao et al. (2019), which may help to address this issue. GCNs
can be understood as a generalization of CNNs, where the data has graph
structure and locality is defined by the connectivity of the graph. As input,
a GCN receives features that connect to a set of nodes. From layer to layer,
the features of a node are updated as weighted combinations of its neighbors'
features. In our case, the graph is defined as follows: There is a node for
every word in the vocabulary and for every document. The collection of nodes
is $V=\left\\{D_{1},D_{2}\cdots D_{n_{D}},W_{1},W_{2},\cdots
W_{n_{w}}\right\\}$, where $D_{i}$ and $W_{i}$ indicate the document and word
nodes, respectively. $n_{D}$ is the number of documents and $n_{W}$ is the
number of unique words in the corpus. The edges between word nodes are
weighted with the word co-occurrence, the edges between document-word pairs
are weighted with the term frequency-inverse document frequency (TF-IDF).

A fixed-size sliding window with step size 1 is used to gather the word co-
occurrence information through the entire dataset. The point-wise mutual
information (PMI) is employed to measure the relationship between the words i
and j as follows:

$\begin{matrix}\textrm{PMI}(i,j)=\textrm{log}\frac{p(i,j)}{p(i)p(j)},\\\
p(i,j)=\frac{N(i,j)}{N},\\\ p(i)=\frac{N(i)}{N},\end{matrix}$ (1)

where $N(i)$ counts the sliding windows in the training set that contain word
$i$, $N(i,j)$ is the number of sliding windows that carry both words $i$ and
$j$, and $N$ is the total number of sliding windows in the corpus. As
described in Yao et al. (2019), a positive PMI value indicates a high semantic
correlation of words in corpus and vice versa.

The adjacency matrix A of the graph is then computed elementwise, as follows:

$A_{i,j}=\left\\{\begin{matrix}\textrm{PMI}(i,j)&i,j\textrm{ are word nodes,
}\textrm{PMI}(i,j)>0;\\\ &n_{D}<i,j\leqslant n_{D}+n_{W}\\\ \textrm{TF-
IDF}_{i,j}&\textrm{document node }i\textrm{ and word node }j;\\\ &i\leqslant
n_{D};n_{D}<j\leqslant n_{D}+n_{W}\\\ 1&i=j\\\
0&\textrm{otherwise}\end{matrix}\right.$ (2)

Since the graph is undirected, the adjacency matrix is symmetric. Finally, the
adjacency matrix is normalized by
$\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}=\textbf{{D}}^{-\frac{1}{2}}\textbf{{A}}\textbf{{D}}^{-\frac{1}{2}}$,
where D is the degree matrix of A. The normalized adjacency matrix
$\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}$
is used to weight the graph node features, cf. Section 3.1. A PyTorch
implementation based on Text-GCN Yao et al. (2019), as provided on
GitHub222https://github.com/codeKgu/Text-GCN, was used for the implementation.

##  3 System Overview

In this section, we specify our uni- and bi-modal models.

(a) BERTC

(b) GCAN

(c) ViT

Figure 1: Uni-modal models, where $\textrm{L}_{S}$ is the sequence length,
which depends on the RoBERTa tokenizer.

Figure 1 depicts the three uni-modal models BERTC (1(a)), GCAN (1(b)), and ViT
(1(c)), which form the basis of our further experiments. The bi-modal models
are constructed based on trained uni-modal models and our proposed model
fusion network, which is further detailed in Section 3.2. Finally, we apply
soft and hard voting ensembles on the trained candidate models.

###  3.1 Uni-modal models

As illustrated in Figure 1, every uni-modal model has two outputs: the
classification probabilities $\textbf{{p}}_{\textrm{{i}}}$ and the
classification features $\textrm{{f}}_{\textrm{{i}}}$. All classifier blocks
in our models have the following, identical structure: a fully connected layer
reduces the feature dimension to half the input dimension, followed by a ReLU
activation and a dropout layer. Ultimately, an output layer projects the
features to the output dimension $n$, and a sigmoid function squashes the
range of the output vector components to $(0,1)$, allowing for an
interpretation as a vector of label probabilities, with possible overlap in
categories.

BERTC: We fine-tune a pre-trained large RoBERTa language model (roberta-large)
for classification. The text input is encoded by the RoBERTa model with the
embedding dimension 1024. The Pooler layer returns the first classification
token [cls] embedding $\textrm{{f}}_{\textrm{{BERTC}}}$ and feeds it into the
classifier to obtain the probabilities $\textbf{{p}}_{\textrm{{BERTC}}}$.

GCAN: Again, a pre-trained RoBERTa model extracts contextual text information.
Each token is considered as a word node and each meme is a document node. Thus
the word node representation is given by the corresponding RoBERTa word
embedding vector. We denote the input embedding sequence of document $k$ as
$\textbf{{x}}^{k}=[\textbf{{x}}_{1}^{k},\textbf{{x}}_{2}^{k},\cdots\textbf{{x}}_{\textrm{L}_{S}}^{k}]$,
where $\textbf{{x}}_{i}^{k}$, $i\in\\{1,\ldots,\textrm{L}_{S}\\}$ is a
1024-dimensional embedding vector of the $i$-th token. As depicted in Figure
1(b), $\textbf{{x}}^{k}$ is an $\textrm{L}_{S}\times$1024 matrix. The first
classification token [cls] embedding represents the document classification
information. Thus, we use the document-word co-occurrence information TF-IDF
as the edge weights for the [cls] embedding. All other token embeddings are
weighted with the word co-occurrence information PMI.

For each document $k$, we extract its specific adjacency matrix
${\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}}_{k}$
from the complete adjacency matrix
${\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}}$
by reducing it to rows and columns of all the document and word nodes ($i$ and
$j$ in Equation 2) that are present in this document. The extracted document
adjacency matrix
${\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}}_{k}$
is an $\textrm{L}_{S}\times\textrm{L}_{S}$ matrix.

The GCAN block in Figure 1(b) adopts the multi-head self-attention mechanism
in 3 successive GCAN layers to embed the node representations. The queries Q,
keys K and values V are identical and set to the respective layer input. The
first layer input is given by the RoBERTa word embeddings $\textbf{{x}}^{k}$
of the input text. The attention of head $j$ is obtained by

$\bm{\alpha}_{j}=\textrm{softmax}\left(\frac{\left(\textbf{W}_{j}^{Q}\textbf{Q}^{T}\right)^{T}\left(\textbf{W}_{j}^{K}\textbf{K}^{T}\right)}{\sqrt{d_{k}}}\right)\left(\textbf{W}_{j}^{V}\textbf{V}^{T}\right)^{T}$
(3)

where $\textbf{W}_{j}^{\ast}$ are learned parameters for the queries Q, keys
K, and values V, respectively. A superscript $T$ denotes the transpose;
$d_{k}=\frac{d_{att}}{h}$, $d_{att}$ is the attention dimension and $h$ is the
number of attention heads. Having computed the multi-head self-attention, each
attention head output is multiplied by the document adjacency matrix
${\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}}_{k}$

$\overset{\sim}{\bm{{\alpha}}}_{j}={\stackrel{{\scriptstyle\sim}}{{\smash{\textbf{{A}}}\rule{0.0pt}{4.73611pt}}}}_{k}\bm{\alpha}_{j}.$
(4)

Equation 5 describes the output $\bm{\alpha}$ of the GCAN layer: The weighted
outcomes all heads are concatenated (concat), and a fully connected layer (FC)
projects the representation to the attention dimension. Inspired by Veličković
et al. (2017), instead of concatenating the weighted attention head outputs,
we employ averaging (avg) to fuse these weighted outputs in the last GCAN
layer. A fully connected layer again projects the final representation to the
attention dimension. Thus, after the GCAN block, the text representation is
still an $\textrm{L}_{S}\times$1024 matrix. The document classification
feature vector $\textrm{{f}}_{\textrm{{GCAN}}}$ is obtained by summing all
node representations.

$\bm{\alpha}=\left\\{\begin{matrix}\textrm{FC}\left(\textrm{concat}\left(\overset{\sim}{\bm{{\alpha}}}_{1},\cdots\overset{\sim}{\bm{{\alpha}}}_{h}\right)\right)&\textrm{not
in last layer}\\\
\textrm{FC}\left(\textrm{avg}\left(\overset{\sim}{\bm{{\alpha}}}_{1},\cdots\overset{\sim}{\bm{{\alpha}}}_{h}\right)\right)&\textrm{in
last layer}\\\ \end{matrix}\right.$ (5)

ViT: To extract the visual contextual information, we utilize the pre-trained
ViT model vit-large-patch16-224 to encode the input image. For this purpose,
the input image is split into fixed-size patches, and a linear projection of
the flattened patches is used to obtain the patch embedding vectors. The
Transformer encoder transforms the embedding vectors. Finally, the embedding
$\textrm{{f}}_{\textrm{{ViT}}}$ of the first classification token, [cls], is
fed to the classifier to obtain the prediction probabilities
$\textbf{{p}}_{\textrm{{ViT}}}$.

###  3.2 Bi-modal models

Figure 2 shows our fusion model structure.

Figure 2: Fusion model structure

Each model $\textrm{M}_{i}$ has two outputs: the vector of its classification
probabilities $\textbf{{p}}_{i}$ and the classification features
$\textrm{{f}}_{i}$. We concatenate the model classification probabilities and
features as a multi-modal representation to make the final decision.

Two fusion strategies—stream-weighting-based decision fusion and
representation fusion—are considered. The weight predictor and the classifier
in Figure 2 both have the same structure as the classifier block in Figure 1.
The weight predictor output dimension is the number $m$ of candidate models
for fusion. The stream weighting probability $\textbf{{p}}_{sw}$ is obtained
through a weighted combination of the class probability vectors of all uni-
modal model outcome probabilities, i.e.

$\textbf{{p}}_{sw}=\sum_{i}\textbf{{p}}_{i}\cdot w_{i}.$ (6)

The classifier output dimension is the same as the number of classes $n$. A
sigmoid function computes the representation fusion probabilities
$\textbf{{p}}_{rf}$ from the combined multi-modal representation. Finally, we
average the stream weighting and the representation fusion probabilities. The
following model combinations are attempted, where $\textrm{M}_{i}$,
$i\in{\\{1,2,3\\}}$ is the $i$-th pre-trained uni-modal model.

Bi-modal model | $\textrm{M}_{1}$ | $\textrm{M}_{2}$ | $\textrm{M}_{3}$
---|---|---|---
BERTC-ViT | BERTC | ViT | -
GCAN-ViT | GCAN | ViT | -
BERTC-GCAN | BERTC | GCAN | -
BERTC-GCAN-ViT | BERTC | GCAN | ViT

###  3.3 Ensemble learning

Having established a number of possible uni-modal and bi-modal models, we now
combine these trained models into ensembles. It has been reported in many
studies that ensemble learning can enhance performance in comparison to single
learners Onan et al. (2016); Zhu (2020); Gomes et al. (2017). Therefore, we
consider soft and hard voting ensemble approaches.

We use the Python sklearn package333https://github.com/scikit-learn/scikit-
learn for 10-fold cross-validation. Thus, each model structure was trained ten
times with different inner test sets. Finally, these ten models are used to
evaluate the official test set and deliver ten predictions for every sample.
The soft voting ensemble method is implemented as follows:
$\textbf{{p}}_{\textrm{M}_{i}}$, the ensemble probabilities that are used in
the overall class decisions, are computed via

$\textbf{{p}}_{\textrm{M}_{i}}=\sum_{j=0}^{9}w_{\textrm{M}_{i}}^{j}\cdot\textbf{{p}}_{\textrm{M}_{i}}^{j}.$
(7)

Here, $\textbf{{p}}_{\textrm{M}_{i}}^{j}$ denotes the probabilities of model
$\textrm{M}_{i}$ in the $j$-th fold. The weights $w_{\textrm{M}_{i}}^{j}$ are
computed by

$w_{\textrm{M}_{i}}^{j}=\frac{\textrm{F1}_{\textrm{M}_{i}}^{j}}{\sum_{f}\textrm{F1}_{\textrm{M}_{i}}^{f}}.$
(8)

$\textrm{F1}_{\textrm{M}_{i}}^{j}$ corresponds with the best F1-score of model
$\textrm{M}_{i}$ over all epochs, computed on the inner test set in fold $j$.
This soft voting ensemble, using the same model structure, but with the
multiple outcomes from 10-fold cross-validation, is referred to as a _dataset-
level ensemble_ in the following.

The second type of ensemble—the _model-level ensemble_ —is constructed from
the _dataset-level ensemble_ results of each model. We use a hard voting
strategy with seven candidate models (BERTC, GCAN, ViT, BERTC-ViT, GCAN-ViT,
BERTC-GCAN, and BERTC-GCAN-ViT). In this approach, we set the final prediction
for a data point to one, if at least half of the considered models vote one,
making it a simple majority-voting strategy.

##  4 Experimental Setup

In this section, we describe our data processing and training pipeline in more
detail.

###  4.1 Data pre-processing

The challenge dataset provides a transcription text stream that was obtained
via OCR. Via image captioning, we derive a second text stream that contains a
description of the image in a few words.

For the OCR text, we first use the Python ftfy
package444https://github.com/rspeer/python-ftfy to fix the garbled sequences
that result from unexpected encodings (the _mojibake_) like "à¶´à¶§à·". Next,
all "@", "#" symbols and website addresses are removed from the text. The
emojis are converted to text form by the Python emoji
package555https://github.com/carpedm20/emoji. Finally, we remove non-English
characters and convert the text to lowercase.

For image captioning, we utilize a pre-trained encoder-decoder attention model
Xu et al. (2016)666https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-
Captioning. Although the translation from image to text is not very accurate,
most likely owing to issues like the overlaid meme text, it was nonetheless
beneficial for our classification task. We found that the description becomes
more precise, when we split the memes into their constituent sub-images where
applicable. In that case, the image caption is extracted over every sub-image
as well as the entire meme. Finally, the image captions are combined with the
word "and" and then concatenated with the OCR text, separated by ". ". With
this rule, the final text of the meme in Figure 3 is: "mgo ci aindo make make
me sandwich!!. a couple of baseball players standing next to each other and a
woman holding a sign in front of a sign and a woman standing next to a group
of people."

(a) a woman holding a sign in front of a sign

(b) a couple of baseball players standing next to each other

(c) a woman standing next to a group of people

Figure 3: In (a) and (b), we see "sub-images" and corresponding captions. (c)
shows the meme and its caption (when not considering the sub-image structure).

We use the entire meme as the image input for ViT. All memes are first resized
to 256$\times$256 and center-cropped to 224$\times$224 dimensions. The ViT
model uses all 3 RGB channels, so we retain the RGB structure, thus the input
image dimension is 3$\times$224$\times$224\. We regularize the entire image
database to range 0 to 1, then normalize each individual image to have zero
mean and unit variance.

###  4.2 Loss function

We decided to use the binary cross-entropy (BCE) loss for both subtasks.

Due to the imbalance in the class distributions (see Table 1), in sub-task B,
we weighted the class-specific loss terms by their support as follows:

$w_{c}=\frac{\frac{\textrm{NoS}}{\textrm{NoS}(c)}}{\sum_{c^{\prime}}\frac{\textrm{NoS}}{\textrm{NoS}(c^{\prime})}},c\in[\textrm{Shm},\textrm{Ste},\textrm{Obj},\textrm{Vio}]$
(9)

where NoS is the total number of samples in the training set and
$\textrm{NoS}(c)$ represents the number of true instances for class $c$. The
loss is then computed through the weighted combination of the single BCE
terms:

$\mathcal{L}_{1}=\sum_{c}w_{c}\cdot\textrm{BCE}(\textbf{{p}}_{c}^{B},\textbf{{y}}_{c}^{B}).$
(10)

Here, $\textbf{{p}}_{c}^{B}$ represents the system’s output probability of
class $c$ and $\textbf{{y}}_{c}^{B}$ is the binary ground truth for sub-task
B.

Additionally, we employ a teacher forcing loss to connect both subtasks. The
idea is that an instance should be identified as misogynous and possibly
grouped into sub-categories simultaneously. The teacher forcing loss is
defined as:

$\mathcal{L}_{2}=\lVert\textbf{{p}}^{A}-\textbf{{y}}^{A}\rVert,$ (11)

where the system’s output probability for sub-task A is determined as:

$\textbf{{p}}^{A}=\textrm{max}\big{(}\textbf{{p}}_{{}_{\textrm{Shm}}}^{B},\textbf{{p}}_{{}_{\textrm{Ste}}}^{B},\textbf{{p}}_{{}_{\textrm{Obj}}}^{B},\textbf{{p}}_{{}_{\textrm{Vio}}}^{B}\big{)}.$
(12)

The final loss is computed by

$\mathcal{L}=0.7\cdot\mathcal{L}_{1}+0.3\cdot\mathcal{L}_{2}.$ (13)

###  4.3 Model training

All models are trained using the PyTorch library Paszke et al. (2019) for 50
epochs. The AdamW optimizer Loshchilov and Hutter (2017) is used for
backpropagation, using a linear learning rate scheduler with a warm-up to
adapt the learning rate during the first four epochs in the training stage.
The dropout rate is 0.5. The RoBERTa model parameters in the BERTC and the
GCAN model are optimized separately.

In our GCAN model, the adjacency matrix is computed with a sliding window of
length 10. An 8-head self-attention is applied over 3 GCAN layers with an
attention dimension of 1024.

For all uni-modal models, the batch size is 16 and the initial learning rate
is $2\cdot 10^{-5}$. The RoBERTa and ViT block parameters in Figure 1 are also
fine-tuned. The bi-modal models are trained based on the pre-trained uni-modal
models. Here, we choose the batch size as 32, the initial learning rate is
$5\cdot 10^{-6}$. As the RoBERTa and ViT block parameters in Figure 1 are
already updated during the uni-modal training stage, we froze these parameters
in bi-modal re-training.

To avoid overfitting, we adopt early stopping to exit the training process
when the computed F1-score on the inner test set does not increase over 4
epochs. Inspired by Huang et al. (2017), we finally averaged those two epoch-
wise model parameters, which had the highest validation F1-score during the
training stage.

The models have the same structure for sub-tasks A and B. The only differences
are that in sub-task A, the classifier output dimension $n$ is $1$, and the
BCE is used as the loss function (_Setup A_), whereas in sub-task B, the
classifier output dimension $n$ equals $4$ and training uses the weighted BCE
with teacher forcing (Equation 13) as the loss function (_Setup B_). All
models are trained using NVIDIA’s Volta-based DGX-1 multi-GPU system, using 3
Tesla V100 GPUs with 32 GB memory each.

##  5 Results

In summary, we investigated two configurations, displayed in Table 2. _Setup
A_ represents the binary classification for sub-task A, resulting in an output
dimension $n=1$. _Setup B_ additionally deals with the multi-label
classification of sub-task B, returning an output of dimension $n=4$. All
results are evaluated on the official test set.

Setup | Task | Dimension | Loss
---|---|---|---
_Setup A_ | sub-task A | $n=1$ | BCE
_Setup B_ | sub-tasks A/B | $n=4$ |  | Weighted BCE &
---
Teacher Forcing
Table 2: Summary of the considered configurations.

###  5.1 Results for _Setup A_ (Sub-task A)

In the first stage, we trained three different uni-modal models (i.e., BERTC,
GCAN, and ViT). In the second stage, we optimized the bi-modal models (i.e.,
BERTC-ViT, GCAN-ViT, BERTC-GCAN, and BERTC-GCAN-ViT). The evaluation results
in terms of the macro-average F1-score are displayed in Figure 4 and Table 3,
showing the performance in identifying misogynous memes. To assess the
statistical significance of performance differences, we applied a 10-fold
cross validation and computed the Mann-Whitney-U test Mann and Whitney (1947).

Figure 4: Macro-average F1-scores for sub-task A based on 10-fold cross
validation. Asterisks indicate a statistically significant difference, where
** denotes 1e-04 < p <= 1e-03, * corresponds to 1e-02 < p <= 5e-02, and ns
indicates results where p > 5e-02.

As we can see, the text-only models (BERTC and GCAN) generally show a superior
performance compared to the image-only model (ViT). The results in Figure 4
clearly indicate robust performance for our bi-modal models. They are more
accurate and robust. In summary, the GCAN-ViT model yields the best results
w.r.t. the reported median F1-score.

Model | Ensemble | Model | Ensemble
---|---|---|---
BERTC | 0.663 | GCAN-ViT | 0.707
GCAN | 0.674 | BERTC-GCAN | 0.677
ViT | 0.619 |  | BERTC-
---
GCAN-ViT
0.689
BERTC-ViT | 0.697 | - | -
Table 3: Macro-average F1-scores of soft voting ensembles for sub-task A.

Table 3 lists the averaged F1-scores for soft voting ensembles, obtained by
combining all learned models from the 10-fold cross-validations. The results
show that our GCAN-ViT model outperforms all other models, achieving an
F1-score of 0.707.

###  5.2 Results for _Setup B_ (Sub-tasks A/B)

Next, we addressed sub-task B, i.e. to classify the misogynous memes into
four, potentially overlapping, categories. Similar to _Setup A_ , we trained
the same uni- and bi-modal models, but incorporating a different loss (see
Table 2). For sub-task B, the weighted-average F1-score is applied. The
results are presented in Figure 5.

(a) Results for sub-task A.

(b) Results for sub-task B.

Figure 5: Performance for _Setup B_. The notation is defined in Figure 4.

Interestingly, the models optimized for sub-task B also perform better for
sub-task A. In this case, we set the estimated label "misogynous" to 1 if at
least one of the labels for "shaming", "stereotype", "objectification", or
"violence" is 1.

Figure 5(a) depicts the sub-task A results while Figure 5(b) shows the
corresponding performance for sub-task B. Again, we see that the bi-modal
model GCAN-ViT outperforms all other models.

In addition, Tables 4 and 5 show the results for soft and hard voting
ensembles. By comparing Table 4 with Table 3 (both tables represent soft
voting results), we observe significantly improved F1-scores for _Setup B_.

Model | Sub-task A | Sub-task B
---|---|---
BERTC | 0.714 | 0.684
GCAN | 0.725 | 0.695
ViT | 0.666 | 0.641
BERTC-ViT | 0.746 | 0.692
GCAN-ViT | 0.758 | 0.704
BERTC-GCAN | 0.724 | 0.696
BERTC-GCAN-ViT | 0.755 | 0.704
Table 4: F1-scores of soft voting ensembles for _Setup B_ (sub-tasks A and B). Combination | Sub-task A | Sub-task B
---|---|---
Three uni-modal models | 0.728 | 0.698
Four bi-modal models | 0.752 | 0.709
All seven models | 0.755 | 0.706
Oracle model combination | 0.762 | 0.716
Table 5: Model-level hard voting ensemble performance with _Setup B_ for sub-
task A and B.

As a last experiment, we applied hard voting on the ensembles. Again, sub-task
A results are derived from sub-task B.

Table 5 shows the results of different combinations. Generally, the
combination of the four bi-modal models in the 2nd row outperforms a
combination of three uni-modal models in the 1st row. If we combine all uni-
and bi-modal models (3rd row), the F1-score is 0.755 for sub-task A, and 0.706
for sub-task B.

The results in bold print represent our submitted approaches for both sub-
tasks, showing an F1-score of 0.755 for sub-task A and 0.709 for sub-task B.

After the challenge ended, we again evaluated all possible subset combinations
of the seven candidate models. The followed combinations give the best
achievable results by knowing the official test set reference labels: ViT,
BERTC-GCAN-ViT, BERTC-ViT, GCAN-ViT achieves an F1-score of 0.762 for sub-task
A, while an ensemble consisting of BERTC-ViT and BERTC-GCAN-ViT yields an
F1-score 0.716 on sub-task B. These results are shown for comparison in the
final row of Table 5 as oracle results.

##  6 Conclusion

This paper presents our ensemble-based approach to address two sub-tasks of
the SemEval-2022 MAMI competition. The challenge aims to identify misogynous
memes and classify them into—potentially overlapping—categories. We train
different text models, an image model, and via our proposed fusion network, we
combine these in a number of different bi-modal models.

Among the uni-modal systems, all text models show a far better performance
than the image model. As expected, our proposed graph convolutional attention
network (GCAN), which also considers the graph structure of the input data
while using pre-trained RoBERTa word embeddings as node features, consistently
outperforms the pre-trained RoBERTa model.

The proposed fusion network further improves the performance by combining the
ideas of stream-weighting and representation fusion. We additionally adopt
10-fold cross-validation and use a dataset-level soft voting ensemble to
obtain better and more robust results. Finally, our model-level hard voting
ensemble integrates the soft voting ensemble predictions of our best uni- and
bi-modal models. Our experiments indicate that this layered ensemble approach
can significantly improve the model accuracy. Ultimately, our submitted system
results in an F1-score of 0.755 for sub-task A and 0.709 for sub-task B.

Overall, we believe that the identification of misogyny in memes is best
addressed through bi-modal recognition, considering both textual and image
information. Concerning the text-based classification, we found a graph
convolutional attention neural network to be beneficial as an integrative
model for Transformer embeddings. This helps in the text classification, when
the documents are short, as for the given meme classification task.

To cope with the bi-modality of the task at hand, we have implemented a range
of systems for integrating the information from both streams. An idea that
proved to be effective here was that of bringing together the strengths of
early fusion and decision fusion in a joint framework. This allowed us to
dynamically adjust the contributions of the two modalities through dynamic
stream weighting, while still being able to combine information at the feature
level across the streams, thanks to the representation fusion branch of our
bi-modal systems.

## Acknowledgements

The work was supported by the PhD School ”SecHuman - Security for Humans in
Cyberspace” by the federal state of NRW.

## References

  * Chetty and Alathur (2018) Naganna Chetty and Sreejith Alathur. 2018.  Hate speech review in the context of online social networks.  _Aggression and violent behavior_ , 40:108–118.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Fersini et al. (2022) Elisabetta Fersini, Francesca Gasparini, Giulia Rizzi, Aurora Saibene, Berta Chulvi, Paolo Rosso, Alyssa Lees, and Jeffrey Sorensen. 2022.  SemEval-2022 Task 5: Multimedia automatic misogyny identification.  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_. Association for Computational Linguistics.
  * Gomes et al. (2017) Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet. 2017.  A survey on ensemble learning for data stream classification.  _ACM Computing Surveys (CSUR)_ , 50(2):1–36.
  * Huang et al. (2017) Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. 2017.  Snapshot ensembles: Train 1, get M for free.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _Advances in Neural Information Processing Systems_ , 33:2611–2624.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Lu et al. (2020) Zhibin Lu, Pan Du, and Jian-Yun Nie. 2020.  VGCN-BERT: augmenting BERT with graph embedding for text classification.  _Advances in Information Retrieval_ , 12035:369.
  * Mann and Whitney (1947) H. Mann and D. Whitney. 1947.  On a test of whether one of two random variables is stochastically larger than the other.  _The annals of mathematical statistics_ , pages 50–60.
  * Onan et al. (2016) Aytuğ Onan, Serdar Korukoğlu, and Hasan Bulut. 2016.  Ensemble of keyword extraction methods and classifiers in text classification.  _Expert Systems with Applications_ , 57:232–247.
  * Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_.
  * Sabat et al. (2019) Benet Oriol Sabat, Cristian Canton Ferrer, and Xavier Giro-i Nieto. 2019.  Hate speech in pixels: Detection of offensive memes towards automatic moderation.  _arXiv preprint arXiv:1910.02334_.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Torrey and Shavlik (2010) Lisa Torrey and Jude Shavlik. 2010.  Transfer learning.  In _Handbook of research on machine learning applications and trends: algorithms, methods, and techniques_ , pages 242–264. IGI global.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_.
  * Wang et al. (2020) Ling Wang, Chengyun Zhang, Renren Bai, Jianjun Li, and Hongliang Duan. 2020.  Heck reaction prediction using a transformer model based on a transfer learning strategy.  _Chemical Communications_ , 56(65):9368–9371.
  * Xu et al. (2016) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2016.  Show, attend and tell: Neural image caption generation with visual attention.
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  Graph convolutional networks for text classification.  In _Proceedings of the AAAI conference on artificial intelligence_ , 01, pages 7370–7377.
  * Zhu (2020) Ron Zhu. 2020.  Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution.  _arXiv preprint arXiv:2012.08290_.