# Infrastructure Ombudsman: Mining Future Failure Concerns from Structural
Disaster Response

Md Towhidul Absar Chowdhury
Rochester Institute of Technology
<EMAIL_ADDRESS>
Soumyajit Datta∗
Maulana Abul Kalam Azad University of Technology
<EMAIL_ADDRESS>
Naveen Sharma
Rochester Institute of Technology
<EMAIL_ADDRESS>
Ashiqur R. KhudaBukhsh
Rochester Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

Current research concentrates on studying discussions on social media related
to structural failures to improve disaster response strategies. However,
detecting social web posts discussing concerns about anticipatory failures is
under-explored. If such concerns are channeled to the appropriate authorities,
it can aid in the prevention and mitigation of potential infrastructural
failures. In this paper, we develop an infrastructure ombudsman – that
automatically detects specific infrastructure concerns. Our work considers
several recent structural failures in the US. We present a first-of-its-kind
dataset of 2,662 social web instances for this novel task mined from Reddit
and YouTube.

##  1 Introduction

On January 28, 2022, at 6.39 a.m. EST, the Fern Hollow Bridge in Pittsburgh,
Pennsylvania collapsed. Due to the timing of the failure, thankfully, fewer
vehicles were on the bridge and only ten people were injured with no
fatalities. Pittsburgh, also known as the City of Bridges, was getting ready
for a visit from President Biden that day. Biden visited the collapse site and
assured federal assistance to rebuild the bridge on the spot. This
infrastructural failure, coinciding with a high-profile political visit and a
push towards passing the Build Back Better infrastructure bill, attracted
considerable media attention to the flailing infrastructural health in the US.

As we were sifting through the social web discussions surrounding this issue,
broad themes such as words of compassion for the victims and typical responses
in social web political discourse such as political name-calling, conspiracy
theories, and partisan mud-slinging emerged. However, apart from these
expected social web reactions, we noticed a small minority of interactions
that talked about anticipatory failures of other bridges in the US. Table 1
lists a few illustrative examples.

Social Media Post
---
There is a bridge in Lowell Massachusetts, it goes over the Merrimack river
and it is rusted strait through. It won’t be long before we suffer major
injuries because that bridge is always bumper to bumper traffic!
I’m surprised the New Kensington bridge wasn’t the first to go. Haha. Terrible
condition.
The bridge on 1-81 that spans the Potomac between WV and MD that could be
next. It has a lot of 18 wheelers beating it up. I kept a hammer in my car to
get out if it collapsed when I was on it.
Table 1: Illustrative examples indicating concerns over other vulnerable
bridges following the collapse of the Fern Hollow Bridge in Pittsburgh.

The comments in Table 1, if mined efficiently and surfaced to the appropriate
authorities, can present an effective path to intercept structural failure
concerns. Failures that are yet to happen - but a responsible citizen is
worried that they might. Vulnerabilities that they perhaps noticed before, but
the sudden, exogenous shock in the form of a structural failure gives them an
outlet to voice their concerns. Extant research on disaster response discourse
focuses on a diverse set of tasks that include: efficient distribution of
relief Varga et al. (2013), crisis management, handling emergencies etc Horita
et al. (2017). Much of this research has focused on natural disasters such as
typhoons Zou et al. (2023a), earthquakes Sakaki et al. (2012), floods Feng et
al. (2020), and accidents with severe fatalities Liu et al. (2020). Analyzing
social media responses to structural failures such as bridges or building
collapses is rather new and looking beyond the immediate, and focusing on
future potential crises has no prior literature to the best of our knowledge.

This paper presents infrastructure ombudsman, an automated social media
listener tool that surfaces infrastructure concerns. Via a curated corpus of
2,662 instances (271 positives and 2391
negatives)111https://github.com/towhidabsar/InfrastructureOmbudsman, we
demonstrate that state-of-the-art NLP and AI methods can be harnessed to build
such tools that can aid humans in identifying infrastructure concerns. While
the Pittsburgh Bridge collapse was an exogenous shock that got people
discussing other potential vulnerabilities, it is possible that our findings
point to a broader human pattern where structural collapses trigger similar
thoughts about anticipatory failures. Our dataset thus considers several
recent infrastructure failures in the US and combines effective rare-class
mining methods to present a more holistic resource to this novel task.

Our contributions are the following.

$\bullet$ Novel task: detecting infrastructure concerns: We define a new task
of detecting infrastructure concerns from social web posts. Our task presents
a marked departure from existing disaster response literature where our goal
is to identify citizen concerns about possible, future structural failures.

$\bullet$ New resource: We release a dataset of 2,662 instances (271 positives
and 2,391 negatives) of infrastructure concerns mined from millions of Reddit
posts and YouTube comments. We present a suite of strong baselines trained on
this dataset and demonstrate the feasibility of automatically detecting
infrastructure concerns with reasonably high precision and recall. On unseen
data, we demonstrate that our content classifier can effectively aid humans in
detecting potential candidates that express infrastructure concerns.

Social Media Post |  Label
---|---
There is a bridge in Lowell Massachusetts, it goes over the Merrimack river and it is rusted strait through. It won’t be long before we suffer major injuries because that bridge is always bumper to bumper traffic! |  Positive
Pennsylvania has some of the worst roads in the country. As soon as you cross the state line from New York state to PA state the roads immediately are filled with potholes and are built with shitty rugged concrete |  Positive
There’s a railroad overpass not far from where I live. About four years ago myself and my daughter were stopped waiting for a car to go through, it’s a one lane back road and only one vehicle at a time can drive through. We both noticed that there was a large crack, at least a foot long and maybe two or three inches wide? … |  Negative
Man, here in southeast ct theyre sandasting and repainting every bridge every couple years. Its crazy watching how fast they rust back up over the years. Ive seen them hit the same one twice enough years later. I would assume our bridges would collapse too if they didnt do this regularly. Some states are just stright up neglectful |  Negative
Table 2: Illustrative examples highlighting subtleties of our task.

##  2 The Needle and the Haystack

###  2.1 Needle

Intuitively, users expressing similar concerns following a structural failure
is a rare event. But what kind of concerns would be useful to surface to
appropriate authorities? Broad, generic concerns (e.g., Eventually, all US
bridges will go down given they are so old), while helpful to inform overall
government priorities, specific, actionable concerns are what we are looking
for in this task. Precise location information will be key for the concern to
be actionable.

Table 2 presents a few illustrative examples to elucidate this nuance. Unless
the content is geo-tagged, it is impossible to infer the location from There’s
a railroad overpass not far from where I live. In contrast, the post about the
bridge over the Merrimack River in Lowell Massachusetts uniquely identifies
the bridge 222As per bridgemeister.com, only four bridges go over the
Merrimack River of which only one is in Lowell.. Similarly, the complaint
about roads in Pennsylvania has a reasonably precise location close to the
state line coming from NY. Our final example in Table 2 has precise location
information and talks about infrastructural issues. However, it does not
qualify for what we are looking for as the post is praising the administration
for doing a responsible job rather than worrying about a possible failure.

###  2.2 Haystack

To provide a better sense of the haystack, we present a characterization of
the social web discourse following the Pittsburgh bridge collapse. We randomly
selected 100 YouTube comments with the keyword bridge on videos relevant to
the Pittsburgh bridge collapse. Filtering by keywords is an established high-
recall approach to obtain relevant discussion Halterman et al. (2021); Dutta
et al. (2022). We next characterize these comments through an open-coding
approach. We broadly observe the following categories: partisan blame game;
conspiracy theory; relief over no casualty; critical of US infrastructure;
critical of local administration; sarcasm; and infrastructure concern. Table 3
summarizes the distribution with illustrative examples. Note that, we do not
intend these categories to be formal or exhaustive, but rather to be
illustrative of the broad themes present in the discourse following this
incident and provide a rough estimate of the relative distribution of these
themes to underscore that infrastructure concerns are indeed a rare class.

Category |  Count |  Social Media Post
---|---|---
Partisan blame game |  29 |  This is why the dum republicans need to approve the Biden relief package and help build the bridges and airports. Republicans need to wake up and smell the coffee.
Conspiracy theory |  17 |  Someone really should check the area for explosives residue this does not seem like an accident it just so happened the president was coming to the exact town a big bridge collapsed in smells kinda fishy
Relief over no casualty |  12 |  Could u imagine driving on the bridge when it collapsed Thank goodness no one died.
Critical of US infrastructure |  15 |  The US is also literally falling apart. A couple of trillion won’t fix roads, bridges, airports, water infrastructure etc. and stop the decay
Critical of local administration |  8 |  Why has this bridge been left in poor condition for so long? Who in Pittsburgh and/or Pennsylvania is responsible for maintaining these structures? How much money have they spent, if any, to maintain and repair these bridges? Why are local and state elected officials and agencies not doing more? …
Sarcasm |  13 |  Don’t worry everyone, nobody will be fired over this. It’s perfectly normal for bridges to just fall the Fck down.
Infrastructure concern |  2 |  Bay bridge in Maryland is next. Christ, that thing is rickety
Others |  4 |
Table 3: Illustrative examples highlighting the diversity in discourse
following the Pittsburgh bridge collapse.

##  3 Dataset

Figure 1: Dataset Creation Pipeline

###  3.1 Platforms: Reddit and YouTube

To compile a diverse corpus with broad participation, we select a multi-
platform approach. We consider two well-known social web platforms: Reddit and
YouTube. Reddit provides a wide range of subreddit forums catering to
different political affiliations and demographics, allowing us to gather
infrastructure-related discussions from communities with varied ideological
leanings. For YouTube, we consider a longitudinal dataset of 110 million
comments on the official handles of three major US cable news networks: CNN,
Fox News, and MSNBC (denoted by $\mathcal{D}_{\textit{politics}}^{YT}$). This
dataset is considered a reliable and diverse snapshot of US political
discourse KhudaBukhsh et al. (2021); Yoo and KhudaBukhsh (2023); Weerasooriya
et al. (2023). In addition, we also search YouTube with a set of keywords
(described next) using publicly available API and collect comments on these
videos ($\mathcal{D}_{\textit{targeted}}^{YT}$). By consolidating data from
two different platforms, we seek to build a corpus covering perspectives
across the political spectrum, facilitating a more balanced analysis that is
less skewed by any individual platform’s biases.

Figure 1 presents a high-level, schematic diagram of our data curation
pipeline. In what follows, we present a brief description of each step.

###  3.2 Keyword Filtering

We curate the corpus to include discussions and media content related to
several specific search keywords:
{train derailment, infrastructure, infrastructure collapse, infrastructure
concern, Ohio train derailment, Missouri train derailment, Champlain Towers
South collapse, AdventHealth Orlando parking garage crane collapse, Charlotte
scaffolding collapse, Pittsburgh bridge collapse, Fern Hollow Bridge Collapse,
I-85 Overpass collapse}

Keyword-based filtering is a well-established, high-recall approach to pruning
datasets tailored for specific interests Demszky et al. (2019); Halterman et
al. (2021); Dutta et al. (2022). For $\mathcal{D}^{\textit{Reddit}}$ and
$\mathcal{D}_{targeted}^{\textit{YT}}$, the keyword set serves as search
keywords. For $\mathcal{D}_{politics}^{\textit{YT}}$, we consider comments on
videos where at least one of the keywords is present in the video title or
description.

Our keyword set captures a broad spectrum of incidents ranging from structural
failures such as building collapses to railroad failures such as train
derailments. By including specific incidents such as the Champlain Towers
South collapse and the AdventHealth Orlando parking garage crane collapse, the
corpus allows for in-depth case studies and comparative analyses, shedding
light on the unique challenges and responses associated with different types
of infrastructure failures.

To summarize, our initial corpus consists of the following sub-corpora:
$\mathcal{D}^{Reddit}$ (140,326 posts);
$\mathcal{D}_{\textit{politics}}^{YT}$(405,758 YouTube comments); and
$\mathcal{D}_{\textit{targeted}}^{YT}$(10,251 YouTube comments). We set aside
10,000 posts from $\mathcal{D}^{Reddit}$ and
$\mathcal{D}_{\textit{targeted}}^{YT}$ for in-the-wild evaluation (denoted by
$\mathcal{D}_{\textit{in-the-wild}}$).

###  3.3 Harnessing LLMs and Machine Annotation

####  3.3.1 Textual Entailment

Textual entailment, also known as natural language inference (NLI) MacCartney
and Manning (2008), has found recent applications in a broad range of social
inference tasks that include (1) estimating media stance on police portrayal
Halterman et al. (2021); Dutta et al. (2022); (2) detecting COVID-19
misinformation (Hossain et al., 2020); and (3) aggregating social media
opinions on election fairness (KhudaBukhsh et al., 2022). The NLI task takes a
premise $\mathcal{P}$ and a hypothesis $\mathcal{H}$ as input, and it outputs
entailment, contradiction, or semantic irrelevance. Textual entailment is much
more relaxed than pure logical entailment; it can be viewed as a human reading
$\mathcal{P}$ would infer most likely $\mathcal{H}$ is true. For instance, the
hypothesis _some men are playing a sport_ is entailed by the premise _a soccer
game with multiple males playing_ 333This example is taken from Bowman et al.
(2015)..

Following the formation of the initial corpus using keyword filtering, we
further prune our corpus with natural language inference (NLI). Our premise is
a Reddit post or a YouTube comment. Our hypothesis is “There is a growing
infrastructure concern somewhere.”. For each instance in our dataset, we run
an off-the-shelf NLI system Yin et al. (2019); Lewis et al. (2019). We retain
all instances where entailment is predicted with a probability greater than
0.5. This pruning step leaves us with 1,926 Reddit posts and 127,815 YouTube
comments.

####  3.3.2 LLM Annotation

Recent research has investigated Large language models (LLMs) for text
annotation tasks with considerable success Gilardi et al. (2023); Savelka and
Ashley (2023). In our next step, we use a well-known LLM (PaLM 2 Chowdhery et
al. (2022)) to further prune our data using a zero-shot setting. Grounded in
best practices prescribed in literature Ziems et al. (2023), we craft a prompt
to guide the annotation process ensuring that the model’s responses are
aligned with the desired annotations as shown in the first column of Table 4.
To enhance the LLM’s performance and streamline the classification task, we
supplied more representative examples. After this step, we obtained 243
examples from Reddit and 2,419 examples from YouTube.

###  3.4 Human Annotation Process

Following pruning steps using NLI and machine annotation, we start our human
annotation process leveraging Amazon Mechanical Turk (MTurk), a well-known
crowdsourcing marketplace. MTurk has documented liberal bias Sap et al.
(2022); Weerasooriya et al. (2023). While our task is politically neutral,
infrastructure as a priority is fiercely debated in US politics Crain and
Oakley (1995); Westphal (2008) (also, see Figure 2). To ensure balanced
political participation, we design our annotation tasks to involve three
annotators per instance: one identified as a Democrat, one as a Republican,
and one as an independent. Each annotator was tasked with classifying whether
each comment was talking about a future infrastructure concern at a specific
location. We compensate the annotator 0.1 USD for each instance. Each batch
with 30 instances would thus fetch 3 USD. Compensation is grounded in prior
literature Leonardelli et al. (2021); Bugert et al. (2020); Weerasooriya et
al. (2023).

Perhaps due to the subtleties associated with our text annotation task, we
observe relatively poor agreement between the MTurk annotators, with
Krippendorff’s alpha of 0.16 for classifying conversations regarding
anticipatory infrastructure concerns. This low Krippendorff’s alpha indicates
substantial disagreement between annotators and suggests the task itself was
challenging. Since raising concerns for potential future infrastructural
issues encompasses a broad range of topics such as transportation and
utilities. Additionally, anticipating and categorizing concerns about
hypothetical future infrastructure issues requires subjective judgment. The
cognitive load and semantic complexity of this annotation task likely
contributed to the low inter-annotator agreement.

We also observe annotation disagreement across partisan boundaries. Cohen’s
$\kappa$ between the Democrat and Republican annotators was just 0.054,
indicating slight agreement. Cohen’s $\kappa$ between the Democrat and
Independent was moderately higher at 0.331, while the $\kappa$ between the
Independent and Republican was 0.112, indicating minimal agreement. This
pattern suggests partisan identities and associated ideologies influenced how
annotators interpreted and categorized anticipatory infrastructure concerns
within the corpus. Conservatives and liberals appear to conceptualize
infrastructure issues through different frames, leading them to perceive
concerns differently even when analyzing the same textual content. Accounting
for partisan perspectives remains an ongoing challenge when attempting
balanced, unbiased annotation of politicized issues Sap et al. (2022);
Weerasooriya et al. (2023).

Figure 2: Distribution of crowdsourced workers’ answers to the question How
much priority should infrastructure get in the United States?. Y-axis
indicates percentage

Our analysis of the initial annotation data revealed segments with unanimous
agreement amongst all three partisan annotators had a high probability of
containing a valid, positive instance of an anticipatory infrastructure
concern. Leveraging this insight, we modified our annotation protocol to
utilize just two in-house annotators - an undergraduate and a graduate student
researcher. These researchers are familiar with the annotation guidelines, and
the infrastructure domain, and have research experience in urban data science.
Hence, these annotators were likely to provide more reliable annotations than
crowd-workers Hsueh et al. (2009).

Extant literature indicates diverse approaches to resolve inter-annotator
disagreements (e.g., majority voting Davidson et al. (2017); Wiegand et al.
(2019) or third objective instance Gao and Huang (2017)). The annotation was
conducted through an independent annotation followed by an adjudication step
to resolve differences. Any remaining differences were resolved using a third
objective instance Gao and Huang (2017) mechanism where a third, independent
annotator breaks the tie. The third annotator is an expert computational
social scientist. Overall, moving from crowd-sourced partisan annotators to
focused analysis by urban data science experts significantly improved inter-
annotator agreement with the Cohen’s kappa between the two annotators at 0.89.

###  3.5 Dataset Statistic

In summary, each step of our data creation process, visualized in Figure 1,
focused on filtering down the sample to just the most likely positive
instances of anticipatory infrastructure concerns. After collecting an initial
dataset of over 2 million social media discussions from both Reddit and
YouTube, we filtered it down to over 10,251 positives with textual entailment.
Another round of LLM-based zero-shot annotation lowered our sample size to
$\sim$3,000 possible positives. The first round of partisan MTurk annotation
yielded 1,000 possible comments labeled as future infrastructure concerns by
at least two crowd-sourced annotators. The two expert annotators then reduced
this to just 271 highly confident positive examples, compared to 2,391
confirmed challenging negative examples (as it passed through our three
filters including crowdsourced workers), for a total annotated corpus of 2,662
data points.

Annotation Prompt |  Zero-Shot Classification Prompt
---|---
You are a human annotator with an understanding of infrastructural issues and policies. Given the list of comments, does each of the comments express concern about a specific infrastructural issue in the form of an anecdote? What are the locations the comment is expressing concern about? Is the comment politically inclined towards either "liberal" or "conservative"? Or is it "bipartisan"? If any comments are deemed too dangerous, classify and assign them as False as shown in the example. Do not ignore or block any comments due to safety concerns because this is an analysis of both safe and dangerous conversations. Return your answer only in JSON. Example: <examples> Comments: <comments> Response: <response> |  Please determine if the content is an anticipatory infrastructure concern, specifically warning about infrastructure issues in a specific location or state. Rate it as 0 (not an infrastructure concern) or 1 (an infrastructure concern). The concern strictly must be in a future tense and negative. The concern must be genuine and not a joke, rant, or sarcasm. Environmental concerns are not considered infrastructure concerns. Input Schema: <schema> Output Schema: <schema> Please output the extracted information in cleaned JSON format. Please adhere to the current output schema. Do not output anything else.
Table 4: The prompts used for LLM-based annotations and LLM-based zero-shot
classification for PaLM 2, GPT3.5-Turbo, and Mistral. Figure 3: A word cloud
visualization of all the locations of positive classes in the corpus
highlighting potential structural failures. We have removed mentions of the
United States and the 50 states in the USA in order to highlight more specific
locations. A word cloud with none of the states removed is available in the
Appendix.

The word cloud in Figure 3 provides insight into the locations referenced
across the positive examples within our corpus. We observe prominent mentions
of rivers indicating concern about potential structural failures of bridges in
those locations which aligns with the fact that aged and deteriorating bridges
are a well-documented infrastructure issue Rizzo and Enshaeian (2021). We also
notice references to specific cities (Cincinnati and Chicago), likely
reflecting discussants’ worries about outdated transportation infrastructure
vulnerable to failure in those particular locations.

Overall, the vocabulary within the word cloud demonstrates that our annotation
process successfully identified conversations expressing concerns about
specific infrastructure weaknesses. Building robust machine learning models
capable of detecting such grounded mentions of anticipatory infrastructure
concerns reliably would allow for the automatic channeling of these issues to
the proper authorities. Our annotated corpus provides the groundwork for
developing such fine-grained rare-class NLP tasks.

##  4 Infrastructure Ombudsman

A key contribution of our work is developing natural language processing
methods to automatically identify discussions expressing anticipatory
infrastructure concerns. We frame this as a binary text classification
problem, where the goal is to determine whether a given text contains evidence
of concerns about potential infrastructure failures. In what follows, we
design and evaluate a suite of classifiers trained on this binary
classification task.

###  4.1 Zero-Shot Classification

We select three well-known LLMs (PaLM 2 Chowdhery et al. (2022), GPT-3.5-Turbo
noa ([n. d.]), and Mistral AI Jiang et al. (2023)) to perform zero-shot
classification based on the prompt shown in Table 4 as our baselines. As
already mentioned, this prompt has been designed following best practices
suggested by Ziems et al. (2023). We present the social web post following the
prompt.

In most cases, the models effectively delivered correct answers and
information. However, there were instances where they generated additional
content that extended beyond the initial request. To maintain the accuracy and
relevance of our findings, we implemented a filtering process to remove any
extraneous information.

###  4.2 Supervised Classification

We consider multiple classifier models including smaller language models with
around 700M parameters (BERT Devlin et al. (2018) and RoBERTa Liu et al.
(2019)) as well as large language models with 7B parameters (Mistral and Llama
2 Touvron et al. (2023)). We fine-tuned each model on our dataset using a
70/30 train/test split for 5 epochs using Adam optimizer Kingma and Ba (2017).
We consider standard machine learning performance metrics precision, recall,
F1, and accuracy on the held-out validation set.

Most of the instances present in our dataset mention physical locations. All
these locations can be clubbed into a placeholder <LOCATION> which might
benefit a text classifier not to attend to irrelevant information. To isolate
the impact of locale-specific references, we ran experiments under two
settings: (1) masked locations, and (2) no masked locations. Under the masked
location setting, we utilized named entity recognition (NER) to identify
locations and geopolitical entities like cities and states within each comment
using spaCy Montani et al. (2023), replacing them with the generic token
<LOCATION>. With no masked locations, we left the comments unmodified.

This enabled us to evaluate whether classifiers rely heavily on local
references to identify infrastructure concerns, or can infer these concerns
solely from high-level semantic content. Also, more location-dependent models
may overfit to particular places frequently mentioned as worrisome within the
training data, hindering generalization. Evaluating performance with masked
and unmasked locations can reveal opportunities to improve model robustness.
Our experiments aim to determine the degree to which anticipatory
infrastructure concerns can be detected from language patterns alone, without
relying on localization signals.

Model | Precision | Recall | F1 | Accuracy
---|---|---|---|---
$\texttt{BERT}_{\textit{nomask}}$ | 0.79$\pm$8e-5 | 0.82$\pm$1e-4 | 0.81$\pm$5e-4 | 0.92$\pm$1e-5
$\texttt{BERT}_{\textit{mask}}$ | 0.80$\pm$2e-4 | 0.83$\pm$2e-4 | 0.81$\pm$1e-4 | 0.93$\pm$2e-5
$\texttt{RoBERTa}_{\textit{nomask}}$ | 0.78$\pm$8e-5 | 0.83$\pm$1e-4 | 0.80$\pm$3e-5 | 0.92$\pm$2e-5
$\texttt{RoBERTa}_{\texttt{mask}}$ | 0.82$\pm$1e-5 | 0.83$\pm$3e-4 | 0.82$\pm$9e-5 | 0.93$\pm$3e-5
$\texttt{LLAMA2}_{\texttt{nomask}}$ | 0.83$\pm$2e-5 | 0.78$\pm$5e-5 | 0.80$\pm$2e-5 | 0.93$\pm$4e-5
$\texttt{LLAMA2}_{\texttt{mask}}$ | 0.81$\pm$1e-4 | 0.79$\pm$1e-4 | 0.80$\pm$1e-4 | 0.93$\pm$4e-5
$\texttt{Mistral}_{\texttt{nomask}}$ | 0.77$\pm$9e-6 | 0.75$\pm$5e-5 | 0.76$\pm$8e-5 | 0.91$\pm$6e-5
$\texttt{Mistral}_{\texttt{mask}}$ | 0.83$\pm$2e-5 | 0.79$\pm$1e-4 | 0.80$\pm$5e-5 | 0.93$\pm$3e-5
$\texttt{Mistral}_{\texttt{zero}}$ | 0.54 | 0.57 | 0.31 | 0.32
$\texttt{GPT3.5-Turbo}_{\texttt{zero}}$ | 0.53 | 0.57 | 0.35 | 0.38
$\texttt{PaLM2}_{\texttt{zero}}$ | 0.58 | 0.63 | 0.59 | 0.80
Table 5: The results of the classifier performances on our dataset. We split
our corpus into training and validation sets (70:30). All values are the macro
average results evaluated across a 5-fold run over the validation set except
for the zero-shot classification. We report the mean along with the variance.
Social Media Post
---
Right!!?! Those buildings that are built on supports right over the FDR also
freak me out. Those supports look old and crumbling and I doubt they get much
maintenance. There are 30 story buildings built on them. No doubt they will
collapse eventually as well.
Shortly after the I-35 bridge fell, I had to take a 50 wheel oversize load
from Chicago to Miami. Indiana was so nervous about their bridges they
required 6 escort cars and every time we crossed a bridge the escorts blocked
traffic and I had to cross alone at 5 MPH. They even shut down the Ohio River
bridge in Louisville. I was in Boston when the Mystic River bridge fell and in
Connecticut when the I-95 bridge fell. Now the I-40 bridge in Memphis is
showing a huge stress crack. Bridges do not seem to be something we are good
at.
Table 6: Illustrative examples of True Positive cases where our classifier
succeeds in identifying future infrastructure concerns Social Media Post
---
This is just a typical bridge in the northeast. This one isn’t even that high
and I’ve been on bridges that were literal one ways and the opposite direction
had to wait to go. Metal grating is quite common where I’m from. If you go
throughout anywhere in the northeast, especially the tristate area, many the
bridges make this sound and are very narrow. I’m just not getting why this is
scary. Are all your bridges concrete out there? You Midwestern and west
coasters haven’t caught up to our level
"The iconic PCH in California, is another scary drive. There are areas
peppered along the way to pull over because of stress and yes, the signs along
the highway actually say ""stress"" in the description. The road is narrow and
curvy with plenty of hairpin turns. One one side is the mountains (beware of
falling rocks) the other side is a direct drop straight down to the Pacific
Ocean with no guardrails. It’s really intense! You know it’s bad when the
state provides stress rest areas. That being said, the scenery is one of the
most beautiful in the world."
Table 7: Illustrative examples of False Positive cases where our classifier
identifies conversations as being specific anticipatory concerns but ground
truth is false.

##  5 Results and Analyses

Table 5 summarizes our supervised solutions’ performance. We observe all
supervised solutions attaining reasonable precision, recall, and F1 score. We
do not observe any across-the-board discernible benefit in masking or not
masking the location information indicating that our models are most likely
learning from the semantic content of the infrastructure concern rather than
being fixated on location mentions.

In contrast with the fine-tuned models, we observe that the zero-shot models
perform considerably poorly, with PaLM 2 emerging as the winner among the
zero-shot LLMs. This is possibly due to the nuanced nature of our task.

ID |  Social Media Post |  Classified
---|---|---
1 |  This was near where I recently lived, off I-35 in Fort Worth, I also lived in Austin off I-35, its a very long and dangerous interstate that I am thankful I don’t have to use anymore. Edit: people drive dangerously, not the roads fault necessarily. |  Positive
2 |  "You should see the bridges from Miami city. There is one that goes from Miami city to Miami Beach, and that thing is tall as hell! And its pretty long and scary, also it is very steep!" |  Positive
3 |  I always say I don’t like to be above the third floor that’s mostly because I figured that’s about as far up as I could go out the window and down a ladder I don’t care what the fire department might have that would be my absolute outside. This is just another reason to avoid high rises in general. I remember when this happened and it just seems so bizarre in Korea I always think of Korean culture is being very on the up and up. But I guess greed gets to too many people. Luckily I live in the Midwest in the US in a city that has maybe 10 buildings I won’t go in. |  Positive
4 |  They keep working on it, but the Second Street Bridge is getting old and scary! 1927 it was built! LOUISVILLE KY, Ohio River! |  Positive
5 |  I’m calling it right now. The CSX tracks that run over 25th st. in south philly are gonna collapse any day now. |  Positive
Table 8: A sample of illustrative examples from our in-the-wild performance
evaluation. Red posts are true negatives while blue posts are true positives,
as determined by annotators. The Classified column indicates what our best
classifier assigned to the post.

###  5.1 Error Analysis

Table 6 lists two instances of high-value conversations pointing out potential
infrastructure failures that may happen. These examples focus on crumbling
support pillars of buildings in the FDR Drive in New York City, and the I-40
bridge in Memphis, Tennessee. In both cases, there is a mention of a specific
location and a specific infrastructural concern - which our classifier
successfully captures.

Conversely, we highlight challenging negative cases in Table 7 where the
classifier incorrectly predicts a positive label. In the first example, while
expressing concerns about a bridge infrastructure and referencing a specific
location, the main topic is focused on comparing how the commenter’s location
has better bridges (making a sarcastic comment). The concern is not about an
actual anticipatory failure. The second example expresses concern about the
stress of driving on a scenic road but does not indicate any structural
issues.

In all these cases, our classifier fails to fully understand the nuanced
semantics and identifies that the concern is hypothetical, sarcastic, or
focused on a different topic altogether rather than actionable infrastructure
failures. Challenges in detecting negation, sarcasm, and subtle topic changes
contribute to these false positives. Further tuning to handle such linguistic
nuances could help address these errors. This analysis underscores the
difficulty of this rare class detection task, as many superficially related
discussions do not entail true infrastructure concerns on deeper semantic
inspection.

###  5.2 Performance In The Wild

To be practically useful, the infrastructure ombudsman, i.e., our content
classifier, will have to effectively mine infrastructure concerns in the wild.
To evaluate in-the-wild performance, we employ the top-performing classifier
from Table 5 ($\texttt{RoBERTa}_{\textit{mask}}$) to previously unseen
$\mathcal{D}_{\textit{in-the-wild}}$.

Running on this data, our classifier flagged 2,116 comments as positive
anticipatory infrastructure concerns, and 7,884 as negative. To estimate
precision and recall on this unlabeled set, we manually annotated a random
sample of 100 predicted positives and 100 predicted negatives.

On these manually annotated samples, our classifier achieved a macro precision
of 0.82, recall of 0.85, F1 score of 0.85, and an accuracy of 0.82, in
identifying true infrastructure concerns outside of the training data. This
demonstrates promising generalization to unseen data, with a high precision
indicating most flagged instances were true positives.

Table 8 presents a few illustrative examples. We first examine the false
positives. Post 1 highlights the challenges associated with this task as the
last sentence in the comment indicates it is not a concern. Posts such as post
2 in Table 8 were also classified as positive where it expressed concern but
not particularly of infrastructure but rather concern about driving on a tall
bridge. Detecting sarcasm is a well-documented NLP challenge Joshi et al.
(2017). Our classifier faces difficulties in recognizing hypothetical
statements and sarcasm. Comments like post 3 are classified as positive
despite the comment focusing on a hypothetical scenario. Moving on to true
positives, posts 4 and 5 highlight the Web-for-Good potential of our approach.
Posts like these can be channeled to the proper authorities in the local
community who can take remedial actions.

The capacity to automatically bring forward dialogues expressing apprehensions
regarding vulnerable infrastructure carries significant potential for
informing urban planners and transportation authorities. Our findings
underscore the power of natural language processing techniques, as pioneered
in this research, to unveil valuable insights within the intricate landscape
of textual data. This approach unveils latent challenges in infrastructure
oversight, typically concealed and arduous to decipher, providing a
transformative lens through which to tackle these critical concerns.

##  6 Related Work

Established literature has shown that microblogs are an important source of
real-time information to aid affected people and aid disaster relief
operations Gao et al. (2011); Vieweg et al. (2014). This research on disaster
response discourse [Neppalli et al. (2017) Zou et al. (2023b) Chowdhury et al.
(2020a) Desai et al. (2020) Sosea et al. (2021) Sirbu et al. (2022)] focuses
on a diverse set of tasks that include: efficient distribution of relief Varga
et al. (2013), crisis management, handling emergencies etc Horita et al.
(2017). Much of this research has focused on natural disasters such as
typhoons Zou et al. (2023a), earthquakes Sakaki et al. (2012), floods Feng et
al. (2020), and accidents with severe fatalities Liu et al. (2020).

Prior studies have delved into the practical implementation of domain
adaptation algorithms during the critical early stages of crises. For
instance, Li et al. (2017) explored the application of domain adaptation
algorithms to alleviate information overload on social media platforms during
emergency situations. Similarly, Ray Chowdhury et al. (2019) addressed the
paucity of research concerning the extraction of keyphrases from disaster-
related tweets. Their work proposed enhancements to a stacked Recurrent Neural
Network model by incorporating contextual word embeddings, POS tags,
phonetics, and phonological features. Chowdhury et al. (2020b) investigated
hashtag identification in disaster-related Twitter to identify actionable
hashtags in tweets that lacked user-provided hashtags. Furthermore, Rudra et
al. (2019) utilizes an extractive-abstractive approach to extract situational
summaries with a focus on using named entity recognition for missing person
updates in disaster events. Nguyen and Rudra (2022) also focuses on providing
an interpretable summarization framework for humanitarian microblogs during
disaster events.

In contrast to these prior works, our research differs in two key ways. First,
existing literature predominantly concentrates on natural disasters such as
typhoons, floods, earthquakes, and tsunamis, while our focus is on structural
failures. Second, and most saliently, our study breaks new ground by centering
on anticipatory failures. This novel focus on pre-emptive identification of
failures is, to the best of our knowledge, an unexplored dimension in the
current body of research.

##  7 Conclusion and Discussions

This paper presents a novel direction in mining anticipatory concerns from
social media following structural failures (e.g., bridge or building
collapses). To this end, we present the first dataset resource to perform this
novel task. We present a suite of strong baselines relying on the recent
advancements in large language models. Our automated anticipatory
infrastructure concern mining tool, dubbed infrastructure ombudsman, performs
effectively in the wild.

Our work can be extended along the following directions.

$\bullet$ Sampling methods for rare class-classification tasks: active
learning has a rich literature on effective sampling methods to address class
imbalance (e.g., minority class certainty sampling Sindhwani et al. (2009))
and improve in-the-wild robustness (e.g., anticontent sampling Yoo and
KhudaBukhsh (2023)). Incorporating active learning in our data curation
pipeline can be used to expand and enrich our dataset.

$\bullet$ Beyond infrastructure concerns: Our study reveals an interesting
pattern in human behavior where we observe that people often discuss potential
vulnerabilities following a structural failure. There could be broader
generalizability to this phenomenon. Following a brawl in a bar that results
in a shooting incident can trigger discussions on other bars where violence
might happen. Similar methods can be employed to intercept such concerns.

## Acknowledgement

This material is based upon work supported by the National Science Foundation
under Award No. 1943002.

## References

  * (1)
  * noa ([n. d.]) [n. d.].  GPT-3.5 Turbo fine-tuning and API updates.  https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Bugert et al. (2020) Michael Bugert, Nils Reimers, Shany Barhom, Ido Dagan, and Iryna Gurevych. 2020.  Breaking the Subtopic Barrier in Cross-Document Event Coreference Resolution.. In _Text2story@ ecir_. 23–29.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and others. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Chowdhury et al. (2020a) Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2020a.  Cross-lingual disaster-related multi-label tweet classification with manifold mixup. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_. 292–298.
  * Chowdhury et al. (2020b) Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2020b.  On identifying hashtags in disaster twitter data. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 498–506.
  * Crain and Oakley (1995) W Mark Crain and Lisa K Oakley. 1995.  The politics of infrastructure.  _The Journal of Law and Economics_ 38, 1 (1995), 1–17.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 11. 512–515.
  * Demszky et al. (2019) Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. 2019.  Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019)_. Association for Computational Linguistics, 2970–3005.
  * Desai et al. (2020) Shrey Desai, Cornelia Caragea, and Junyi Jessy Li. 2020\.  Detecting perceived emotions in hurricane disasters.  _arXiv preprint arXiv:2004.14299_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dutta et al. (2022) Sujan Dutta, Beibei Li, Daniel S. Nagin, and Ashiqur R. KhudaBukhsh. 2022.  A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Disparate News Media Stance. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022_. 5059–5065.
  * Feng et al. (2020) Yu Feng, Claus Brenner, and Monika Sester. 2020.  Flood severity mapping from Volunteered Geographic Information by interpreting water level from images containing people: A case study of Hurricane Harvey.  _ISPRS Journal of Photogrammetry and Remote Sensing_ 169 (2020), 301–319.
  * Gao et al. (2011) Huiji Gao, Geoffrey Barbier, and Rebecca Goolsby. 2011\.  Harnessing the Crowdsourcing Power of Social Media for Disaster Relief.  _IEEE Intelligent Systems_ 26, 3 (May 2011), 10–14.  https://doi.org/10.1109/MIS.2011.52 Conference Name: IEEE Intelligent Systems.
  * Gao and Huang (2017) Lei Gao and Ruihong Huang. 2017.  Detecting Online Hate Speech Using Context Aware Models. In _RANLP 2017_ , Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., 260–266.  https://doi.org/10.26615/978-954-452-049-6_036
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023\.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences of the United States of America_ 120 (2023).  https://api.semanticscholar.org/CorpusID:257766307
  * Halterman et al. (2021) Andrew Halterman, Katherine A. Keith, Sheikh Muhammad Sarwar, and Brendan O’Connor. 2021\.  Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ _(Findings of ACL, Vol. ACL/IJCNLP 2021)_. Association for Computational Linguistics, 4240–4253.  https://doi.org/10.18653/V1/2021.FINDINGS-ACL.371
  * Horita et al. (2017) Flávio EA Horita, João Porto de Albuquerque, Victor Marchezini, and Eduardo M Mendiondo. 2017\.  Bridging the gap between decision-making and emerging big data sources: An application of a model-based framework to disaster management in Brazil.  _Decision Support Systems_ 97 (2017), 12–22.
  * Hossain et al. (2020) Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020.  COVIDLies: Detecting COVID-19 Misinformation on Social Media. In _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_.
  * Hsueh et al. (2009) Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009\.  Data quality from crowdsourcing: a study of annotation selection criteria. In _Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing_. 27–35.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  https://doi.org/10.48550/arXiv.2310.06825 arXiv:2310.06825 [cs].
  * Joshi et al. (2017) Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017\.  Automatic sarcasm detection: A survey.  _ACM Computing Surveys (CSUR)_ 50, 5 (2017), 1–22.
  * KhudaBukhsh et al. (2021) Ashiqur R KhudaBukhsh, Rupak Sarkar, Mark S Kamlet, and Tom Mitchell. 2021.  We Don’t Speak the Same Language: Interpreting Polarization through Machine Translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14893–14901.
  * KhudaBukhsh et al. (2022) Ashiqur R. KhudaBukhsh, Rupak Sarkar, Mark S. Kamlet, and Tom M. Mitchell. 2022.  Fringe News Networks: Dynamics of US News Viewership following the 2020 Presidential Election. In _WebSci ’22: 14th ACM Web Science Conference 2022_. ACM, 269–278.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A Method for Stochastic Optimization.  https://doi.org/10.48550/arXiv.1412.6980 arXiv:1412.6980 [cs].
  * Leonardelli et al. (2021) Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021\.  Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 10528–10539.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  (2019).  https://doi.org/10.48550/ARXIV.1910.13461 Publisher: arXiv Version Number: 1.
  * Li et al. (2017) Hongmin Li, Doina Caragea, and Cornelia Caragea. 2017\.  Towards Practical Usage of a Domain Adaptation Algorithm in the Early Hours of a Disaster.. In _ISCRAM_.
  * Liu et al. (2020) Tiezhong Liu, Huyuan Zhang, and Hubo Zhang. 2020.  The impact of social media on risk communication of disasters—a comparative study based on sina weibo blogs related to tianjin explosion and typhoon pigeon.  _International journal of environmental research and public health_ 17, 3 (2020), 883.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  (2019).  https://doi.org/10.48550/ARXIV.1907.11692 Publisher: arXiv Version Number: 1.
  * MacCartney and Manning (2008) Bill MacCartney and Christopher D Manning. 2008.  Modeling semantic containment and exclusion in natural language inference. In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_. 521–528.
  * Montani et al. (2023) Ines Montani, Matthew Honnibal, Matthew Honnibal, Adriane Boyd, Sofie Van Landeghem, and Henning Peters. 2023.  explosion/spaCy: v3.7.2: Fixes for APIs and requirements.  https://doi.org/10.5281/ZENODO.1212303
  * Neppalli et al. (2017) Venkata K Neppalli, Cornelia Caragea, Anna Squicciarini, Andrea Tapia, and Sam Stehle. 2017\.  Sentiment analysis during Hurricane Sandy in emergency response.  _International journal of disaster risk reduction_ 21 (2017), 213–222.
  * Nguyen and Rudra (2022) Thi Huyen Nguyen and Koustav Rudra. 2022.  Towards an Interpretable Approach to Classify and Summarize Crisis Events from Microblogs. In _Proceedings of the ACM Web Conference 2022_ _(WWW ’22)_. Association for Computing Machinery, New York, NY, USA, 3641–3650.  https://doi.org/10.1145/3485447.3512259
  * Ray Chowdhury et al. (2019) Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2019\.  Keyphrase extraction from disaster-related tweets. In _The world wide web conference_. 1555–1566.
  * Rizzo and Enshaeian (2021) Piervincenzo Rizzo and Alireza Enshaeian. 2021.  Challenges in bridge health monitoring: A review.  _Sensors_ 21, 13 (2021), 4336.
  * Rudra et al. (2019) Koustav Rudra, Pawan Goyal, Niloy Ganguly, Muhammad Imran, and Prasenjit Mitra. 2019.  Summarizing Situational Tweets in Crisis Scenarios: An Extractive-Abstractive Approach.  _IEEE Transactions on Computational Social Systems_ 6, 5 (Oct. 2019), 981–993.  https://doi.org/10.1109/TCSS.2019.2937899 Conference Name: IEEE Transactions on Computational Social Systems.
  * Sakaki et al. (2012) Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2012\.  Tweet analysis for real-time event detection and earthquake reporting system development.  _IEEE transactions on knowledge and Data Engineering_ 25, 4 (2012), 919–931.
  * Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022\.  Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 5884–5906.
  * Savelka and Ashley (2023) Jaromir Savelka and Kevin Dean Ashley. 2023.  The Unreasonable Effectiveness of Large Language Models in Zero-shot Semantic Annotation of Legal Texts.  _Frontiers in Artificial Intelligence_ 6 (2023), 1279794.
  * Sindhwani et al. (2009) Vikas Sindhwani, Prem Melville, and Richard D. Lawrence. 2009\.  Uncertainty sampling and transductive experimental design for active dual supervision. In _ICML_. 953–960.
  * Sirbu et al. (2022) Iustin Sirbu, Tiberiu Sosea, Cornelia Caragea, Doina Caragea, and Traian Rebedea. 2022.  Multimodal Semi-supervised Learning for Disaster Tweet Classification. In _Proceedings of the 29th International Conference on Computational Linguistics_. 2711–2723.
  * Sosea et al. (2021) Tiberiu Sosea, Iustin Sirbu, Cornelia Caragea, Doina Caragea, and Traian Rebedea. 2021.  Using the image-text relationship to improve multimodal disaster tweet classification. In _The 18th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2021)_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  https://doi.org/10.48550/arXiv.2307.09288 arXiv:2307.09288 [cs].
  * Varga et al. (2013) István Varga, Motoki Sano, Kentaro Torisawa, Chikara Hashimoto, Kiyonori Ohtake, Takao Kawai, Jong-Hoon Oh, and Stijn De Saeger. 2013\.  Aid is Out There: Looking for Help from Tweets during a Large Scale Disaster. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1619–1629.
  * Vieweg et al. (2014) Sarah Vieweg, Carlos Castillo, and Muhammad Imran. 2014\.  Integrating Social Media Communications into the Rapid Assessment of Sudden Onset Disasters.  In _Social Informatics: 6th International Conference, SocInfo 2014, Barcelona, Spain, November 11-13, 2014. Proceedings_ , Luca Maria Aiello and Daniel McFarland (Eds.). Springer International Publishing, Cham, 444–461.  https://doi.org/10.1007/978-3-319-13734-6_32
  * Weerasooriya et al. (2023) Tharindu Cyril Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zamperi, Christopher M. Homan, and Ashiqur R. KhudaBukhsh. 2023\.  Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Westphal (2008) Joseph W Westphal. 2008\.  The politics of infrastructure.  _Social Research: An International Quarterly_ 75, 3 (2008), 793–804.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019\.  Detection of abusive language: the problem of biased datasets. In _Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers)_. 602–608.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  _arXiv preprint arXiv:1909.00161_ (2019).
  * Yoo and KhudaBukhsh (2023) Clay H Yoo and Ashiqur R KhudaBukhsh. 2023.  Auditing and robustifying COVID-19 misinformation datasets via anticontent sampling. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 37. 15260–15268.
  * Ziems et al. (2023) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023\.  Can Large Language Models Transform Computational Social Science?  https://doi.org/10.48550/arXiv.2305.03514 arXiv:2305.03514 [cs].
  * Zou et al. (2023b) Henry Peng Zou, Yue Zhou, Cornelia Caragea, and Doina Caragea. 2023b.  Crisismatch: Semi-supervised few-shot learning for fine-grained disaster tweet classification.  _arXiv preprint arXiv:2310.14627_ (2023).
  * Zou et al. (2023a) Lei Zou, Danqing Liao, Nina SN Lam, Michelle A Meyer, Nasir G Gharaibeh, Heng Cai, Bing Zhou, and Dongying Li. 2023a.  Social media for emergency rescue: An analysis of rescue requests on Twitter during Hurricane Harvey.  _International Journal of Disaster Risk Reduction_ 85 (2023), 103513\.

##  Appendix A Word Cloud with States

See Figure 4

Figure 4: A word cloud visualization of all the locations of positive classes
in the corpus highlighting potential structural failures.

##  Appendix B MTurk Annotation Questions

The list of questions we used for our crowdsourced annotation step can be
found here444https://osf.io/csqg5/?view_only=51488fbb433542dd872521e566e8aede

.