# Seeing the Intangible: Surveying Automatic High-Level Visual Understanding
from Still Images

Delfina Sol Martinez Pandiani<EMAIL_ADDRESS>0000-0003-2392-6300 University of BolognaDepartment of Computer Science and
Engineering (DISI)BolognaItaly and  Valentina Presutti
<EMAIL_ADDRESS>0000-0002-9380-5160 University of
BolognaDepartment of Modern Languages, Literatures and Cultures
(LILEC)BolognaItaly

###### Abstract.

The field of Computer Vision (CV) was born with the single grand goal of
complete image understanding: providing a complete semantic interpretation of
an input image. What exactly this goal entails is not immediately
straightforward, but theoretical hierarchies of visual understanding point
towards a top level of full semantics, within which sits the most complex and
subjective information humans can detect from visual data. In particular, non-
concrete concepts including emotions, social values and ideologies seem to be
protagonists of this “high-level” visual semantic understanding. While such
“abstract concepts” are critical tools for image management and retrieval,
their automatic recognition is still a challenge, exactly because they rest at
the top of the “semantic pyramid”: the well-known semantic gap problem is
worsened given their lack of unique perceptual referents, and their reliance
on more unspecific features than concrete concepts. Given that there seems to
be very scarce explicit work within CV on the task of abstract social concept
(ASC) detection, and that many recent works seem to discuss the similar non-
concrete entities by using different terminology, in this survey we provide a
systematic review of CV work that explicitly or implicitly approaches the
problem of abstract (specifically social) concept detection from still images.
Specifically, this survey performs and provides:

  1. (1)

A study and clustering of high level visual understanding semantic elements
from a multidisciplinary perspective (computer science, visual studies, and
cognitive perspectives);

  2. (2)

A study and clustering of high level visual understanding computer vision
tasks dealing with the identified semantic elements, so as to identify current
CV work that implicitly deals with AC detection

abstract concepts, high level semantics, scene understanding, image
processing, image classification, datasets, neural networks

††ccs: Computing methodologies Computer vision problems††ccs: Computing
methodologies Visual content-based indexing and retrieval

##  1\. Introduction

Visual imagery has been an efficient means to communicate both abstract and
concrete ideas since the birth of humanity. In modern life especially, images
have become central to communication, with modern media oversatured by
billions of pictures getting posted to social media every day (edwards2014).
Such rapidly expanding image collections, including ever-growing image
catalogues from the Cultural Heritage field, have attracted significant
research efforts for automatic indexing, retrieval and management of visual
data. Abstract concepts (ACs) are a powerful resource for these tasks, because
visual media are thought to illustrate concepts both by providing links to
concrete depicted objects through raw features–such as lines, color, shape,
and size–as well as through what Barthes called an image’s ‘connotation’: a
second layer of meaning made from culturally coded elements (Barthes, 1980).

Since the field of Computer Vision (CV) was born with the single grand goal of
complete image understanding, i.e. providing a complete semantic
interpretation of an input image, automatic abstract concept detection from
visual data seems like a natural task/subfield to exist. However, while much
progress has been made towards the general goal of complete image
understanding in computer vision, including outstanding performance for tasks
like image classification, object detection, and image generation (Bagi et
al., 2020), image understanding methods for explicitly detecting ACs seem to
be scarce. This may be partly because a main focus of computer vision has been
image segmentation for the detection of physical (concrete) objects based on
low-level features, while—by definition—ACs lack unique and distinctive
physical features, such as defined form, color and texture (Borghi and
Binkofski, 2014).

This survey follows the cognitive science terminology in its use of the term
abstract concepts (ACs) (Borghi et al., 2018; Harpaintner et al., 2018;
Villani et al., 2019; Yee, 2019), e.g. freedom, tradition or, consumerism,
which are used to refer to complex situations which do not possess a single
and perceptually bounded object as referent, and which have more variable
content both within and across individuals than concrete concepts (Barsalou,
2003; Borghi and Binkofski, 2014). The mechanisms that underlie the formation
and use of abstract concepts are the object of study of the ”Words As social
Tools” (WAT) cognitive theory (Borghi and Binkofski, 2014; Borghi et al.,
2019), which sees words as tools to perform actions modifying the state of our
social environment. In this sense, abstract concepts are seen as tools that
change the state of humans’ inner processes, helping us formulate predictions
and facilitating perception, categorization, and thought (Borghi and
Binkofski, 2014). In this sense, ACs underlie humans’ ability to convey
feelings, opinions and ideas (i.e. our abstract thoughts) through language
(Kousta et al., 2011). Ontologically, most of these concepts are social
objects (Gangemi and Mika, 2003), and include emotions (e.g., sadness), as
well as abstract social, cultural, economic, and political values (e.g.,
freedom, leadership) and ideologies (e.g., racism, consumerism, communism).

As such, ACs are a powerful resource for tasks such as automatic indexing,
retrieval and management of visual data. Importantly, the automatic
association of ACs to images could lead to breakthroughs in a wide range of
applications, including Web image search, online picture-sharing communities,
and scientific and academic endeavors. It would benefit multiple applications
by improving multimedia querying of digital libraries, as automatically
generated ACs could be used for Boolean search, as traditional metadata are.
Furthermore, enabling machines to recognize the potential of images to
communicate ACs could be useful to CH institutions, to enhance their visual
collections’ documentation, descriptions, and mediation–for example, using
detected ACs to inform the design of multimodal interactive environments.
Additionally, it could benefit public institutions building narratives about
their visual objects to engage people from different backgrounds or with
different abilities, companies in the creative sector exploiting their already
existing catalogues, and companies building products or services related to
specific abstract concepts.

Given that there seems to be very scarce work within computer vision that
explicitly tackles the task of abstract concept detection, and that many
recent works seem to discuss the similar non-concrete entities by using
different terminology, we find it is first necessary to characterize what
exactly such ”abstract”, or “high level”, semantics of an image entail, and
then what computer vision tasks and automatic detection approaches are related
to it. As such, in order to provide a state of the art regarding the automatic
detection of ACs from images, in this survey we investigate those questions
and provide a systematic review of CV work that explicitly or implicitly
approaches the problem of abstract concept detection (ACD) from still images.
Specifically, this survey performs and provides:

  1. (1)

An investigation and initial clustering of semantic entities that have been
recognized to be part of “high level semantics” of visual data, based on a
multidisciplinary investigation of high-level visual understanding (computer
science, visual studies, and cognitive perspectives);

  2. (2)

A clustering of Computer Vision “high-level semantic visual understanding”
tasks, and its direct mapping to the multidisciplinary understanding of “high
level semantics”

##  2\. Survey Methodology

The main interest behind this survey was the identification of Computer Vision
methodologies, tools, and architectures for abstract concept detection from
still images. This section will first explain our criteria for selecting
papers from the field of Computer Vision that deal with high level abstract
visual understanding tasks, including the criteria used to identify the
publication venues selected, as well as the keywords used to mine such
repositories and venues. It will also explain how we decided upon the sub-
selection of works that were considered for the detailed survey work on
abstract concept detection. Finally, this section explains the criteria that
the detailed survey uses to categorize the works implicitly or explicitly be
involved in the task of abstract concept detection task.

Keywords and Query Building. To the best of our knowledge, abstract concept
detection has not yet been standardized or even named as a self-contained CV
task. We intuited that this was not due to a complete lack of work on the
topics, but rather due to the use of a variety of terms to refer to the same
ontological class of abstract concepts. From an initial manual investigation
of high level visual semantics (see 3 units, we composed a list of keywords
based on a preliminary and interdisciplinary investigation of how such
abstract concepts and more generally high level abstract conceptual
understanding from visual media are referred to in multiple disciplines
(including visual studies, cognitive science, and CV). This investigation thus
included a top down search on major academic databases for publications
dealing with abstract concept detection from a CV perspective. The final list
of keywords has three kinds of terms in order to formulate queries: (1)
adjectives and (2) nouns that, when combined, referred to the same/similar
ontological objects as abstract concepts; and (3) nouns referring to tasks
like detection, identification, etc. The idea is that any combination of a
term from (1), one from (2), and one from (3) may be referring to the task of
abstract concept detection or a similar one (see Table 1).

| ”Abstract” - Adjectives
---
(synonyms, hyponyms of,
or similar to “abstract””)
| abstract, intangible, non-concrete, non-physical, symbolic,
---
latent, evoked, implied, subjective, social,
cultural, moral, political, economic, affective
| ”Concept” Nouns
---
(synonyms, hyponyms of,
or similar to “concept”)
| concept, object, idea, class, value,
---
ideology, emotion, affect, sentiment, signal, attribute
| CV Task Nouns
---
(synonyms, hyponyms of,
or similar to “detection”)
detection, classification, recognition, identification

Table 1. Three types of keywords used to build queries to search through the
selected venues to identify works dealing, implicitly or explicitly, with the
task of abstract concept detection. The keywords can be clustered in three
clusters: (1) adjectives and (2) nouns that, when combined, referred to the
same/similar ontological objects as abstract concepts; and (3) nouns referring
to tasks involving the identification of these ontological objects. The idea
is that any combination of a term from (1), one from (2), and one from (3) may
be referring to the task of abstract concept detection or a similar one.

Venue Selection Criteria. One of the most important factors to consider was
the relevance of the venues of publication. Since our central interest was on
Computer Vision (CV) approaches, so we decided to initially restrict our
search to venues that focus CV. A second factor considered to select venues
was the impact value of the venues, as measured by their impact factor. For
the selection of journals, we only considered journals with a Q1 rating for
the field of Computer Vision and Pattern Recognition according to
Scimago111https://www.scimagojr.com/. This led to the identification of 19
journals and one book series. Medical field-related journals (Journal of the
Optical Society of America A: Optics and Image Science, and Vision;
International Journal of Computer Assisted Radiology and Surgery; Medical
Image Analysis; and Computerized Medical Imaging and Graphics) were excluded,
leading to 15 journal venues. For the selection of conferences, we only
considered ones with a score of A* or A according to CORE222
http://portal.core.edu.au/conf-ranks/. This led to the identification of 4
conferences. The list of the 19 final venues selected can be found in Appendix
A.

Bottom-Up Search. The list of keywords was used to determine queries to search
the selected venues, with the addition of the word “images”. All of the search
through the selected venues was performed through the SCOPUS between February
and May of 2022), with the exception of the ECCV Conference and Workshop,
which are not accessible through it. Results were manually scanned in order to
only collect papers that actually deal with some kind of high-level visual
understanding of still images (i.e., dealt with some of the semantic elements
identified in section 3. Through this method, 52 papers were collected: 6
journal papers (2017-2021), and 44 conference papers (2008-2021). A subsequent
pruning was done so as to not consider articles that met any of the following
criteria:

  * •

Articles not in English. Only articles in English language were considered for
this survey.

  * •

Articles describing high-level visual semantics tasks but applied only to
moving images (i.e., videos, such as (Dibeklioğlu et al., 2012) and (da Silva
et al., 2020)). While articles that only focus on video were not considered,
articles that applied methods to both video and still images were considered.

  * •

Articles dealing with image captioning in general, not with high-level
reasoning task (e.g. (Li et al., 2019))

  * •

Articles describing methods focused on multimodal or cross-modal data (i.e.,
social media images plus their captions, news articles’ text and images
jointly, etc., for e.g., (Corchs et al., 2019; Thomas and Kovashka, 2020;
Vadicamo et al., 2017))

  * •

Articles dealing with image generation/iconic summaries of concepts (e.g.
(Raguram and Lazebnik, 2008)).

  * •

Articles dealing with high-level visual semantics (e.g., cultural event
recognition) but produced in the context of the Chalearn LAP challenge (e.g.,
(Liu et al., 2015; Rothe et al., 2015; Wang et al., 2015; Wei et al., 2015))
as these have been surveyed before (Escalera et al., 2015).

This led to a total of 38 publications: 4 journal papers, and 34 conference
papers, which are thoroughly surveyed in Table 4.2.

General Criteria. We survey all 38 of the selected publications in order to
identify some key criteria, including:

  * •

Image type (natural photographs, art images, etc.);

  * •

Dataset used and whether the authors created it;

  * •

The explicit Computer Vision-related task it deals with;

  * •

Related semantic elements previously identified to belong to ”high level”
visual semantics (see Section 3), with examples;

  * •

The cluster(s) of CV Tasks the work belongs to

Abstract Concept Detection Sub-selection. As will be explained in section 4, a
manual exploration of the selected 38 publications made clear that while all
of them dealt with high level semantic understanding of still images, the
tasks at hand were very varied (intent recognition, intimacy prediction,
aestheticness prediction, etc.) and dealt with a wide variety of semantic
units (e.g., affordances, atypicality, symmetry, friendship, freedom). Thus,
since a main interest of this work is to fully survey and identify methods for
abstract social concept detection as defined in this paper (see Section
LABEL:sec:acd), we needed to identify the subset of the publications which
actually dealt with these abstract notions. This process was done manually, by
focusing on the examples of semantic units each of the publications worked
with. We decided to only fully survey 19 publications. Given the small number
of publications, we expanded the search in a bottom-up way, by ?????????, and
added 7 other publications published in other venues .

Specific Criteria. For the selected ?? papers, we added other dimensions for
the full survey:

  * •

Method (supervised, unsupervised, heuristics, etc.)

  * •

Reproducibility (Does it make available software? data? FAIR principles?)

  * •

Hybridity of the system (data-driven, knowledge-driven, both, etc (van Bekkum
et al., 2021))

##  3\. Tip of the Iceberg: Characterizing High-Level Visual Semantics

###  3.1. Three-Tiered Semantics

Following the idea that understanding images completely would be providing a
“complete” semantic interpretation of an input image (Hoiem et al., 2008), in
the context of this work, visual understanding refers to the identification,
extraction and representation of semantic units from visual material. In the
context of computer vision (CV), these semantic units have mostly taken the
shape of what are generally called ‘objects’, like physical objects and agents
(”person”, “bottle”), attributes of those objects (“old”, “tall”, “rusty”),
actions (e.g., ”sitting”, “skiing”) and/or relationships between those objects
(e.g., ”person holding bottle”). Most of the systems and benchmark datasets
used in CV are tailored towards these types of “concrete” semantic entities.
However, a “complete” semantic interpretation of an image goes beyond such
concrete semantics. Instead, full visual semantics is usually attributed to a
“highest level of abstraction”, one in which “we humans are able to ‘see’
within the primitive attributes of two dimensional imagery the portrayal of
love, power, benevolence, hardship, discrimination, triumph, persecution and a
host of other aspects of the human condition (Enser and Enser, 1999).

The idea that the the perception and interpretation of visual meaning is a
multi-layered process is shared across domains and applications, including
cognitive science, computer vision, content-based image retrieval (CBIR), and
visual studies. This multi-layering was already noted as important to computer
vision in the seminal paper (Hare et al., 2006) which introduced the semantic
gap, which also referenced how a multi-layer view was common to refer to
different strata of meaning that humans identify, seek, and see in images, and
has been crucial for CBIR. We further delved into multiple of these multi-
layered approaches, including works by Panofsky (1955) (Panofsky and Drechsel,
1955), Shatford (1986) (Shatford, 1986), Greisdorf and O’Connor (2002)
(Greisdorf and O’Connor, 2002), Eakins (2000) (Eakins, 2000), Jorgensen (2003)
(Jörgensen, 2003), Hare et al (2006) (Hare et al., 2006), and Aditya et al
(2019) (Aditya et al., 2019). This led us to identified a general analogy of
three semantic tiers being used to separate the human visual understanding
process between a “low-level”, a “mid-level”, and an “upper-” or “high-level”
of semantics corresponding to increasing complexity and variability, and as
such subjectivity (see Figure 1). We noticed that many of the different
approaches identified similar semantic units as belonging to each of the
layers, and that a common visual analogy for the tiers was that of a pyramid
to convey a hierarchical connection, with the tip referring to an idyllic
“full semantics” (Hare et al., 2006). Perhaps most importantly, we noticed
that while semantic units of both the low- and mid- layers were not only
identified but also generally agreed on, the same was not true of the third
layer of “high level semantics”.

Figure 1. Visual understanding has been previously conceptualized as a
multilayered process, in which three main levels of semantics can be
identified. The low-level is generally connected to raw or primitive features;
the mid-level is generally connected with individual objects, persons, and
regions; the high-level is not very deeply characterized.

Specifically, the “low-level” layer of at the bottom of the hierarchy of
visual semantics (in light blue in Figure 1) has been characterized as being
the stage of raw or primitive features such as regions, edges, textures,
colors, shapes and textures. The “mid-level” layer of semantics (in light
purple in Figure 1) has instead been characterized as the tier in which
semantic units like objects, persons, regions and places belong. Much of
Computer Vision work has focused on this layer, with the centrality of object
recognition and image segmentation.

Compared to these two levels, the layer of “high-level” semantics (in dark
purple in Figure 1) seems both much less characterized in detail, and much
less agreed on. This highest level of “full semantics” does not seem to an
explicit definition, especially not one used consistently in CV-related work.
It is Instead, a shared understanding of what types of content may actually
reside/be conceived to be there seems to be generally assumed implicitly. As
such, we took on the task of further describing this highest level of
semantics, which seems to be the layer where abstract concepts belong in, and
which—throughout our investigation—emerged as a “tip of the iceberg” of visual
semantics.

###  3.2. The Upper Layer of Visual Semantics

Images may be sought “on the basis of their holistic content or message, as
opposed to the information embedded within them by dint of their depiction of
certain features” (Enser and Enser, 1999; its39). Most work that attempts to
name and characterize where and how such holistic content arises thus moves in
a layered way further away from raw or primitive features, to arrive to the
“highest” level of meaning, referred to with different names: iconological
layer (Panofsky and Drechsel, 1955), higher level of understanding (Jörgensen,
2003), abstract content (Shatford, 1986), abstract attributes (Eakins, 2000),
subjective beliefs (Greisdorf and O’Connor, 2002), higher level semantics
(Aditya et al., 2019), full semantics (Hare et al., 2006).

Part of the difficulty of having a shared understanding of cross-disciplinary
high level semantics is that, in comparison to the other levels, high-level
understanding by humans is increasingly cognitively complex. Complex cognitive
processes, including abstraction, metonymic conveyance, adumbration,
impression, prototypical displacement (Greisdorf and O’Connor, 2002),
connotation (Hare et al., 2006; Shatford, 1986), evocation, and synthetic
intuition (Panofsky and Drechsel, 1955) are considered crucial tools for
understanding visual semantics at this “high level” of abstraction. However,
it is generally thought that it is practically hard to grasp them using
typical automatic image understanding and indexing methods. As such, this
highest level of abstraction in the interpretation of image meaning or content
is seen as a “seemingly insurmountable obstacle” to the application of content
based image retrieval techniques (Enser and Enser, 1999).

Figure 2. Top of the semantic pyramid. The dark blue refers to ”high level
semantics” as a whole. Based on a multidisciplinary investigation of the kinds
of semantic entities that have been placed within this upper layer of
semantics, four clusters of knowledge have been identified.

Apart from cognitive complexity, subjectivity is another of the features that
make this level difficult to characterize and automatically recognize.
Shatford’s (Shatford, 1986, p. 42) oft-quoted observation put it succinctly: “
…the delight and frustration of pictorial resources is that a picture can mean
different things to different people” puts it succintly. As such, a picture
can mean different things to the same person at different times or under
different circumstances of need. In this vein, Greisdorf (Greisdorf and
O’Connor, 2002) identifies the need for interdisciplinary perspectives to
serve as a foundation for modeling the attributes of the human image
cataloguing process because

> Those attributes tend to elude the indexing/cataloging process by exceeding
> the image indexing threshold due to individual viewer cognitive displacement
> of objects and object features that give rise to disjunctive prototypes that
> viewers may associate with the objects included as part of the image
> composition. These adumbrative, impressionistic and abstractionist concepts
> that relate viewer to image need to be captured with some type of retrieval
> mechanism in order to enhance retrieval effectiveness for system users.
> (Greisdorf and O’Connor, 2002, p. 11)

Part of the work needed to better understand and communicate about these
abstract semantics is to more specifically identify semantic units that may
belong to this layer, and potentially characterize how the semantic units may
relate to one another. As such, we went though the cited literature and
attempted to more detailedly characterize this tip of visual semantics (see
Figure 2). We collected the types of elements have been mentioned as belonging
to high-level visual semantics, and clustered them into four general groups:

  * •

Commonsense semantics. This cluster is the closest to mid-level semantics in
the sense that ti is the one that is perhaps the least subjective of the
clusters. It includes semantic elements such as actions (“running”),
activities (“dance”), events (“parade”), relationships between objects or
object-object interactions (”man holding cup”), and object purposes. These
were grouped under the term “commonsense” knowledge because they are the types
of semantic elements that may have a high level of agreement between viewers
and can be described under some logical framework.

  * •

Emotional semantics. This cluster holds semantic information that relates to
feelings, emotions, affects, moods, etc.

  * •

Aesthetic semantics. This small cluster refers to global attributes relating
to a judgement of the image as a global entity.

  * •

Inductive interpretative semantics. This last cluster contains perhaps the
most complex, subjective and culturally-coded of the semantic elements
identified within “high-level semantics”, including the meaning, significance,
aboutness and purpose of the image, as well as symbols, values and generally
abstract concepts.

These clusters are only an initial attempt at better characterizing this layer
of visual semantics. While the clusters are depicted in a mutually exclusive
way, this is not necessarily true – there are probably semantic elements that
belong to two or more clusters (e.g. a mood and an aesthetic; an atmosphere
and an abstract concept; an emotion and an abstract concept). Further
revisions to the diagram are possible and underway.

##  4\. Automatically Seeing the Intangible: CV Tasks Dealing with High-Level
Conceptual Semantics

###  4.1. Minding the Gap(s)

According to (Liu et al., 2007), the last few decades of image understanding
have been shaped by two main directions. The first is low-level processing,
where basic information is extracted from pixel values in the form of color
histograms, frequency responses, and used to create a representation in a
vectorial space, where tasks such as clustering or classification can be
performed (e.g., (Carson et al., 1999)). In the second path, segmentation,
classification, and detection techniques are used to extract the image’s
semantic information, which is then employed for tasks like content-based
indexing and retrieval (Li et al., 2010; Smeulders et al., 2000).

This second path focused on image semantics is plagued by the major challenge
described by Smeulders as the semantic gap (Smeulders et al., 2000, p. 1352):

> The semantic gap is the lack of coincidence between the information one can
> extract from the visual data and the interpretations that the same data has
> for a user in a given situation.

Studying computational iconic image analysis, van Noord (van Noord, 2022)
recognizes that the width of the semantic gap is not consistent across images
and extends this concept into social and temporal aspects, as the meaning of
an image shifts over time and is different across social and cultural groups.
In this sense, Van Noord is inspired by theories and challenges related to the
study of visual culture, and introduces the idea of the cultural gap in the
following manner (van Noord, 2022, p. 3):

> The cultural gap is the lack of coincidence between the information one can
> extract from the visual data and the interpretations that the same data have
> for cultural groups across time.

In this sense, the width of the gap between a user’s semantics and the visual
data can be widest for images which are polysemic, or for which the semantics
are only partially described. The author explains the idea by presenting the
example of alternative readings of the iconic ’Tank Man’ image: the common
reading as an evocation of the AC of protest, and the Chinese officials’
reading as an evocation of a different AC, military restraint.

###  4.2. Past Work on High Level Semantic Understanding in CV

After characterizing more in detail the kinds of semantic elements that
inhabit the high level of abstraction in visual understanding where abstract
concepts are generally placed, we wanted to survey the state of the art within
the field of computer vision to automatically detect them from still images.
By following the methodology explained in Section 2, we searched through
publications venues with the highest ratings and impact value in the field of
Computer Vision and Pattern Recognition and ended up collecting 38
publications of interest: 5 journal papers (2017-2021), and 34 conference
articles (2008-2021).

As reported in Table 4.2, we first surveyed the published works to understand
a series of trends in the use of CV technologies for high level visual
semantic understand. We took note of the image domains of focus (e.g., natural
photographs, art images, etc.), the datasets they used and whether they
created the dataset themselves, as well as the explicit CV tasks the articles
dealt with. Perhaps most importantly, we wanted to identify, for each of the
papers what, if any, high level semantic units the articles deal with in the
context of still images, along with examples. This allowed us to understand
what types of abstract notions have been grappled with already in the
literature.

Citation |  Image Type |  Dataset |  Own Data |  Task(s) |  Related Semantic Element |  Semantic Element Examples |  CV Cluster(s)
---|---|---|---|---|---|---|---
(Achlioptas et al., 2021) |  artwork images |  ArtEmis |  Y |  affective image captioning; dominant emotion prediction |  feelings; abstract concepts |  awe, freedom, love |  visual sentiment analysis
(Bossard et al., 2013) |  natural photograph |  PEC Data Set |  Y |  event recognition |  social events |  roadtrip, halloween, christmas |  situational analysis
(Chandrasekaran et al., 2016) |  clipart images |  AVH (Abstract Visual Humor) Dataset |  Y |  visual humor prediction |  atmosphere; meaning |  humor/funninness |  visual rhetorical analysis
(Chuang et al., 2018) |  natural photographs of scenes |  ADE-Affordance |  Y |  affordance reasoning in scenes |  object purposes |  socially awkward affordance, socially forbidden affordance |  situation recognition
(Datta et al., 2006) |  natural photograph |  Untitled |  Y |  aesthetic quality inferrence |  aesthetics |  aesthetic value |  aesthetic analysis
(Gallagher and Chen, 2009) |  natural photographs of groups of people |  The Images of Groups Dataset |  Y |  demographic information detection; event recognition |  events |  dining, age, gender |  social signal processing; situational analysis
(Ghosh and Dhall, 2019) |  natural photographs of groups of people |  GAF-personage database |  Y |  most influential person identification |  affect; object-object interactions |  leader, influential person |  social signal processing
(Goel et al., 2019) |  natural photograph |  PIPA-relation; PISC |  N |  social relationship recognition |  object-object interaction; atmosphere |  friends, couple, professional relationship |  social signal processing
(Gray et al., 2010) |  natural photographs of frontal female faces |  Untitled |  Y |  female facial beauty prediction |  aesthetics; abstract concept |  female beauty |  aesthetic analysis; social signal processing
(Guo et al., 2021) |  image advertisements |  Ads Dataset |  N |  persuasive atypicality detection |  object purposes; object-object interactions |  atypicality |  visual rhetorical analysis; situational analysis
(Hsiao and Grauman, 2017) |  natural photographs (fashion images) |  Style Embedding Dataset |  Y |  style embedding learning |  aesthetics |  hipster, bohemian, preppy |  social signal processing; aesthetic analysis
(Huang and Kovashka, 2016) |  natural photographs of politicians |  Visual Persuasion Dataset |  N |  communicative intent recognition; visual persuasion understanding |  object purposes; purpose/meaning |  trustworthy, powerful, energetic |  visual rhetorical analysis
(Hussain et al., 2017) |  image advertisements |  Ads Dataset |  Y |  automatic advertisement understanding; symbolism predicttion |  symbols; symbolical values |  danger, beauty, death |  visual rhetorical analysis
(Jia et al., 2021) |  natural photographs of everyday scenes |  Intentonomy |  Y |  intent recognition |  object purposes; purpose/meaning |  harmony, mastery, perseverance |  visual rhetorical analysis
(joo2014visualpersuasiona) |  natural photographs of politicians |  Visual Persuasion Dataset |  Y |  communicative intent recognition; visual persuasion understanding |  object purposes; purpose/meaning |  trustworthy, powerful, energetic |  visual rhetorical analysis
(Joo et al., 2015) |  natural photographs of politicians |  Visual Persuasion Dataset |  Y |  political affiliation detection; personality trait detection |  individualized affect; object purposes |  intelligence, competence, democrat affiliation |  social signal processing; visual rhetorical analysis
(Khosla et al., 2014) |  natural street view images |  Untitled |  Y |  environment navigation |  situational analysis; atmosphere |  proximity to a mcdonalds, crime rate |  situational analysis
(Kiapour et al., 2014) |  natural photographs of full body outfits |  Untitled |  Y |  fashion style classification |  aesthetics |  hipster, bohemian, preppy |  social signal processing
(Li et al., 2017b) |  natural photograph |  PISC |  Y |  social relationship recognition |  object-object interaction; atmosphere |  friends, couple, professional relationship |  social signal processing
(Li et al., 2017a) |  natural photograph |  imSitu |  N |  situation recognition |  actions; implied actions; object purposes; object relationships |  obstacle, victim, destination |  situation recognition
(Li et al., 2020a) |  natural photograph |  PIPA-relation; PISC |  N |  social relationship recognition |  object-object interaction |  colleagues, lovers, audience |  social signal processing
(Masip Rodo et al., 2012) |  natural photographs of faces |  Untitled |  Y |  social dimension trait detection |  individualized affect; object purposes |  dominance, aggressiveness, threatening |  social signal processing
(Pratt et al., 2020) |  natural photograph |  SWiG |  Y |  grounded situation recognition |  actions; implied actions; object purposes; object relationships |  obstacle, victim, destination |  situation recognition
(Segalin et al., 2017) |  natural photograph |  PsychoFlickr corpus |  N |  personality trait-based image classification |  individualized affect; significance |  neuroticism, extraversion, openness |  social signal processing
(Shao et al., 2013) |  natural photograph |  Occupation Database |  Y |  occupation prediction |  object purposes |  waiter, customer, clergy |  social signal processing
(Stabinger and Rodriguez-Sanchez, 2017) |  artificial chessboard images |  Untitled |  Y |  symmetry task, identity task |  object-object interaction |  identity, symmetry |  situational analysis
(Suhail and Sigal, 2019) |  natural photograph |  imSitu |  N |  situation recognition, role assignment |  actions; implied actions; object purposes; object relationships |  audience, obstacle, source |  situation recognition
(sun2017domainbaseda) |  natural photograph |  PIPA-relation |  Y |  social relationship recognition |  object-object interaction; atmosphere |  friends, colleagues, lovers |  social signal processing
(Thomas and Kovashka, 2021) |  images from news sources |  Politics |  Y |  political bias prediction |  symbolical values; intrinsic meaning |  diversity, tradition, homelessness |  visual rhetorical analysis
(Toisoul et al., 2021) |  natural photographs of faces |  AffectNet, SEWA, AFEW-VA |  N |  categorical emotion recognition; valence estimation; arousal estimation |  emotion |  anger, calming, deppressed |  visual sentiment analysis; social signal processing
(Wang et al., 2010) |  natural photographs of groups of people |  The Images of Groups Dataset |  N |  social relationship recognition |  object-object interaction |  siblings, mother-child, husband-wife |  social signal processing; situational analysis
(Workman et al., 2017) |  ground-level and overhead natural photographs |  ScenicOrNot (SoN) dataset |  Y |  scenicness prediction |  atmosphere; aesthetics |  scenicness of natural landscapes |  aesthetic analysis
(Yuanjun Xiong et al., 2015) |  natural photographs of events |  WIDER |  Y |  event recognition |  event |  parade, press conference, meeting |  event recognition
(Yao et al., 2019) |  natural photographs, art images and art paintings |  Flickr and Instagram (FI); IAPSa; ArtPhoto; Abstract |  N |  affective image retrieval |  emotion |  contentment, awe, excitement, disgust |  visual sentiment analysis
(Yatskar et al., 2016) |  natural photograph |  Situnet |  Y |  situation recognition |  actions; implied actions; object purposes; object relationships |  obstacle, audience, source |  situation recognition
(Ye and Kovashka, 2018) |  image advertisements |  Ads Dataset |  Y |  automatic advertisement understanding |  symbols; symbolical values |  freedom, happiness, adventure |  visual rhetorical analysis
(Ye et al., 2019) |  image advertisements |  Ads Dataset |  Y |  automatic advertisement understanding |  symbols; emotions; abstract concepts |  danger, adventure, death |  visual rhetorical analysis
(Zhang et al., 2015) |  natural photographs with visible faces |  Social Relation Dataset |  Y |  social relationship recognition |  object-object interaction |  friendliness, warmth, attachment |  social signal processing; situational analysis

One of the most immediately clear results is that a majority (around 75
percent) of works grappling with these questions have acted on a need to
create their own datasets. This could be pointing towards the difficulty of
generalizing over these more cognitively complex tasks. Our results also show
that there is a plethora of work taking place and being published at top CV
venues that deals with high-level visual semantics, and that specifically
attempts to relate cognitively complex and subjective elements (concepts) to
still images. Most of the related work is centered upon a few CV tasks:
situation recognition (Yatskar et al., 2016; Suhail and Sigal, 2019; Pratt et
al., 2020; Li et al., 2017a), social relation recognition (Zhang et al., 2012;
Wang and He, 2008; Sun et al., 2017; Li et al., 2020a, 2017b; Goel et al.,
2019), event recognition (Yao et al., 2019; Yuanjun Xiong et al., 2015;
Bossard et al., 2013), visual persuasion and intent analysis (Joo et al.,
2014; Jia et al., 2021; Huang and Kovashka, 2016; Guo et al., 2021), automatic
advertisement understanding (Ye et al., 2019; Ye and Kovashka, 2018; Hussain
et al., 2017), visual sentiment analysis (Yao et al., 2019; Toisoul et al.,
2021; Achlioptas et al., 2021), aesthetic evaluations (Workman et al., 2017;
Gray et al., 2010; Datta et al., 2006), social and personality traits (Segalin
et al., 2017; Masip Rodo et al., 2012; Joo et al., 2015), including occupation
(Shao et al., 2013), fashion (Kiapour et al., 2014; Hsiao and Grauman, 2017),
and other group-level analyses (Ghosh and Dhall, 2019; Gallagher and Chen,
2009). Other CV tasks that seem relevant include abstract reasoning (Stabinger
and Rodriguez-Sanchez, 2017), affordance reasoning (Chuang et al., 2018),
political bias detection (Thomas and Kovashka, 2021), visual humor detection
(Chandrasekaran et al., 2016), and environmental variables prediction (Khosla
et al., 2014). These clusters are identifiable also by looking at the examples
of the related semantic units (sixth column in Table 4.2, it seems that most
of the work is centered around some of the semantic units previously
identified in section 3: emotions, events, aesthetics, atmosphere, object
purposes, object-object interactions, symbols, etc.

Based on these results, we came up with a series of clusters of CV tasks
dealing with high level semantic units, and assigned each work to one or
multiple of them (last column of Table 4.2). We used the results of this last
part to create a cluster-based diagram of CV tasks dealing with high level
visual reasoning Fig. LABEL:fig:experiments) that is analog to the one of
high-level visual semantic units (Fig. LABEL:fig:tip). In this diagram, we
attempt to map the plethora of work within Computer Vision dealing with high-
level visual reasoning and visual semantics into the same conceptual and
terminological space we have referenced in section 3.

Figure 3. Computer Vision tasks that deal with ”high level semantics” or ”high
level visual understanding”, which have been mapped also to the previous
multidisciplinary characterization of high level semantics. Circled in red are
the tasks which were found to implicitly or explicitly deal with abstract
concept detection.

  * •

Situational analysis. This cluster corresponds to the commonsense semantics
cluster, involved with actions, activities, roles and purposes of objects, and
the (mostly logical or common) interactions between these objects. Three main
sub-clusters of CV tasks can be further identified within situational
analysis. The first is abstract reasoning, dealing with global semantic tasks
based on logic, such as the symmetry of a chessboard or its identity in
different images (e.g., (Stabinger and Rodriguez-Sanchez, 2017)). The second
is situation recognition, a task first defined by (Yatskar et al., 2016) and
focused on the problem of producing a concise summary of the situation an
image depicts including: the main activity, the participating actors, objects,
substances, and locations and most importantly the roles these participants
play in the activity. It is based on the idea that understanding an image
involves more than just predicting the most salient action, and is a
structured prediction task that aims to predict the verb and its frame that
consists of multiple role-noun pairs. Finally there is a large body of work
centered on event recognition, including specifically social and cultural
events (e.g., “Rio Carnaval” or “Chinese New Year”).

  * •

Visual sentiment analysis. This cluster corresponds to the previously
identified cluster of emotional semantics. Also known as Image Emotion
Analysis (IEA), this subdomain aims to understand how images affect people, in
terms of evoked emotions. Although this field is rather new, over the last
years, a broad range of techniques have been developed for various data
sources and problems, resulting in a large body of research and multiple
surveys (Ortis et al., 2020). Most studies have focused on the detection of
emotions (e.g., fear, sadness, excitement, contentment) in natural images
(survey by (Zhao et al., 2021)), with some works focusing instead on group
emotion in images (overview by (Veltmeijer et al., 2021). Apart from natural
images, other types of visual media have been investigated. So far, automatic
detection of emotion has been attempted both in art images (Cetinic et al.,
2019; Achlioptas et al., 2021) and in memes (Sharma et al., 2020).

  * •

Aesthetic analysis. This cluster corresponds to the previously characterized
cluster of aesthetic semantics. Work related to it is sparse, but overall
focused on detecting or predicting aesthetic value from images, including from
natural scenes (Workman et al., 2017), human faces (Gray et al., 2010), or
from the image as a whole (Datta et al., 2006).

  * •

Social signal processing. We identified this cluster that does not have an
analog in the previous high-level semantics pyramid. Social Signal Processing
is a broad term to describe research conducted in constructing computational
models to sense and understand human social signals including emotion,
attitude, personality, skill, role, and other forms of communication between
humans. Some efforts within the field have focused on the use of computer
vision tools to detect various non-concrete social aspects of images. In
natural images, some efforts have focused on the detection of the persuasive
intent in photographs of politicians (Joo et al., 2014), deception (Bruer et
al., 2020; Zanette et al., 2016), social relationship type (e.g., kinship,
friendship, romantic, and professional relationships) (Li et al., 2020b); and
intimacy (Chu et al., 2015). An especially relevant image type for analysis
has been group or crowd images, with various methods developed to
automatically identify non-concrete aspects of groups such as: social cohesion
(Hung and Gatica-Perez, 2010; B Kantharaju et al., 2020), leadership (Solera
et al., 2017), warmth and dominance (Zhang et al., 2018), excitement (Varghese
and Thampi, 2018), engagement (in groups of students) (Vanneste et al., 2021).
Social signal processing tasks have been much lower for art images. Some work
has focused on the automatic detection of trustworthiness of a depicted person
in art images (Safra et al., 2020).

  * •

Visual rhetorical analysis. This cluster is perhaps the most correlated to
inductive interpretative semantics. It includes work that relates to visual
persuasion and rhetorical techniques to understand what has been previously
defined as the ”intrinsic meaning” or the purpose of the image as a whole.
Much of the work within this cluster has taken place within the sub-domain of
automatic advertisement understanding. While the visual rhetoric of images has
been studied in Media Studies [from Hussain: 79, 71, 51, 55, 54, 5, 44, 12],
it is only in the last 5 years that it is being studied computationally, with
much of the work by the authors of (Hussain et al., 2017). One of the sub-
clusters is that of visual persuasion (Joo et al., 2014), which analyzes
whether images of politicians portray them in a positive or negative light by
using features that capture facial expressions, gestures, and image
backgrounds.

Importantly, it is not always straightforward to separate some of these works
into these clusters. For example, (Ghosh and Dhall, 2019) investigates the
role of group-level affect to find the most influential person in images of
groups–thus touching both upon the subdomains of both visual sentiment
analysis and social signal processing.

## References

  * (1)
  * Achlioptas et al. (2021) Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J. Guibas. 2021\.  ArtEmis: Affective Language for Visual Art. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11569–11579.  https://openaccess.thecvf.com/content/CVPR2021/html/Achlioptas%5FArtEmis%5FAffective%5FLanguage%5Ffor%5FVisual%5FArt%5FCVPR%5F2021%5Fpaper.html
  * Aditya et al. (2019) Somak Aditya, Yezhou Yang, and Chitta Baral. 2019\.  Integrating knowledge and reasoning in image understanding.  _arXiv preprint arXiv:1906.09954_ (2019).  arXiv:1906.09954
  * B Kantharaju et al. (2020) Reshmashree B Kantharaju, Caroline Langlet, Mukesh Barange, Chloé Clavel, and Catherine I Pelachaud. 2020.  Multimodal Analysis of Cohesion in Multi-party Interactions. In _LREC_. Marseille, France.  https://hal.archives-ouvertes.fr/hal-02953469
  * Bagi et al. (2020) Randheer Bagi, Tanima Dutta, and Hari Prabhat Gupta. 2020\.  Deep learning architectures for computer vision applications: a study.  In _Advances in data and information sciences_. Springer, 601–612.
  * Barsalou (2003) Lawrence W Barsalou. 2003\.  Abstraction in perceptual symbol systems.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 358, 1435 (July 2003), 1177–1187.  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1693222/
  * Barthes (1980) Roland Barthes. 1980\.  Camera Lucida: Reflections on Photography, trans. R. Howard, New York: Hill & Wang. Orig.  _La Chambre Claire, Note sur la Photographie_ (1980).
  * Borghi et al. (2019) Anna M. Borghi, Laura Barca, Ferdinand Binkofski, Cristiano Castelfranchi, Giovanni Pezzulo, and Luca Tummolini. 2019.  Words as social tools: Language, sociality and inner grounding in abstract concepts.  _Physics of Life Reviews_ 29 (July 2019), 120–153.  https://doi.org/10.1016/j.plrev.2018.12.001
  * Borghi et al. (2018) Anna M. Borghi, Laura Barca, Ferdinand Binkofski, and Luca Tummolini. 2018.  Varieties of abstract concepts: development, use and representation in the brain.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 373, 1752 (Aug. 2018), 20170121\.  https://doi.org/10.1098/rstb.2017.0121 Publisher: Royal Society.
  * Borghi and Binkofski (2014) Anna M. Borghi and Ferdinand Binkofski. 2014.  _Words as social tools: An embodied view on abstract concepts_. Vol. 2.  Springer.
  * Bossard et al. (2013) Lukas Bossard, Matthieu Guillaumin, and Luc Van. 2013\.  Event Recognition in Photo Collections with a Stopwatch HMM. In _2013 IEEE International Conference on Computer Vision_. IEEE, Sydney, Australia, 1193–1200.  https://doi.org/10.1109/ICCV.2013.151
  * Bruer et al. (2020) Kaila C. Bruer, Sarah Zanette, Xiao Pan Ding, Thomas D. Lyon, and Kang Lee. 2020.  Identifying liars through automatic decoding of children’s facial expressions.  _Child development_ 91, 4 (2020), e995–e1011.  Publisher: Wiley Online Library.
  * Carson et al. (1999) Chad Carson, Megan Thomas, Serge Belongie, Joseph M Hellerstein, and Jitendra Malik. 1999\.  Blobworld: A system for region-based image indexing and retrieval. In _International conference on advances in visual information systems_. Springer, 509–517.
  * Cetinic et al. (2019) Eva Cetinic, Tomislav Lipic, and Sonja Grgic. 2019\.  A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art.  _IEEE Access_ 7 (2019), 73694–73710.  https://doi.org/10.1109/ACCESS.2019.2921101 Conference Name: IEEE Access.
  * Chandrasekaran et al. (2016) Arjun Chandrasekaran, Ashwin K. Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2016\.  We are Humor Beings: Understanding and Predicting Visual Humor. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Las Vegas, NV, USA, 4603–4612.  https://doi.org/10.1109/CVPR.2016.498
  * Chu et al. (2015) Xiao Chu, Wanli Ouyang, Wei Yang, and Xiaogang Wang. 2015\.  Multi-task Recurrent Neural Network for Immediacy Prediction. In _2015 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Santiago, Chile, 3352–3360.  https://doi.org/10.1109/ICCV.2015.383
  * Chuang et al. (2018) Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. 2018.  Learning to Act Properly: Predicting and Explaining Affordances from Images. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, 975–983.  https://doi.org/10.1109/CVPR.2018.00108
  * Corchs et al. (2019) Silvia Corchs, Elisabetta Fersini, and Francesca Gasparini. 2019\.  Ensemble Learning on Visual and Textual Data for Social Image Emotion Classification.  _International Journal of Machine Learning and Cybernetics_ 10, 8 (Aug. 2019), 2057–2070.  https://doi.org/10.1007/s13042-017-0734-0
  * da Silva et al. (2020) Emely Pujólli da Silva, Paula Dornhofer Paro Costa, Kate Mamhy Oliveira Kumada, José Mario De Martino, and Gabriela Araújo Florentino. 2020\.  Recognition of Affective and Grammatical Facial Expressions: A Study for Brazilian Sign Language. In _Computer Vision – ECCV 2020 Workshops_ _(Lecture Notes in Computer Science)_ , Adrien Bartoli and Andrea Fusiello (Eds.). Springer International Publishing, Cham, 218–236.  https://doi.org/10.1007/978-3-030-66096-3_16
  * Datta et al. (2006) Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006\.  Studying Aesthetics in Photographic Images Using a Computational Approach. In _Computer Vision – ECCV 2006_ _(Lecture Notes in Computer Science)_ , Aleš Leonardis, Horst Bischof, and Axel Pinz (Eds.). Springer, Berlin, Heidelberg, 288–301.  https://doi.org/10.1007/11744078_23
  * Dibeklioğlu et al. (2012) Hamdi Dibeklioğlu, Albert Ali Salah, and Theo Gevers. 2012\.  Are You Really Smiling at Me? Spontaneous versus Posed Enjoyment Smiles. In _Computer Vision – ECCV 2012_ _(Lecture Notes in Computer Science)_ , Andrew Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid (Eds.). Springer, Berlin, Heidelberg, 525–538.  https://doi.org/10.1007/978-3-642-33712-3_38
  * Eakins (2000) John P Eakins. 2000\.  Retrieval of still images by content. In _European Summer School on Information Retrieval_. Springer, 111–138.
  * Enser and Enser (1999) Peter Enser and Peter Enser. 1999.  Visual image retrieval: seeking the alliance of concept-based and content-based paradigms.
  * Escalera et al. (2015) Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baro, Jordi Gonzalez, Hugo J. Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. 2015.  ChaLearn Looking at People 2015: Apparent Age and Cultural Event Recognition Datasets and Results. In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_. IEEE, Santiago, Chile, 243–251.  https://doi.org/10.1109/ICCVW.2015.40
  * Gallagher and Chen (2009) Andrew C. Gallagher and Tsuhan Chen. 2009.  Understanding Images of Groups of People. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Miami, FL, 256–263.  https://doi.org/10.1109/CVPR.2009.5206828
  * Gangemi and Mika (2003) Aldo Gangemi and Peter Mika. 2003.  Understanding the semantic web through descriptions and situations. In _OTM Confederated International Conferences “On the Move to Meaningful Internet Systems”_. Springer, 689–706.
  * Ghosh and Dhall (2019) Shreya Ghosh and Abhinav Dhall. 2019.  Role of Group Level Affect to Find the Most Influential Person in Images. In _Computer Vision – ECCV 2018 Workshops_ _(Lecture Notes in Computer Science)_ , Laura Leal-Taixé and Stefan Roth (Eds.). Springer International Publishing, Cham, 518–533.  https://doi.org/10.1007/978-3-030-11012-3_39
  * Goel et al. (2019) Arushi Goel, Keng Teck Ma, and Cheston Tan. 2019.  An End-To-End Network for Generating Social Relationship Graphs. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Long Beach, CA, USA, 11178–11187.  https://doi.org/10.1109/CVPR.2019.01144
  * Gray et al. (2010) Douglas Gray, Kai Yu, Wei Xu, and Yihong Gong. 2010\.  Predicting Facial Beauty without Landmarks. In _Computer Vision – ECCV 2010_ _(Lecture Notes in Computer Science)_ , Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer, Berlin, Heidelberg, 434–447.  https://doi.org/10.1007/978-3-642-15567-3_32
  * Greisdorf and O’Connor (2002) Howard Greisdorf and Brian O’Connor. 2002.  Modelling what users see when they look at images: a cognitive viewpoint.  _Journal of Documentation_ 58, 1 (Jan. 2002), 6–29.  https://doi.org/10.1108/00220410210425386 Publisher: MCB UP Ltd.
  * Guo et al. (2021) Meiqi Guo, Rebecca Hwa, and Adriana Kovashka. 2021.  Detecting Persuasive Atypicality by Modeling Contextual Compatibility. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Montreal, QC, Canada, 952–962.  https://doi.org/10.1109/ICCV48922.2021.00101
  * Hare et al. (2006) Jonathon Hare, Paul Lewis, Peter Enser, and Christine Sandom. 2006.  Mind the Gap: Another look at the problem of the semantic gap in image retrieval.  6073 (Jan. 2006).  https://doi.org/10.1117/12.647755
  * Harpaintner et al. (2018) Marcel Harpaintner, Natalie M. Trumpp, and Markus Kiefer. 2018\.  The Semantic Content of Abstract Concepts: A Property Listing Study of 296 Abstract Words.  _Frontiers in Psychology_ 9 (2018), 1748.  https://doi.org/10.3389/fpsyg.2018.01748
  * Hoiem et al. (2008) Derek Hoiem, Alexei A. Efros, and Martial Hebert. 2008\.  Putting Objects in Perspective.  _International Journal of Computer Vision_ 80, 1 (Oct. 2008), 3–15.  https://doi.org/10.1007/s11263-008-0137-5
  * Hsiao and Grauman (2017) Wei-Lin Hsiao and Kristen Grauman. 2017.  Learning the Latent “Look”: Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images. In _2017 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Venice, 4213–4222.  https://doi.org/10.1109/ICCV.2017.451
  * Huang and Kovashka (2016) X. Huang and A. Kovashka. 2016.  Inferring Visual Persuasion via Body Language, Setting, and Deep Features. In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_. 778–784.  https://doi.org/10.1109/CVPRW.2016.102
  * Hung and Gatica-Perez (2010) Hayley Hung and Daniel Gatica-Perez. 2010.  Estimating cohesion in small groups using audio-visual nonverbal behavior.  _IEEE Transactions on Multimedia_ 12, 6 (2010), 563–575.  Publisher: IEEE.
  * Hussain et al. (2017) Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. 2017\.  Automatic Understanding of Image and Video Advertisements. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1705–1715.  https://openaccess.thecvf.com/content%5Fcvpr%5F2017/html/Hussain%5FAutomatic%5FUnderstanding%5Fof%5FCVPR%5F2017%5Fpaper.html
  * Jia et al. (2021) Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. 2021\.  Intentonomy: a Dataset and Study towards Human Intent Understanding. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Nashville, TN, USA, 12981–12991.  https://doi.org/10.1109/CVPR46437.2021.01279
  * Joo et al. (2014) Jungseock Joo, Weixin Li, Francis F. Steen, and Song-Chun Zhu. 2014\.  Visual Persuasion: Inferring Communicative Intents of Images. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 216–223.  https://openaccess.thecvf.com/content%5Fcvpr%5F2014/html/Joo%5FVisual%5FPersuasion%5FInferring%5F2014%5FCVPR%5Fpaper.html
  * Joo et al. (2015) Jungseock Joo, Francis F. Steen, and Song-Chun Zhu. 2015\.  Automated Facial Trait Judgment and Election Outcome Prediction: Social Dimensions of Face. In _2015 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Santiago, Chile, 3712–3720.  https://doi.org/10.1109/ICCV.2015.423
  * Jörgensen (2003) Corinne Jörgensen. 2003\.  _Image Retrieval: Theory and Research_.  Scarecrow Press.  Google-Books-ID: ckwiP4FyPBMC.
  * Khosla et al. (2014) Aditya Khosla, Byoungkwon An, Joseph J. Lim, and Antonio Torralba. 2014.  Looking Beyond the Visible Scene. In _2014 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Columbus, OH, USA, 3710–3717.  https://doi.org/10.1109/CVPR.2014.474
  * Kiapour et al. (2014) M. Hadi Kiapour, Kota Yamaguchi, Alexander C. Berg, and Tamara L. Berg. 2014.  Hipster Wars: Discovering Elements of Fashion Styles. In _Computer Vision – ECCV 2014_ _(Lecture Notes in Computer Science)_ , David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 472–488.  https://doi.org/10.1007/978-3-319-10590-1_31
  * Kousta et al. (2011) Stavroula-Thaleia Kousta, Gabriella Vigliocco, David P. Vinson, Mark Andrews, and Elena Del Campo. 2011\.  The representation of abstract words: Why emotion matters.  _Journal of Experimental Psychology: General_ 140, 1 (2011), 14–34.  https://doi.org/10.1037/a0021446 Place: US Publisher: American Psychological Association.
  * Li et al. (2019) Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. 2019\.  Entangled Transformer for Image Captioning. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Seoul, Korea (South), 8927–8936.  https://doi.org/10.1109/ICCV.2019.00902
  * Li et al. (2017b) Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2017b.  Dual-Glance Model for Deciphering Social Relationships. In _2017 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Venice, 2669–2678.  https://doi.org/10.1109/ICCV.2017.289
  * Li et al. (2020b) Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2020b.  Visual Social Relationship Recognition.  _International Journal of Computer Vision_ 128, 6 (June 2020), 1750–1764.  https://doi.org/10.1007/s11263-020-01295-1
  * Li et al. (2010) Li-Jia Li, Hao Su, Li Fei-Fei, and Eric Xing. 2010\.  Object bank: A high-level image representation for scene classification & semantic feature sparsification.  _Advances in neural information processing systems_ 23 (2010).
  * Li et al. (2017a) Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. 2017a.  Situation Recognition with Graph Neural Networks. In _2017 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Venice, 4183–4192.  https://doi.org/10.1109/ICCV.2017.448
  * Li et al. (2020a) Wanhua Li, Yueqi Duan, Jiwen Lu, Jianjiang Feng, and Jie Zhou. 2020a.  Graph-Based Social Relation Reasoning. In _Computer Vision – ECCV 2020_ _(Lecture Notes in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 18–34.  https://doi.org/10.1007/978-3-030-58555-6_2
  * Liu et al. (2015) Mengyi Liu, Xin Liu, Yan Li, Xilin Chen, Alexander G. Hauptmann, and Shiguang Shan. 2015.  Exploiting Feature Hierarchies with Convolutional Neural Networks for Cultural Event Recognition. In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_. IEEE, Santiago, Chile, 274–279.  https://doi.org/10.1109/ICCVW.2015.44
  * Liu et al. (2007) Ying Liu, Dengsheng Zhang, Guojun Lu, and Wei-Ying Ma. 2007\.  A survey of content-based image retrieval with high-level semantics.  _Pattern recognition_ 40, 1 (2007), 262–282.
  * Masip Rodo et al. (2012) David Masip Rodo, Alexander Todorov, and Jordi Vitrià Marca. 2012.  The Role of Facial Regions in Evaluating Social Dimensions. In _Computer Vision – ECCV 2012. Workshops and Demonstrations_ _(Lecture Notes in Computer Science)_ , Andrea Fusiello, Vittorio Murino, and Rita Cucchiara (Eds.). Springer, Berlin, Heidelberg, 210–219.  https://doi.org/10.1007/978-3-642-33868-7_21
  * Ortis et al. (2020) Alessandro Ortis, Giovanni Maria Farinella, and Sebastiano Battiato. 2020.  Survey on Visual Sentiment Analysis.  _IET Image Processing_ 14, 8 (June 2020), 1440–1456.  https://doi.org/10.1049/iet-ipr.2019.1270 arXiv: 2004.11639.
  * Panofsky and Drechsel (1955) Erwin Panofsky and Benjamin Drechsel. 1955.  _Meaning in the visual arts_.  University of Chicago Press Chicago.
  * Pratt et al. (2020) Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020.  Grounded Situation Recognition. In _Computer Vision – ECCV 2020_ _(Lecture Notes in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 314–332.  https://doi.org/10.1007/978-3-030-58548-8_19
  * Raguram and Lazebnik (2008) Rahul Raguram and Svetlana Lazebnik. 2008.  Computing iconic summaries of general visual concepts. In _2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_. IEEE, Anchorage, AK, USA, 1–8.  https://doi.org/10.1109/CVPRW.2008.4562959
  * Rothe et al. (2015) Rasmus Rothe, Radu Timofte, and Luc Van Gool. 2015\.  DLDR: Deep Linear Discriminative Retrieval for Cultural Event Classification from a Single Image. In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_. IEEE, Santiago, Chile, 295–302.  https://doi.org/10.1109/ICCVW.2015.47
  * Safra et al. (2020) Lou Safra, Coralie Chevallier, Julie Grèzes, and Nicolas Baumard. 2020.  Tracking historical changes in trustworthiness using machine learning analyses of facial cues in paintings.  _Nature Communications_ 11, 1 (Sept. 2020), 4728\.  https://doi.org/10.1038/s41467-020-18566-7 Bandiera_abtest: a Cc_license_type: cc_by Cg_type: Nature Research Journals Number: 1 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: History;Human behaviour Subject_term_id: history;human-behaviour.
  * Segalin et al. (2017) Cristina Segalin, Dong Seon Cheng, and Marco Cristani. 2017\.  Social Profiling through Image Understanding: Personality Inference Using Convolutional Neural Networks.  _Computer Vision and Image Understanding_ 156 (March 2017), 34–50.  https://doi.org/10.1016/j.cviu.2016.10.013
  * Shao et al. (2013) Ming Shao, Liangyue Li, and Yun Fu. 2013.  What Do You Do? Occupation Recognition in a Photo via Social Context. In _2013 IEEE International Conference on Computer Vision_. IEEE, Sydney, Australia, 3631–3638.  https://doi.org/10.1109/ICCV.2013.451
  * Sharma et al. (2020) Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Bjorn Gamback. 2020.  SemEval-2020 Task 8: Memotion Analysis – The Visuo-Lingual Metaphor!  _arXiv:2008.03781 [cs]_ (Aug. 2020).  http://arxiv.org/abs/2008.03781 arXiv: 2008.03781.
  * Shatford (1986) Sara Shatford. 1986\.  Analyzing the Subject of a Picture: A Theoretical Approach.  (1986).  https://doi.org/10.1300/J104V06N03_04
  * Smeulders et al. (2000) A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Dec./2000.  Content-Based Image Retrieval at the End of the Early Years.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 22, 12 (Dec./2000), 1349–1380.  https://doi.org/10.1109/34.895972
  * Solera et al. (2017) Francesco Solera, Simone Calderara, and Rita Cucchiara. 2017\.  From Groups to Leaders and Back.  In _Group and Crowd Behavior for Computer Vision_. Elsevier, 161–182.  https://doi.org/10.1016/B978-0-12-809276-7.00010-2
  * Stabinger and Rodriguez-Sanchez (2017) Sebastian Stabinger and Antonio Rodriguez-Sanchez. 2017\.  Evaluation of Deep Learning on an Abstract Image Classification Dataset. In _Proceedings of the IEEE International Conference on Computer Vision Workshops_. 2767–2772.  https://openaccess.thecvf.com/content%5FICCV%5F2017%5Fworkshops/w40/html/Stabinger%5FEvaluation%5Fof%5FDeep%5FICCV%5F2017%5Fpaper.html
  * Suhail and Sigal (2019) Mohammed Suhail and Leonid Sigal. 2019.  Mixture-Kernel Graph Attention Network for Situation Recognition. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Seoul, Korea (South), 10362–10371.  https://doi.org/10.1109/ICCV.2019.01046
  * Sun et al. (2017) Qianru Sun, Bernt Schiele, and Mario Fritz. 2017\.  A Domain Based Approach to Social Relation Recognition. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Honolulu, HI, 3481–3490.  https://doi.org/10.1109/CVPR.2017.54
  * Thomas and Kovashka (2020) Christopher Thomas and Adriana Kovashka. 2020.  Preserving Semantic Neighborhoods for Robust Cross-Modal Retrieval. In _Computer Vision – ECCV 2020_ _(Lecture Notes in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 317–335.  https://doi.org/10.1007/978-3-030-58523-5_19
  * Thomas and Kovashka (2021) Christopher Thomas and Adriana Kovashka. 2021.  Predicting Visual Political Bias Using Webly Supervised Data and an Auxiliary Task.  _International Journal of Computer Vision_ 129, 11 (Nov. 2021), 2978–3003.  https://doi.org/10.1007/s11263-021-01506-3
  * Toisoul et al. (2021) Antoine Toisoul, Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. 2021\.  Estimation of Continuous Valence and Arousal Levels from Faces in Naturalistic Conditions.  _Nature Machine Intelligence_ 3, 1 (Jan. 2021), 42–50.  https://doi.org/10.1038/s42256-020-00280-0
  * Vadicamo et al. (2017) Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. 2017.  Cross-Media Learning for Image Sentiment Analysis in the Wild. In _2017 IEEE International Conference on Computer Vision Workshops (ICCVW)_. IEEE, Venice, 308–317.  https://doi.org/10.1109/ICCVW.2017.45
  * van Bekkum et al. (2021) Michael van Bekkum, Maaike de Boer, Frank van Harmelen, André Meyer-Vitali, and Annette ten Teije. 2021\.  Modular Design Patterns for Hybrid Learning and Reasoning Systems: a taxonomy, patterns and use cases.  _arXiv:2102.11965 [cs]_ (March 2021).  http://arxiv.org/abs/2102.11965 arXiv: 2102.11965.
  * van Noord (2022) Nanne van Noord. 2022\.  A Survey of Computational Methods for Iconic Image Analysis.  _Digital Scholarship in the Humanities_ (Feb. 2022), fqac003.  https://doi.org/10.1093/llc/fqac003
  * Vanneste et al. (2021) Pieter Vanneste, José Oramas, Thomas Verelst, Tinne Tuytelaars, Annelies Raes, Fien Depaepe, and Wim Van den Noortgate. 2021\.  Computer Vision and Human Behaviour, Emotion and Cognition Detection: A Use Case on Student Engagement.  _Mathematics_ 9, 3 (Jan. 2021), 287\.  https://doi.org/10.3390/math9030287 Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
  * Varghese and Thampi (2018) Elizabeth B. Varghese and Sabu M. Thampi. 2018.  A Deep Learning Approach to Predict Crowd Behavior Based on Emotion. In _Smart Multimedia_ _(Lecture Notes in Computer Science)_ , Anup Basu and Stefano Berretti (Eds.). Springer International Publishing, Cham, 296–307.  https://doi.org/10.1007/978-3-030-04375-9_25
  * Veltmeijer et al. (2021) Emmeke Anna Veltmeijer, Charlotte Gerritsen, and Koen Hindriks. 2021.  Automatic emotion recognition for groups: a review.  _IEEE Transactions on Affective Computing_ (2021), 1–1.  https://doi.org/10.1109/TAFFC.2021.3065726 Conference Name: IEEE Transactions on Affective Computing.
  * Villani et al. (2019) Caterina Villani, Luisa Lugli, Marco Tullio Liuzza, and Anna M. Borghi. 2019.  Varieties of abstract concepts and their multiple dimensions.  _Language and Cognition_ 11, 3 (Sept. 2019), 403–430.  https://doi.org/10.1017/langcog.2019.23 Publisher: Cambridge University Press.
  * Wang et al. (2010) Gang Wang, Andrew Gallagher, Jiebo Luo, and David Forsyth. 2010.  Seeing People in Social Context: Recognizing People and Social Relationships. In _Computer Vision – ECCV 2010_ _(Lecture Notes in Computer Science)_ , Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer, Berlin, Heidelberg, 169–182.  https://doi.org/10.1007/978-3-642-15555-0_13
  * Wang et al. (2015) Limin Wang, Zhe Wang, Sheng Guo, and Yu Qiao. 2015\.  Better Exploiting OS-CNNs for Better Event Recognition in Images. In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_. IEEE, Santiago, Chile, 287–294.  https://doi.org/10.1109/ICCVW.2015.46
  * Wang and He (2008) Weining Wang and Qianhua He. 2008.  A survey on emotional semantic image retrieval. In _2008 15th IEEE International Conference on Image Processing_. 117–120.  https://doi.org/10.1109/ICIP.2008.4711705 ISSN: 2381-8549.
  * Wei et al. (2015) Xiu-Shen Wei, Bin-Bin Gao, and Jianxin Wu. 2015.  Deep Spatial Pyramid Ensemble for Cultural Event Recognition. In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_. IEEE, Santiago, Chile, 280–286.  https://doi.org/10.1109/ICCVW.2015.45
  * Workman et al. (2017) Scott Workman, Richard Souvenir, and Nathan Jacobs. 2017\.  Understanding and Mapping Natural Beauty. In _2017 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Venice, 5590–5599.  https://doi.org/10.1109/ICCV.2017.596
  * Yao et al. (2019) Xingxu Yao, Dongyu She, Sicheng Zhao, Jie Liang, Yu-Kun Lai, and Jufeng Yang. 2019\.  Attention-Aware Polarity Sensitive Embedding for Affective Image Retrieval. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Seoul, Korea (South), 1140–1150.  https://doi.org/10.1109/ICCV.2019.00123
  * Yatskar et al. (2016) Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016\.  Situation Recognition: Visual Semantic Role Labeling for Image Understanding. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Las Vegas, NV, USA, 5534–5542.  https://doi.org/10.1109/CVPR.2016.597
  * Ye and Kovashka (2018) K. Ye and A. Kovashka. 2018\.  ADVISE: Symbolism and external knowledge for decoding advertisements.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11219 LNCS (2018), 868–886.  https://doi.org/10.1007/978-3-030-01267-0_51 ISBN: 9783030012663.
  * Ye et al. (2019) Keren Ye, Narges Honarvar Nazari, James Hahn, Zaeem Hussain, Mingda Zhang, and Adriana Kovashka. 2019\.  Interpreting the Rhetoric of Visual Advertisements.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43, 4 (April 2019), 1308–1323.  https://doi.org/10.1109/TPAMI.2019.2947440 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Yee (2019) Eiling Yee. 2019\.  Abstraction and concepts: when, how, where, what and why?  _Language, Cognition and Neuroscience_ 34, 10 (Nov. 2019), 1257–1265.  https://doi.org/10.1080/23273798.2019.1660797 Publisher: Routledge _eprint: https://doi.org/10.1080/23273798.2019.1660797.
  * Yuanjun Xiong et al. (2015) Yuanjun Xiong, Kai Zhu, Dahua Lin, and Xiaoou Tang. 2015\.  Recognize Complex Events from Static Images by Fusing Deep Channels. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Boston, MA, USA, 1600–1609.  https://doi.org/10.1109/CVPR.2015.7298768
  * Zanette et al. (2016) Sarah Zanette, Xiaoqing Gao, Megan Brunet, Marian Stewart Bartlett, and Kang Lee. 2016\.  Automated decoding of facial expressions reveals marked differences in children when telling antisocial versus prosocial lies.  _Journal of Experimental Child Psychology_ 150 (Oct. 2016), 165–179.  https://doi.org/10.1016/j.jecp.2016.05.007
  * Zhang et al. (2012) Dengsheng Zhang, Md. Monirul Islam, and Guojun Lu. 2012\.  A review on automatic image annotation techniques.  _Pattern Recognition_ 45, 1 (Jan. 2012), 346–362.  https://doi.org/10.1016/j.patcog.2011.05.013
  * Zhang et al. (2015) Zhanpeng Zhang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. 2015\.  Learning Social Relation Traits from Face Images. In _2015 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Santiago, Chile, 3631–3639.  https://doi.org/10.1109/ICCV.2015.414
  * Zhang et al. (2018) Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2018\.  From Facial Expression Recognition to Interpersonal Relation Prediction.  _International Journal of Computer Vision_ 126, 5 (May 2018), 550–569.  https://doi.org/10.1007/s11263-017-1055-1
  * Zhao et al. (2021) Sicheng Zhao, Quanwei Huang, Youbao Tang, Xingxu Yao, Jufeng Yang, Guiguang Ding, and Björn W. Schuller. 2021.  Computational Emotion Analysis From Images: Recent Advances and Future Directions.  _arXiv:2103.10798 [cs]_ (March 2021).  http://arxiv.org/abs/2103.10798 arXiv: 2103.10798.