Increasing the number of detectors allows us to drive down the amount of noise
in our data. As we do that, though, “systematics” (generally due to the
limitations of the instruments/detectors themselves) become the limiting
factor in our science results. Since we require precise maps of the CMB to do
cosmology, we need to be conscious of these limitations and come up with novel
approaches for the mitigation, calibration, and removal of systematic effects,
which can be achieved through a combination of hardware and software.

“Maybe the best word of advice I have is… to find the topic/field/area that
you are truly passionate about – because we will spend a lot of time and
effort – and go all in. It does not matter what we do, but we need to be the
best version of ourselves.

Another piece of advice is that you should not follow the advice you receive,
and instead you should shape your future the way you like.”

—Prof. Raffaella Margutti, University of California, Berkeley
Associate Professor of Astronomy, Physics

##  9.

Space Policy

Dr. Lindsay DeMarchi (U.S. Congressional Fellow)
Emma Louden (Yale University)

Imagine playing a game without any rules. It would be chaotic, right?
Similarly, space policy acts as the “rulebook” for space activities. With more
countries and even private companies sending satellites, rovers, and
astronauts into space, we need to ensure that everyone plays fair and safe.

### Aerospace/Astronautics

As humans began to dream of reaching the stars, it became clear that rules
were needed. The 1967 Outer Space Treaty, signed by many countries, laid the
groundwork. This treaty states some fundamental principles:

  * —

Space is for Everyone: No country can claim ownership of the Moon or any other
celestial body. Space is the “common heritage of [hu]mankind.”

  * —

Peaceful Purposes: Military activities, like placing nuclear weapons in space,
are prohibited. Space should be a zone of peace.

  * —

Liability and Responsibility: If a country launches an object into space,
they’re responsible for any damage it might cause, both in space and on Earth.

The United Nations (UN) plays a significant role in setting up these rules.
They have a special committee called the “Committee on the Peaceful Uses of
Outer Space” (or COPUOS for short). Members from different countries come
together in this committee to discuss and agree on the best practices for
space exploration.

While the United Nations plays a pivotal role in creating global space rules,
individual countries also have their own national space policies. These
policies guide their space missions, research, and commercial activities.

Moreover, it’s not just countries anymore! Private companies, like SpaceX and
Blue Origin, are now major players in space exploration. They, too, must
follow international and national space rules.

Challenges and Considerations in Space Policy

Space Traffic Management. As more countries and private entities launch
satellites and spacecraft, space is becoming crowded. Just like cars on a
highway, these objects can collide if they’re not carefully managed.

To prevent accidents, we need a system to track all objects in space and
predict their paths. This is similar to air traffic control for airplanes but
on a much grander scale. Countries and organizations are working on creating
“rules of the road” for space to ensure that all spacecraft can coexist
without crashing into each other.

Space Debris. Over the years, many missions have left behind broken
satellites, spent rocket stages, and even lost tools. These pieces of space
junk can travel at speeds of up to 28,000 kilometers per hour! At such speeds,
even a small piece of debris can cause significant damage to a satellite or
space station. In fact, pieces larger than 10 cm can mean the end of a mission
entirely.

There are initiatives to track larger pieces of space debris and predict their
orbits. Additionally, new missions are being designed to be “cleaner,” either
by ensuring they burn up upon re-entry or by building mechanisms to remove
them from orbit at the end of their life. Some companies and agencies are even
exploring technologies to actively clean up space by capturing and de-orbiting
old debris.

The issue of space debris requires top innovative minds, as moving junk to
higher “graveyard” orbits can simply duplicate the problem farther from the
Earth, and burning up all debris in Earth’s atmosphere can pollute the upper
stratosphere with an imbalance of heavy metals.

Planetary Protection. When we send rovers or astronauts to other planets or
moons, there’s a risk that Earth microbes could hitch a ride and contaminate
these celestial bodies. This could jeopardize the search for alien life by
making it hard to distinguish between local and Earth-born organisms.
Conversely, if astronauts or probes return from these destinations, they could
potentially bring back alien microbes that might be harmful to Earth’s
ecosystem.

Strict sterilization procedures are in place for spacecraft visiting sensitive
destinations. For example, rovers destined for Mars undergo rigorous cleaning
to minimize the risk of contamination. Similarly, protocols are being
developed for safe return missions to ensure that any samples brought back are
securely contained and studied without risk to our environment.

Resource Utilization. Celestial bodies, like asteroids and the Moon, are rich
in resources that could be valuable for both space missions and use on Earth.
However, unchecked mining could damage these environments and lead to
conflicts over who has the right to these resources.

International agreements, like the Outer Space Treaty, state that the
exploration and use of outer space shall be carried out for the benefit of all
countries. This means that space resources should be used responsibly and
equitably. Discussions are ongoing about how to regulate space mining to
ensure that it’s sustainable, doesn’t harm the celestial environment, and
benefits humanity as a whole. Most recent in this list is The Artemis Accords,
an international partnership as we explore the Moon and other areas of deep
space, such as asteroids. The signatories agree to maintain transparency in
our science and guide to implement the obligations stated first in the Outer
Space Treaty.

### Dark Sky Conservation

Dark sky conservation can refer to a number of realms, depending on what you
consider to be the “sky.” For some, dark skies hearken to our immediate
atmosphere, and the phenomenon of “noctalgia,” or watching our stars disappear
from an unnatural atmospheric glow or the scattering of artificial light at
night (ALAN). For others, dark sky conservation extends from the surface of
Earth to the deep recesses of space, and anything that interrupts our view of
nature in between.

As you’ve just learned, there are few laws governing outer space and the sky.
Instead, the cutting edge of the conversation preceding additional policy is a
series of “best-practices” conversations, documents, and agreements working to
set the tone for how we should proceed, regulate, or even think about the sky.
These conversations are like a wet clay, and your voice matters.

The Atmosphere You may have noticed that there are some parts of the planet
where you can barely see any stars at all, and others where you can see the
whole Milky Way! That is because of ALAN and the large number of unshielded,
unprotected lights humans use to illuminate the dark. The great thing about
this form of light pollution is that it’s immediately fixable. Just use proper
lighting, et voila! The stars return!

Imagine trying to fill a cup of water using a pipe jetting out of your wall.
Water would spurt everywhere, and you’d be lucky to catch any of it in your
cup. Instead, we tend to use faucets such that we may easily direct the water
into our vessels. Light should be treated the same way! In other words,
solving ALAN doesn’t mean turning all the lights off; it means directing light
like a faucet. The next time you’re high in a plane or on a rooftop, ask
yourself whether or not it’s necessary for photons from the ground to be
reaching your eye. Could that be repaired with a simple cap on top of a naked
light bulb?

The International Dark Sky Association (recently re-branded as “Dark Sky
International”) is one international organization that seeks to educate and
promote more conscious uses of proper lighting. Not only does better lighting
reclaim our view of the night sky, but we save money, increase safety, save
energy, and help critically affected wildlife. Think about it – animals,
plants, trees – pretty much all life on Earth developed for millions of years
in a cycle of daylight and natural darkness.

From Satellites One of the top 20 brightest stars in the sky is an artificial
satellite. Currently, there are about 5,000 functioning satellites in orbit,
but 530,000 are in various stages of planning, and that number is constantly
changing! (Seriously, you should check for yourself what the current estimates
are!)

Dark Sky International published their own list of five principles to preserve
the quiet enjoyment of the night sky and protect the general public from the
impacts of “megaconstellations,” or large swarms of satellites:

  1. 1.

Stewardship of the night sky is a shared responsibility that requires
participation and consultation with all stakeholders.

  2. 2.

The cumulative impact on night sky brightness attributed to satellites does
not exceed 10 percent above natural background levels.

  3. 3.

Maintained satellite brightness is below the threshold for detection by the
unaided eye.

  4. 4.

Satellite visibility is an unusual occurrence.

  5. 5.

Launch schedules and orbital parameters are publicly available in advance.

In addition to the general public, astronomers who take long exposures of the
sky often find their data is tainted or ruined with bright satellite streaks.
Though some companies, such as SpaceX, have agreed to paint their satellites
darker, the fuselages themselves still block photons from reaching the Earth.
This can be particularly harmful to transient astronomy and the world of
cutting edge science, where we may not get a second chance to view an event.

The IAU CPS (or, the International Astronomical Union’s Centre for the
Protection of the Dark and Quiet Sky from Satellite Constellation
Interference) is an international group of stakeholders with various “hubs.”
These hubs are pillars that include Community Engagement, Policy, Industry,
and Science, who all work together to compromise on the issue and suggest
regulation and inform members of the public and industry. Their report and
recommendations in a series “Dark and Quiet Skies for Science and Society”
have been elevated to an agenda item in UN meetings on COPUOS. Additionally,
NSF’s NOIRLab have held two conferences, summarized in proceedings called
SATCON1 and SATCON2 they are invaluable touchstone documents that host
recommendations for astronomers and industry alike, such that the negative
impacts of satellites on astronomy can be mitigated.

(Radio) Quiet Skies Your Wi-Fi, your smart refrigerator, your car, and your
cell phone all communicate via radio waves. In fact, so does everything else
that uses satellites: banks, international trading ships, the military, and
GPS! Oh, and your television broadcasts, too. The radio spectrum is broken up
into wavelengths that can be registered through the ITU (International
Telecommunication Union). As industry in space increases, so does the demand
for radio wavelengths. Oftentimes, this overcrowding can create unintentional
leakage and disrupt protected wavelengths, such as those reserved for
astronomy.

For perspective, the radio observatory called The Very Large Array (VLA) is
sensitive enough to detect a flip-phone on Jupiter! Even if cell phones occupy
bands adjacent to protected wavelengths and place their satellites on opposite
sides of the Earth from the VLA, interference can spill over and be disruptive
to science. Oftentimes, radio telescopes that observe low frequencies near the
5 Ghz band feel their data is too noisy to warrant cleaning or further use.

Figure 14: We can bring the Milky Way back to our cities. With proper
lighting, even urban areas can see the night sky!

“Recognize that the people who have accomplished amazing things have often
done so with the help of mentors who have supported them in different ways;
they didn’t know all the answers from the start.

It is important to have multiple people who you can confide in or ask
questions. Sometimes your most important mentors will be peers or people who
are just one year ahead of you! Lastly, once a mentor, always a mentor – it’s
ok to reach out even if you don’t talk every week!”

—Prof. Katie Breivik, Carnegie Mellon University
Assistant Professor of Physics

##  10.

Machine Learning & Artificial Intelligence

Yasmeen Asali (Yale University)

Imagine you have a cat and a dog, and you want to create a simple rule for
telling them apart. You might say, “Cats have pointy ears, and dogs have
floppy ears.” This is your basic algorithm. An algorithm just means a set of
clear and specific instructions. So for this example, your algorithm might be:
look at the ears, and if they’re pointy, you say it’s a cat; if they’re
floppy, you say it’s a dog.

But what if you come across a cat with floppy ears or a dog with pointy ears?
Your simple rule might not work very well in those cases. That’s where machine
learning comes to the rescue! Now, instead of relying on a simple rule, you
gather lots and lots of pictures of cats and dogs. You show these pictures to
a computer and say, “This is a cat,” or “This is a dog”. Machine learning
refers to how your computer can learn from looking at lots of examples and get
better at recognizing things.

### How do Computers Learn?

Let’s continue with our cat and dog analogy. As we feed cat and dog pictures
to the computer, it will start to come up with rules on its own to decide
whether the animal in a picture is a cat or a dog. Here’s how it works:

  1. 1.

Seeing Examples: You feed the computer thousands of cat and dog pictures. It
doesn’t know anything about cats or dogs at first, but as it looks at all
these pictures, it starts to notice things on its own. It might see that cats
often have whiskers, and dogs have wet noses.

  2. 2.

Learning Patterns: The computer starts to learn from these examples. It learns
to recognize patterns, like pointy ears, whiskers, and wet noses. It figures
out that these are important clues to tell cats and dogs apart.

  3. 3.

Decision-Making: Now, when you show the computer a new picture, it looks for
these clues. If it sees pointy ears and whiskers, it might confidently say,
“This is a cat.” If it sees floppy ears and a wet nose, it might say, “This is
a dog.”

Importantly, the computer has come up with its set of rules on its own! It can
be really hard for a human to sit down and write a set of rules that can
differentiate every single type of dog from every single type of cat. With so
many different breeds, the rules need to get increasingly complex to account
for all the possible combinations of features. The really special thing about
machine learning is that the computer can learn to recognize those patterns on
its own, the same way that a baby human will learn to recognize the difference
between cats and dogs without having to think through a checklist of features.

Training

The process of seeing examples and learning how to make decisions about them
is called training. Once a machine learning algorithm is trained, we can use
it on new examples it’s never seen! There are two main ways to train a ML
algorithm: supervised learning and unsupervised learning. The basic difference
between these two methods is that supervised learning uses labeled data, while
unsupervised using does not require labeled data.

Supervised Learning: Let’s think about our cat and dog sorting example. When
the computer is still learning, we are labeling the pictures as cats or dogs
so it can figure out the difference between them. This is called supervised
learning, because we are providing the computer with labels that inform its
learning. Then, for each subsequent or new image we show the computer, we can
ask it to label the name image either cat or dog.

Unsupervised Learning: Now, think of sorting cat and dog pictures without any
labels. You put similar images together without someone telling you which
group is which. You just notice the similarities and differences. In
unsupervised learning, the computer works similarly. It looks for patterns or
groups in the data without being given specific labels. It’s like exploring
and discovering on its own. The computer might group similar pictures without
being told what’s in each picture, so it might not necessarily create just two
groups. Maybe the computer finds patterns based on breed and it creates more
than two groups separating out types of cats and types of dogs! Unsupervised
learning is really useful for data that we don’t already have human labels
for.

### Machine Learning in Astrophysics

Astronomy generates an enormous amount of data from telescopes and space
missions. Machine learning can help scientists sift through this data,
identify patterns, and make sense of it all. For instance, ML algorithms can
detect and classify celestial objects, like stars and galaxies, by analyzing
the light we detect. They can also recognize transient events, such as
supernovae, gravitational wave signals, or exoplanet transits, by picking out
patterns in the data that may be otherwise hard to identify. Here are a few
examples of the ways in which machine learning is used to aid astronomical
research:

  * —

Galaxy Classification with Galaxy Zoo: Galaxy Zoo is a citizen science project
that uses the collective power of human pattern recognition to generate large
labeled training sets for a machine learning classification algorithm.
Participants are usually given a set of questions about the observed galaxies.
These questions might include whether the galaxy is spiral or elliptical, the
presence of distinctive features like bars or spiral arms, and other
characteristics relevant to astronomers studying galaxy morphology. The
classifications provided by the citizen scientists are aggregated and used by
astronomers to further their understanding of galaxy evolution, structure, and
other properties. The large-scale classification efforts facilitated by Galaxy
Zoo allow researchers to process big datasets much more quickly than would be
possible with automated methods alone!

  * —

Predicting Solar Flares with the Solar Dynamics Observatory: Machine learning
is used to study day-to-day changes on our Sun based on data from the space-
based Solar Dynamics Observatory (SDO). The SDO generates a vast amount of
data, including images and videos of the Sun in different wavelengths. Machine
learning models can analyze historical data from the SDO to identify patterns
and precursors associated with solar flares. These models can then be used for
solar flare prediction, providing valuable information for space weather
forecasting and mitigating potential impacts on satellite communications and
power grids.

  * —

Hunting for Exoplanets: The Kepler/K2 observatory has collected an immense
amount of data on more than 150,000 stars over several years. Machine learning
accelerates the analysis process, allowing us to discover exoplanets that
might be difficult for traditional methods to detect. For instance, some
exoplanet systems host multiple planets orbiting the same star. The signals
from these systems can overlap and become intricate, making them challenging
for human observers to differentiate. Machine learning excels in recognizing
complex patterns within data, enabling the identification of multiple-planet
systems with greater efficiency and accuracy.

These examples represent just a glimpse of the many ways ML is transforming
astronomy. As ML applications continue to evolve, it’s increasingly clear that
this powerful tool not only revolutionizes scientific research but also
quietly influences various facets of our everyday lives. ML is used by social
media companies to sort recommended posts, it’s used in the facial recognition
software that can unlock your phone, even the auto-complete suggestions in
text messaging apps leverage ML. Keep an eye out for the subtle yet impactful
presence of machine learning in your own life!

“There are several pieces of advice that I would like to give. First off:
Trust yourself. Don’t let other people tell you what you are or aren’t good
in. No one knows better than you what your qualities are. I also want to pass
on one of the best pieces of advice I’ve ever received; Some you win, some you
lose, but if you let those get to you, you never get anywhere. Whatever you
want from life, chances are you won’t get it straight away, but don’t give up
too easily. Try, try twice, try three times. And if route A doesn’t work, try
route B, and otherwise route C.”

—Prof. Silvia Toonen, University of Amsterdam
Assistant Professor of Astrophysics

##  11.

FAQ

### So what is the difference between an astronomer and an astrophysicist?

There is none! (At least not anymore … traditionally, the distinction was
whether or not spectroscopy was being used in research.) Now, when someone
uses the term _astronomer_ or _astrophysicist_ , they will use them
interchangeably. Colloquially within the field, some people will reserve the
term _astronomer_ for observers and use _astrophysicist_ as the general term
regardless of focus on instrumentation/observation/theory. In truth, everyone
working in the field is doing astrophysics (i.e., studying the physical
mechanisms that govern the universe) and almost no one is doing astronomy in
the classical sense of the term, so _astrophysicist_ often fits the field
better.

A different perspective comes from the “guidance” you’ll sometimes hear that
if asked what you do, astronomer will continue the conversation and
astrophysicist will end it. From experience, people hear these terms the same
way, so even this is not a foolproof way to decide which term to use.

There are also some subfields of astrophysics that like to further
differentiate themselves with their own names. Cosmologists are
astrophysicists who work on cosmology and, though it is less pervasive as a
term, exoplaneteers work on exoplanets/planetary science. Rarely, if ever,
will cosmologists introduce themselves as _astrophysicists_ – generally they
will say that they are a _cosmologist_ or a _physicist_.

### What is the difference between a physicist and an astrophysicist?

An astrophysicist is just a type of physicist, so the distinction is one of
specificity. Astrophysics is a subfield of physics along with
atomic/molecular/optical, condensed matter, high energy, quantum, … so saying
that someone is an astrophysicist is like saying that someone is a condensed
matter physicist.

Some confusion arises from the fact that astrophysics is housed in a separate
department at some institutions. This is generally due to the historic
significance of astronomy (in many universities in the United States, for
instance, astronomy was one of the first departments established after the
college was founded) and the fact that it is a large and well-funded subfield.
To some extent, this is no different than a biophysicist working in a
biophysics institute, but it does mean that astronomy departments have more
educational autonomy (like running their own classes), which is an advantage
that is often not afforded to institutes.

### What kind of education/training is necessary to become an astrophysicist?

Long-term research jobs in astrophysics almost always require a PhD. To work
at a national lab or a university as a scientist (or professor in the latter
case), the usual track is:

  * -

$\sim 4$ years for a bachelor’s degree (generally in physics, potentially in
astronomy/astrophysics)

  * -

Possibly $\sim 2$ years for a master’s degree – though not required for PhD
programs in the United States, it is increasingly common for PhD applicants to
hold a master’s

  * -

$\sim 6$ years for a PhD – in STEM, PhD programs are fully funded, meaning
that you will not have to pay tuition and will actually receive a livable
(though generally non-competitive compared to industry) wage for the
teaching/research you do

  * -

Generally 3 - 6 years in postdoctoral research positions

After those 10 - 15 years, you will be competitive for permanent positions
like tenure-track professorships at universities or scientist roles at
nationally funded facilities.

There are other positions adjacent to astronomy that require much less in the
way of preparation. For instance, telescope operators are absolutely integral
to the science astronomers do, and may only have a bachelor’s degree (ideally
in physics/astronomy). Science writing positions are generally open to those
with only an undergraduate degree, but may require more of a background in
journalism.

Because graduate school admissions are increasingly competitive, it is
becoming more and more usual for prospective applicants to do post-
baccalaureate research at a university (frequently their undergraduate alma
mater) in order to get full-time research experience. These positions often
last one or two years and are paid comparably to graduate research positions.
Other students may choose to do a master’s degree before (re-)applying for PhD
programs. Depending upon the institution, these programs may or may not charge
tuition and/or pay a stipend.

### Should I study physics or astrophysics?

At the undergraduate level, the answer is generally physics, given the
expectation that students entering graduate programs will have the same
knowledge of advanced math, classical mechanics, statistical mechanics,
electrodynamics and electromagnetism, and quantum mechanics that a physics
undergraduate affords. If there is the option, you may be able to major in
physics with a concentration in astronomy/astrophysics, so that your elective
courses are cross-listed grad/undergrad courses in astro, or you may be able
to double major in astronomy/astrophysics. Many standalone astronomy majors
were established for students with an interest in the science, but no
aspirations to pursue it professionally. There are exceptions, though, and you
should confirm on an institution-specific level.

Some people will also apply to astro grad programs from geology/planetary
science, math, or computer science majors, but this is a non-standard path
and, since graduate admissions are incredibly competitive, there is no
guarantee of success in getting into a PhD program this way (or, frankly, even
with a physics background).

In graduate school, deciding between a PhD in physics and
astronomy/astrophysics is much more dependent on the institution and
supervisor with whom you would like to work. Standard advice is that astro-
specific programs are generally better resourced than the astro portion of a
physics department, and you will be able to focus more heavily on research-
relevant coursework, but even within a university with both departments, you
may prefer the feel and requirements (for coursework, teaching, etc.) of the
physics department. The other deciding factor may be that you prefer to take a
physics vs. astronomy qualifying exam (or vice versa), but this is something
to think about/decide much further down the line.

### How do I get involved with undergraduate research?

There are both funded and unfunded opportunities for undergraduate research.
The simplest positions to get are unfunded research assistantships, which are
generally with a professor in your department. The tried-and-true approach is
to cold contact professors who do research that interests you and ask if they
have any opportunities for undergraduates to participate. This may be somewhat
easier with faculty you know and have impressed – i.e., you’ve already taken a
class with them and performed very well – but not all research groups have
roles for undergraduates, so you may have to reach out to people you don’t
know.

These unfunded positions may become funded after you’ve proven yourself; there
are usually also university-wide undergraduate research grants you can apply
for to support work you’re doing in a professor’s group.

The other avenue for getting a funded position is to apply for more
prestigious undergraduate research fellowships, like the NSF REU (only open to
US citizens) or the DAAD RISE, which fund summer research on specific projects
that are external to your undergraduate institution. These are very
competitive, so you will generally need existing research experience for a
successful application, though there is some priority given to students from
liberal arts colleges or similar schools where research is generally de-
prioritized and less available.

### How do I know what research opportunities exist at a school?

This is a consideration at both the undergraduate and graduate level. For
undergraduates, though, it is somewhat less pressing – you are not locked in
by the research you did in college and can (usually) change your focus in
graduate school. (It serves to remember that astrophysics is already a sub-
field, so further narrowing is generally not crucial until you begin a PhD.)

It may be important to you to have lots of options for research
direction/modality. In that case, it’s best to consider this before you select
your undergraduate institution. Schools with a large astro presence (generally
due to having a dedicated department and/or an institute of some kind) will
often have more diversity in terms of the questions being covered. At the same
time, smaller institutions may offer more research opportunities to
undergraduates. Certainly, liberal arts colleges and predominantly
undergraduate institutions will often have substantially less breadth in
available astrophysics research, but undergraduate research positions may be
more plentiful and come with more responsibility.

If you know in advance that you are going to pursue (astro)physics, check the
department website for universities you are considering attending for college.
They will often list research areas on which their faculty focus and may
discuss specific projects on which students have worked. Some departments will
also tout their commitment to undergraduate research and list fellowship
opportunities or open positions. Some universities require an undergraduate
thesis in order to finish a bachelor’s degree. Theses generally signal a
department commitment to involving college students in research early, so this
can be another positive indicator.

It also does not hurt to ask – once admitted and at the point of deciding
between schools, find the contact information for your prospective
department’s director of undergraduate studies (or equivalent) and ask
explicitly about undergraduate research opportunities. (Note that a non-
response is not a reason to worry. Academics are overworked and often receive
a deluge of emails – if you don’t hear back, your message was likely buried,
not ignored.)

### What skills will make me more competitive for undergraduate research
positions?

At this point, coding is critical for most aspects of astrophysical research.
There are a number of languages that people use – commonly IDL, Julia, C/C++,
and Python, though Python is by far the most used in astro research.
Completing free beginner courses from DataCamp or Codeacademy will get you up
and running with these skills, but the best way to learn is to write code for
a specific application. You can also refer to something like Python for
Astronomers (prappleizer.github.io) as a reference or look at other
guidebooks/tutorials (some listed at astroteaching.github.io/computational).

Additionally, learning version control and collaborative work with git, via
GitHub, Bitbucket, or similar, will give you a leg up. Version control is
critical when you are developing software or even individual scripts, and the
collaborative aspects of git allow you to contribute to larger projects where
you are not necessarily the primary developer, but have reason to access, and
potentially edit, source code.

Learning LaTeX will also help – both for typesetting problem sets/homework
assignments (quick way to become teacher’s pet!) and working on paper drafts.
Instead of Word, Pages, or free/open source alternatives, almost all drafting
in (astro)physics – including for this booklet! – is done in LaTeX . There are
local installations + editors like TeXShop, but if you have LaTeX installed on
your computer, you can also use any text editor (some favorites for coding and
other applications are Sublime Text and VSCode) to draft and compile your
text. Collaborative work is often done in Overleaf, which is a cloud-based
option – even if you’re not at the point of collaborating, it can still be
very useful to access your documents from multiple devices without worrying
about syncing.

For anyone just starting a project, it is also really important to understand
the existing literature on the topic. Your supervisor may give you papers to
read to start out, but finding others that are relevant (particularly as they
are released) is a great way to show interest and initiative and learn about
where the field is headed.

### Where do I find research papers to read?

Every astro-relevant paper, at least for the last decades, is indexed in
NASA/ADS (ui.adsabs.harvard.edu); this is a primary resource for anyone doing
astrophysics research. Pre-prints are also ingested by ADS. Those can be
accessed directly (as they come out) on arXiv (arxiv.org); pre-prints are
released $\sim 9$pm ET (Sunday through Thursday) – to see what’s new, you can
click on “new” next to astro-ph.

Some pre-prints are summarized in astrobites (astrobites.org), which aims to
make the literature accessible to undergraduates. Especially as you’re
starting out, it can be good practice to read both the paper and the
astrobites summary to better digest the content.

### How do I get actively involved in my university’s (astro)physics
department?

In addition to getting involved with research as an undergraduate, which is
often critical to full integration in your department, attending seminars and
colloquia is a fantastic way to be exposed to new science and meet your future
colleagues. Even if you don’t understand what is being discussed initially,
the more talks you attend (and the further you get in your undergraduate study
in general), the more you will take away from each talk. Eventually, you will
be giving talks like that yourself.

Additionally, your department may have a chapter of the Society of Physics
Students or affinity organizations like Women in Physics. These are wonderful
opportunities to take more initiative and be fully involved in pre-
professional activities.

### Outside of class/research, what should I do to prepare myself for a career
in astronomy?

Building personal/professional connections and having a network of mentors is
really important in the early stages of your career. (That’s one of the goals
of SIRIUS B’s VERGE program!) Involving yourself in pre-professional
activities like those listed above does a lot in this respect, and being
visible in your department (as from going to talks) can help as well.

Reading broadly and understanding the common language of astrophysics (some of
which is subfield specific) will also make for an easier transition to doing
science professionally.

### What can I do now to prepare for a career in astrophysics?

The best thing you can do to prepare for a career in astrophysics is gain
solid footing in mathematics. There are also some formal programs through
which you can get advanced pre-college training in astrophysics, namely the
Yale Summer Program in Astrophysics (YSPA; yspa.yale.edu) and Summer Science
Program (SSP; summerscience.org). Some universities will also run daytime
summer research programming for local high school students – this will usually
be less intensive than YSPA or SSP, but will still amount to an enormous leg
up for participating students.

### I want to play with data now. How do I do that?

The good news in astronomy is that so much of our software and data (both from
observations and simulations) are free and open source by default!

Observational data is generally stored in FITS – Flexible Image Transport
System – files. These can be easily opened and perused with Python if you’re
comfortable coding or you can use SAOImageDS9 (DS9 for short;
sites.google.com/cfa.harvard.edu/saoimageds9) or similar to go through a
graphical user interface.

You can find data of all kinds on different survey and data access websites,
for example:

  * —

Barbara A. Mikulski Archive for Space Telescopes (MAST) –
mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html

  * —

Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Survey –
legacysurvey.org

  * —

Hyper Suprime-Cam Subaru Strategic Program –
hsc-release.mtk.nao.ac.jp/doc/index.php/tools

  * —

Sloan Digital Sky Survey – sdss4.org/dr17/data_access

If you are comfortable coding, you have even more options – for instance, you
could play with other types of data like those from:

  * —

Arecibo Legacy Fast ALFA Survey – egg.astro.cornell.edu/alfalfa/data

  * —

Gaia stellar information – gea.esac.esa.int/archive

  * —

IllustrisTNG (cosmological simulation) – illustris-project.org/data

  * —

NASA Exoplanet Archive – exoplanetarchive.ipac.caltech.edu

  * —

X-ray transient light curves – github.com/avapolzin/X-rayLCs

You can generally find documentation by poking around these pages, but if that
isn’t sufficiently helpful, don’t worry – these are resources intended for
professionals and are not made specifically accessible to the public.

There are also websites where you can engage with data via the site including:

  * —

JADES interactive viewer – jades.idies.jhu.edu

  * —

Legacy Viewer – legacysurvey.org/viewer

  * —

Magnetar Outburst Online Catalog analysis tab –
magnetars.ice.csic.es/#/outbursts

  * —

NASA Exoplanet Archive confirmed planet plotting tool –
exoplanetarchive.ipac.caltech.edu/cgi-bin/IcePlotter/nph-
icePlotInit?mode=demo&set=confirmed

If you’re looking for a more sanitized experience where you can engage in
citizen science, the Zooniverse (zooniverse.org) has a number of different
projects in astronomy among other topics. This can be a really fun way to
spend some time that also contributes to active science!

If you have a telescope, you can always take your own data that can be
analyzed! Assuming you have the equipment, you may want to contribute to the
monitoring of time-domain events (like exoplanet transits and stellar
variability); you can then add to the American Association of Variable Star
Observers (AAVSO; aavso.org) repositories. This will require that any data you
take are properly calibrated, which necessitates some additional work, but is
a terrific way for an excited amateur to get involved in observational
astronomy.

### How do I connect with other women in astrophysics?

Universities often have a number of organizations that connect women in STEM
and specific scientific fields. Most universities will have some version of a
Society of Women in Physics, which can be a supportive discipline-specific
community even for undergraduates. There are often also Women in Science
groups (sometimes dominated by one field or another) that you can get involved
with at that level. At the undergraduate level, the American Physical Society
holds the Conference for Undergraduate Women in Physics (CUWiP), which is
hosted simultaneously at a number of regional hubs throughout the United
States. CUWiP generally focuses as much on pre-professional development as it
does on science, so this is an opportunity to talk about your research and
learn best practices for navigating graduate applications, ignoring/mitigating
imposter syndrome, engaging in self-advocacy, etc.

There are initiatives that look to aggregate lists of women in astrophysics in
order to showcase their work and generate a professional network. For instance
1400 Degrees (1400degrees.org) is a directory of women and gender minorities
in (astro)physics. There is also If/Then (ifthenshecan.org), which showcases a
somewhat smaller cohort across all of STEM.

Additionally, as you progress in your career, you will find that individual
mentorship – both informal and formal – will play a role. Most, if not all, of
the women who contributed to this guide, can point to female scientists who
shaped their path. These mentoring situations happen organically over time, so
there isn’t a framework for approaching them; just trust that you, too, will
end up with scientists in your extended network who will advise and support
you, and that you too will eventually be able to reach back and support the
interested girls behind you.

“The gap between us being average and being great often involves limiting
beliefs that we have internalized about ourselves. They can block you in a big
way.”

—Dr. Sthabile Kolwa, University of Johannesburg
Lecturer in Physics

In case you don’t know her story, it’s worth learning about Félicette, the
first cat in space, who was, perhaps apocryphally, nabbed off the streets of
Paris, where she was living happily enough as a stray. She became France’s
first astronomical envoy and was the only cat to ever cross the Kármán line.
Despite surviving her ordeal and returning to Earth unharmed, she eventually
gave her life for science, when she was euthanized to allow French scientists
to better study the effects of her trip to space. She is now memorialized with
a statue at the International Space University in Strasbourg, France.

The phrase under her portrait translates to “Thank you for your participation
in our success on October 18, 1963.”

See img/back.png


# UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of
Vision Language Models

Xinyu Pi*$1$ Mingyuan Wu*$2$ Jize Jiang*$2$ Haozhen Zheng*$2$
Beitong Tian$2$ Chengxiang Zhai$2$ Klara Nahrstedt$2$ Zhiting Hu$1$
$1$University of California San Diego $2$University of Illinois Urbana-
Champaign
<EMAIL_ADDRESS>{mw34, jizej2<EMAIL_ADDRESS>
* indicates equal contribution

###### Abstract

Smaller-scale Vision-Langauge Models (VLMs) often claim to perform on par with
larger models in general-domain visual grounding and question-answering
benchmarks while offering advantages in computational efficiency and storage.
However, their ability to handle rare objects, which fall into the long tail
of data distributions, is less understood. To rigorously evaluate this aspect,
we introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. This
benchmark focuses on systematically testing VLMs with both large and small
parameter counts on rare and specialized objects. Our comprehensive analysis
reveals that while smaller VLMs maintain competitive performance on common
datasets, they significantly underperform on tasks involving uncommon objects.
We also propose an advanced, scalable pipeline for data collection and
cleaning, ensuring the UOUO benchmark provides high-quality, challenging
instances. These findings highlight the need to consider long-tail
distributions when assessing the true capabilities of VLMs.

UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of
Vision Language Models

Xinyu Pi*$1$ Mingyuan Wu*$2$ Jize Jiang*$2$ Haozhen Zheng*$2$ Beitong Tian$2$
Chengxiang Zhai$2$ Klara Nahrstedt$2$ Zhiting Hu$1$ $1$University of
California San Diego $2$University of Illinois Urbana-Champaign<EMAIL_ADDRESS>{mw34, jizej2<EMAIL_ADDRESS>* indicates equal contribution

Figure 1: UOUO Data Curation Pipeline. Snowflake means frozen weights, and
fire means tune-able weights.

##  1 Introduction

The advent of Vision-Language Models (VLMs) has marked a revolutionary leap in
the integration of natural language processing and computer vision, largely
due to the capabilities of the self-attention mechanism and the Transformer
architecture Vaswani et al. (2023). These technologies allow VLMs to
effectively process and fuse information from both text and images, leading to
significant advancements in tasks that require multimodal understanding, such
as visual question answering and image captioning Radford et al. (2021); Li et
al. (2023); Alayrac et al. (2022); Xu et al. (2023); Young et al. (2014).

VLMs, trained on large-scale datasets, typically boast high performance on
general tasks involving everyday objects and common scenarios Li et al.
(2024); Du et al. (2022); Wang et al. (2023). However, models of smaller
scale, defined here as having fewer than 70 billion parameters, often claim to
match the capabilities of their larger counterparts on general domain tasks
Lin et al. (2015); Agrawal et al. (2016); Yu et al. (2016); Liu et al. (2024);
Goyal et al. (2017); Yu et al. (2023b) while offering advantages in
computational efficiency and storage. Despite these claims, the No-Free-Lunch
Theorem Wolpert and Macready (1997) suggests that these smaller models may
compromise on their ability to handle less common or more complex scenarios
that lie in the long tail of data distributions.

One natural and intuitive hypothesis is that they are sacrificing their
fitness to the elements on the long tail of the distribution. Empirical
observations of real-world data frequently align with Zipf’s and Power Law
Piantadosi (2014); Clauset et al. (2009), which indicates that while some
objects and concepts are exceedingly common, a vast number of them are rare
and fall into the long tail of the distribution . Understanding how well VLMs
handle these rare and uncommon instances is crucial for assessing their true
robustness and applicability across diverse and nuanced contexts.

Despite the importance of this evaluation, there is currently a lack of
dedicated benchmarks that systematically test VLMs on objects and concepts
that are significantly outside the everyday norm. To address this gap, we
introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. The object
class distribution of UOUO is systematically out of common image sources such
as ImageNet Russakovsky et al. (2015), COCO Lin et al. (2015), and Open Image
Dataset Kuznetsova et al. (2020). Our goal is to rigorously test and quantify
the performance of both large-scale and small-scale VLMs on elements from the
long tail of the distribution to showcase their knowledge gap.

The contribution of our work is three-fold. (1) We compile a million-scale
dataset specifically designed to include uncommon and uncontextualized
objects, which are rarely encountered in everyday contexts but are significant
in specialized domains. (2) We evaluate the performance gap between large-
scale and small-scale VLMs when dealing with these rare elements, showcasing
the significant knowledge and performance gap between large- and small-scale
model on the long-tail distributions. (3) We propose a systematic pipeline for
automatic and scalable data collection and cleaning, ensuring high-quality and
representative testing instances.

##  2 Related Work

#### Real-world VQA Benchmarks

Based on our survey, the typical real-world visual question answering datasets
(excluding mathematics, celebrity, landmark, place, OCR and chart-reading)
used in popular open-source VLMs such as LLaVa Li et al. (2024), CogVLM Wang
et al. (2023) BLIP2 Li et al. (2023), Qwen VL Bai et al. (2023) and MiniCPM-V
Yu et al. (2023a) includes the following: COCO Lin et al. (2015), RefCOCO Yu
et al. (2016), NoCAPs Agrawal et al. (2019), MMBench Liu et al. (2024), VQA-v2
Goyal et al. (2017), OK-VQA Marino et al. (2019), MME Fu et al. (2024), GQA
Hudson and Manning (2019).

Much to our surprise, it turns out that the image sources of GQA, RefCoCo, OK-
VQA, MME Coarse-Grained Recognition, VQA-v2, and a significant proportion of
MMBench are all direct random samples from COCO. Only NoCAPs features novel
object classes (sourced from the 600-categories Open Image Dataset Kuznetsova
et al. (2020) outside COCO’s less-than-100 common classes. This showcases the
significant limitation of categorical diversity of extant VQA datasets. The
knowledge and performance gap between the small- and large- scale VLMs might
be concealed in such low coverage and diversity.

#### Existing Datasets with Uncommon Object Labels

In extant datasets, Stanford Cars Krause et al. (2013), CUB-bird Wah et al.
(2011), Deepfish Saleh et al. (2020), ROCOv2 Rückert et al. (2024), FGVC-
Aircraft Maji et al. (2013) also features rare object labels. Some non-
academic mine & stone datasets, and chemical objects datasets can also be
found on internet. However, the typical emphasis of these datasets is either
fine-grained subtype or subspecies of common objects, or domain-specific
expert knowledge. In realistic use cases such as autonomous car or embodied
robotics, such knowledge might have limited generalizability.

##  3 Data Curation and Filtering

###  3.1 Domain Selection and Scraping

To construct the UOUO (Uncontextualized Uncommon Objects) benchmark, we began
by selecting specific domains that are rich in specialized knowledge yet
contain objects and tools that are rarely encountered by the general public.
Our focus was on the industry sector, given its diversity and the presence of
numerous specialized tools and equipment. These artificial tools are
significantly out of the distribution of ImageNet, COCO, and Open Image
Dataset.

We used Wikipedia as a starting point, targeting the page dedicated to
manufacturing (https://en.wikipedia.org/wiki/Manufacturing). For each sub-
sector identified within this domain, we employed GPT-4-Turbo OpenAI et al.
(2024) to generate a list of the top 50 objects or tools pertinent to experts
in the field but obscure to the general populace. This list was generated
through prompt-based querying, asking the model to identify objects that are
crucial within the industry but not commonly known.

Once we had our list of uncommon objects, we performed a Google Image Search
for each object name. For each query, we collected the top 50 image results.
This approach allowed us to gather a diverse set of images representing each
object under different conditions and contexts. For detailed dataset
statistics of UOUO, we refer readers to Appendix A.

#### Mannual Annotation

The image instances collected from Google Image Search can be noisy, with
perhaps one fifth irrelevant instances for each queried uncommon category. To
ensure the quality and relevance of the dataset, we implemented a rigorous
annotation and cleaning process, combining manual and automated techniques.
Our team manually reviewed the collected images for each object category to
identify and remove outliers and noisy data. Categories with consistent visual
representation across examples were retained, while those filled with
ambiguous or irrelevant images were discarded. This initial curation aimed to
maintain high fidelity to the object’s intended representation.

#### Automatic Data Cleaning

We utilized the CLIP model to further enhance the dataset. CLIP (Contrastive
Language–Image Pre-training) provides embeddings for both images and text,
enabling us to compute similarities within and across categories. For each
image, we extracted its CLIP image embedding $E_{i}^{c}$ and the text
embedding $T_{c}$ of its corresponding category name Radford et al. (2021);
Sun et al. (2023). We calculated the cosine similarity between all pairs of
image embeddings within each category to construct a GRAM matrix $G$, where
$G_{i,j}=Cosine(E_{i}^{c},E_{j}^{c})$. Additionally, we computed the image-
text similarity for each image as $Cosine(E_{i}^{c},T_{c})$. Furthermore, we
add basic statistical metrics, such as the percentile of the average-
similarity with respect to other category members of a given instance, the
mean and variance of the average-similarity of the category.

Using these computed features, we applied an XGBoost classifier to label each
image instance. This classifier was trained to distinguish between high-
quality and low-quality instances based on their similarity scores.

We optimized our XGBoost classifier Chen and Guestrin (2016) through 5-fold
cross-validation and grid search to identify the best hyper-parameters. The
classifier achieved an accuracy of 0.8754 on cross-validation, closely
aligning with human judgment, and exhibited Macro-Average Precision, Recall,
and F1-Score of 0.8631, 0.8353, and 0.8460, respectively.

Figure 2: With MMD, we can retrieve harder negative examples and construct
higher-quality test instanecs.

##  4 Test Instances Generation

#### Background Removal and Decontextualization

Connectionist neural networks (including VLMs) are notoriously known for their
tendency of overfitting to spurious correlations present in the training data.
For instance, in our collected data, bulldozers are often seen in construction
scenes laden with materials such as sand, concrete, and bricks. This high co-
occurrence can lead models to rely on these contextual cues rather than truly
understanding and recognizing the bulldozer itself. To mitigate this issue and
ensure that models focus on the objects rather than their typical
environments, we implement a robust background removal process to
decontextualize all candidate objects in our dataset. To achieve effective
background removal, we utilize a state-of-the-art, off-the-shelf background
removal model BRIA-AI (2024).

#### Testing Instances Generation

To assess the performance of Vision-Language Models on our UOUO benchmark, we
generated challenging test instances designed to probe the models’
capabilities beyond common knowledge. Specifically, we employ the CLIP
embeddings combined with the Maximum Mean Discrepancy (MMD) with a Gaussian
RBF kernel Dziugaite et al. (2015) to identify and retrieve hard negative
examples.

Let $\mathbf{x}$ and $\mathbf{y}$ be the sets of CLIP embeddings for two
different object categories, each of shape $(n,d)$, where $n$ is the number of
embeddings and $d$ is the embedding dimension.

The Maximum Mean Discrepancy (MMD) between sets of embeddings $\mathbf{x}$ and
$\mathbf{y}$ is calculated as follows:

$\text{MMD}(\mathbf{x},\mathbf{y})=k(\mathbf{x},\mathbf{x})+k(\mathbf{y},\mathbf{y})-2\cdot
k(\mathbf{x},\mathbf{y})$

where the Gaussian Radial Basis Function (RBF) kernel value
$k(\mathbf{a},\mathbf{b})$ is defined as:

$k(\mathbf{a},\mathbf{b})=\frac{1}{n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}\exp\left(-\frac{1}{2\sigma^{2}}\left\|\mathbf{a}_{i}-\mathbf{b}_{j}\right\|^{2}\right)$

For our calculations, we set $\sigma=10$.

We use the Mosaic Image Augmentation Technique Ge et al. (2021) to generate
testing data in a scalable way. Each testing data point is created from four
images, each background-removed. The four images contain objects of different
categories but share some similar visual properties such as structures,
colors, or textures. The selection of these images is determined by the
Maximum Mean Discrepancy (MMD) distance between the categories they belong to.
The closer the MMD distance, the more similar in features they might appear.
We create an 800x800 canvas large enough to accommodate all four images. Then,
each of the four images is augmented and positioned on the canvas’s top-left,
top-right, bottom-left, or bottom-right. The ground-truth bounding box for the
object grounding is generated from the segmentation mask of background removal
and normalized to be dimension-insensitive, accounting for potential
differences in the VLM’s rescaling process. Figure 2 showcases an exemplar
test instance.

Model | mIoU-mmd | mIoU-rand | acc-mmd | acc-rand
---|---|---|---|---
llava-v1.5-7b | 0.1755 | 0.4117 | 0.4160 | 0.6954
llava-v1.5-13b | 0.2334 | 0.4711 | 0.4351 | 0.7300
llava-v1.6-vicuna-7b | 0.2779 | 0.4783 | 0.4924 | 0.7511
llava-v1.6-vicuna-13b | 0.2761 | 0.4945 | 0.5220 | 0.7773
llava-v1.6-34b | 0.3774 | 0.5504 | 0.5745 | 0.8324
cogvlm-llama3-chat-19b | 0.4905 | 0.6935 | 0.4278 | 0.6024
gemini-1.5-pro | 0.2654 | 0.2682 | 0.6326 | 0.7986
gpt-4-turbo | 0.3396 | 0.3774 | 0.6650 | 0.8970
gpt-4o | 0.3286 | 0.3472 | 0.6779 | 0.8814
Table 1: Mosaic Grounding Performance Metrics

##  5 Experiment

#### Procedures.

Following the aforementioned test instance generation, we test both open
source VLMs that are trained to perform grounding, including: llava-v1.5-7b,
llava-v1.5-13b Liu et al. (2023), llava-v1.6-vicuna-7b, llava-v1.6-vicuna-13b,
llava-v1.6-34b Li et al. (2024), cogvlm-v1.5-vicuna-7b Wang et al. (2023), and
propriety VLMs including: gemini-1.5-pro Team et al. (2024), gpt-4-turbo,
gpt-4o OpenAI et al. (2024).

We test VLMs’ performance on both randomly generated test instances and the
MMD-augmented hard instances. We employ two metrics to quantitfy the
performance: mIoU \- Mean IoU (Intersection over Union), a standard metric for
object segmentation; and  Accuracy , which we prompt the VLM to output one
positions from "top-left, top-right, bottom-left, bottom-right", and directly
evaluate whether the answer matches the ground truth.

#### Observations and Analysis.

We present all experimental results in Table 1. (a) Comparing horizontally
across columns, we observe significant performance drops of smaller-scale
models in both mIoU and Accuracy with the application of MMD-based hard
instance generation. Notably, the performance drops of many of them are around
$30\%$. This provides solid support for our initial hypothesis that smaller-
scale models have some, but insufficient fitness to the long-tail distribution
objects. Furthermore, the drastic performance change showcases MMD’s
effectiveness in generating hard instances and non-robustness of existing
grounding models. (b) Comparing vertically within columns, the central
tendency is that larger scale models (except Genimi which might not be trained
to perform grounding) perform much better than small-scale models in accuracy.
This reveals the concealed gap of knowledge horizon of small- and large- scale
models, which is usually unobservable in benchmarks consist of common objects.
(c) The observation that GPT-4 series can still handle the task remarkably
well (near $90\%$ and $70\%$ on random and MMD settings, respectively)
showcases the task’s solvability, revealing the soundness of our automatically
constructed test instances.

##  6 Conclusion

In our work, we introduced the UOUO benchmark to assess VLMs on objects out of
everyday distributions. Our findings show that while smaller VLMs perform well
on tasks of common objects, they struggle significantly with uncommon objects,
unlike larger models which handle these challenges much better. This
highlights the need to consider long-tail distributions in evaluations. The
systematic data curation, filtering, and hard test instance generation
pipeline for UOUO construction has high extensibility, paving the road of
future research of long-tail distribution objects.

## Limitations

One limitation of our work is the reliance on automated data collection and
cleaning processes, though efficient, may introduce biases or fail to capture
nuanced representations compared to fully manual curation. The UOUO benchmark
also currently emphasizes static images, potentially overlooking the dynamic
and context-dependent nature of object recognition in real-world scenarios.
Future extensions should explore a wider range of uncommon objects across
various fields and consider the inclusion of video or sequential data to
better reflect real-world applications. Addressing these limitations will
enhance the comprehensiveness and applicability of the UOUO benchmark.

## References

  * Agrawal et al. (2016) Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. 2016.  Vqa: Visual question answering.  _Preprint_ , arXiv:1505.00468.
  * Agrawal et al. (2019) Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019.  nocaps: novel object captioning at scale.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 8948–8957.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  _Preprint_ , arXiv:2204.14198.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  _Preprint_ , arXiv:2308.12966.
  * BRIA-AI (2024) BRIA-AI. 2024.  Bria background removal v1.4 model card.  https://huggingface.co/briaai/RMBG-1.4.  Accessed: 2024-06-13.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16. ACM.
  * Clauset et al. (2009) Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. 2009.  Power-law distributions in empirical data.  _SIAM review_ , 51(4):661–703.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Dziugaite et al. (2015) Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. 2015.  Training generative neural networks via maximum mean discrepancy optimization.  _arXiv preprint arXiv:1505.03906_.
  * Fu et al. (2024) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _Preprint_ , arXiv:2306.13394.
  * Ge et al. (2021) Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021.  Yolox: Exceeding yolo series in 2021.  _Preprint_ , arXiv:2107.08430.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  _Preprint_ , arXiv:1612.00837.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  _Preprint_ , arXiv:1902.09506.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013.  3d object representations for fine-grained categorization.  In _2013 IEEE International Conference on Computer Vision Workshops_ , pages 554–561.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision_ , 128(7):1956–1981.
  * Li et al. (2024) Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. 2024.  Llava-next: Stronger llms supercharge multimodal capabilities in the wild.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _Preprint_ , arXiv:2301.12597.
  * Lin et al. (2015) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015.  Microsoft coco: Common objects in context.  _Preprint_ , arXiv:1405.0312.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023.  Improved baselines with visual instruction tuning.
  * Liu et al. (2024) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024.  Mmbench: Is your multi-modal model an all-around player? _Preprint_ , arXiv:2307.06281.
  * Maji et al. (2013) S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. 2013.  Fine-grained visual classification of aircraft.  Technical report.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024.  Gpt-4 technical report.  _Preprint_ , arXiv:2303.08774.
  * Piantadosi (2014) Steven T. Piantadosi. 2014.  Zipf’s word frequency law in natural language: a critical review and future directions.  _Psychonomic Bulletin & Review_, 21(5):1112–1130.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  _Preprint_ , arXiv:2103.00020.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015.  Imagenet large scale visual recognition challenge.  _Preprint_ , arXiv:1409.0575.
  * Rückert et al. (2024) Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S. Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, Henning Müller, Peter A. Horn, Felix Nensa, and Christoph M. Friedrich. 2024.  Rocov2: Radiology objects in context version 2, an updated multimodal image dataset.  _Preprint_ , arXiv:2405.10004.
  * Saleh et al. (2020) Alzayat Saleh, Issam H Laradji, Dmitry A Konovalov, Michael Bradley, David Vazquez, and Marcus Sheaves. 2020.  A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis.  _Scientific Reports_ , 10(1):14671.
  * Sun et al. (2023) Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023.  Eva-clip: Improved training techniques for clip at scale.  _Preprint_ , arXiv:2303.15389.
  * Team et al. (2024) Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, Juliette Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Yingjie Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontanon, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Antoine He, Skye Giordano, Lakshman Yagati, Jean-Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yunhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, Anja Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Rohan Jain, Vinay Ramasesh, Anton Briukhov, Da-Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matthew Wiethoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Bortsova, Yonghui Wu, Paul Voigtlaender, Tara Sainath, Shane Gu, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney, Gabriela Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Petrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko Iinuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Justin Chiu, Rory Blevins, Zach Gleicher, Adrià Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, Sébastien M. R. Arnold, Lisa Lee, James Lee-Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuo yiin Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Josh Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aishwarya Kamath, Arnar Mar Hrafnkelsson, Le Hou, Ian Mackinnon, Alexandre Frechette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Anmol Gulati, Sébastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard Powell, Stephan Lee, Kiran Vodrahalli, Samer Hassan, Diana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Aedan Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravi Addanki, Tianhe Yu, Wojciech Stokowiec, Mina Khan, Justin Gilmer, Jaehoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Lučić, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjösund, Elena Buchatskaya, Elspeth White, Natalie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Zeyncep Cankara, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Lora Aroyo, Zhufeng Pan, Zachary Nado, Jakub Sygnowski, Stephanie Winkler, Dian Yu, Mohammad Saleh, Loren Maggiore, Yamini Bansal, Xavier Garcia, Mehran Kazemi, Piyush Patil, Ishita Dasgupta, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Qingze Wang, Chung-Cheng Chiu, Zoe Ashwood, Khuslen Baatarsukh, Sina Samangooei, Raphaël Lopez Kaufman, Fred Alcober, Axel Stjerngren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Chris Welty, Dawn Bloxwich, Charlie Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerxes Dotiwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng, Krishna Haridasan, Gabe Barth-Maron, Craig Swanson, Dominika Rogozińska, Alek Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija Ilić, Yao Zhao, Adam Iwanicki, Alejandro Lince, Alexander Chen, Christina Lyu, Carl Lebsack, Jordan Griffith, Meenu Gaba, Paramjit Sandhu, Phil Chen, Anna Koop, Ravi Rajwar, Soheil Hassas Yeganeh, Solomon Chang, Rui Zhu, Soroush Radpour, Elnaz Davoodi, Ving Ian Lei, Yang Xu, Daniel Toyama, Constant Segal, Martin Wicke, Hanzhao Lin, Anna Bulanova, Adrià Puigdomènech Badia, Nemanja Rakićević, Pablo Sprechmann, Angelos Filos, Shaobo Hou, Víctor Campos, Nora Kassner, Devendra Sachan, Meire Fortunato, Chimezie Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadegh Jazayeri, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingshuk Dasgupta, Shourya Sarcar, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejc Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anaïs White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonam Goenka, Brennan Saeta, Ying Xu, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Çağlar Ünlü, David Reid, Zora Tung, Daniel Finchelstein, Ravin Kumar, Andre Elisseeff, Jin Huang, Ming Zhang, Ricardo Aguilar, Mai Giménez, Jiawei Xia, Olivier Dousse, Willi Gierke, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimoto, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nick Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Jeff Stanway, Mukund Sundararajan, Victor Ungureanu, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloe Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, Ian Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Ananth Agarwal, Mia Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashisht, Rebeca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liechty, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnukov, Matt Miecnikowski, Jiri Simsa, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimitriev, Mohsen Jafari, Remi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishek Chakladar, Annie Louis, Carl Crous, Hardie Cate, Dessie Petrova, Michael Quinn, Denese Owusu-Afriyie, Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanan, Miteyan Patel, Charbel Kaed, Jing Li, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Põder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Alanna Walton, Alicia Parrish, Mark Epstein, Sara McCarthy, Slav Petrov, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _Preprint_ , arXiv:2403.05530.
  * Vaswani et al. (2023) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023.  Attention is all you need.  _Preprint_ , arXiv:1706.03762.
  * Wah et al. (2011) Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011.  The caltech-ucsd birds-200-2011 dataset.
  * Wang et al. (2023) Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023.  Cogvlm: Visual expert for pretrained language models.  _Preprint_ , arXiv:2311.03079.
  * Wolpert and Macready (1997) David H Wolpert and William G Macready. 1997.  No free lunch theorems for optimization.  _IEEE transactions on evolutionary computation_ , 1(1):67–82.
  * Xu et al. (2023) Peng Xu, Xiatian Zhu, and David A. Clifton. 2023.  Multimodal learning with transformers: A survey.  _Preprint_ , arXiv:2206.06488.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78.
  * Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016.  Modeling context in referring expressions.  _Preprint_ , arXiv:1608.00272.
  * Yu et al. (2023a) Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2023a.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  _arXiv preprint arXiv:2312.00849_.
  * Yu et al. (2023b) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023b.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _Preprint_ , arXiv:2308.02490.

##  Appendix A Appendix

Important statistics of UOUO are listed as follows:

  * •

Number of categories:

    * –

Filtered data directory: 25,864

    * –

Original data directory: 27,926

    * –

Percentage of categories kept: 92.6%

  * •

Total number of images:

    * –

Filtered dataset: 678,535

    * –

Original dataset: 956,167

    * –

Percentage of images kept: 71.0%

  * •

Images per category stats:

    * –

Filtered dataset:

      * *

Average: 26.235

      * *

Minimum: 5

      * *

Maximum: 48

    * –

Original dataset:

      * *

Average: 26.235

      * *

Minimum: 5

      * *

Maximum: 48

  * •

Average percentage of images kept in each category: 76.0%

Figure 3: Wikipedia Industry List

Randomly sampled 100 categories:

2D pantograph |  AC Recharge Kit |  Adhesive scale |  Aluminum dross processing machine
---|---|---|---
Artificial insemination gun |  Ballistic clipboard |  Ballot Box (for collecting anonymous feedback) |  Banjo rim lathe
Bingo balls |  Broodstock tanks |  Broom |  Burnishing Stone
Cable Retention Sleeve |  Carding Machine |  Cattle Curtain |  Cell Model
Climbing rope |  Coal centrifuge |  Coffee roaster |  Cold Storage Backpack
Compressor (hardware) |  Cooling Incubator |  Copy Stand |  Culture trays
Dehooking tool |  Deposit Slip Printer |  Disc golf basket welder |  Disc repair kit
Display Turntables |  Distillation column |  Electronic rate board |  Evaporating Dish
Extrusion laminator |  Fiber disc |  Fishing rod holders |  Flange spreader
Flower press |  Foundation crack ruler |  Fume Extraction Hood |  Goniophotometer
Graduated cylinders |  Granule Filler |  Inductively Coupled Plasma (ICP) Spectrometer |  Irrigation pipelayer
Lacquer polishing brush |  Leachate Collection Pipe |  Live Feed Incubator |  Longlines and ropes
Martingale |  Metal scribe |  Mobile manufacturing unit (MMU) |  Mushroom grow tent
Music on hold player |  Network Firewall Hardware |  Offshore aquaculture cage |  Ore skip
Oscillating shaker |  Oxygen concentrators |  Packing Gauge |  Pellets coating system
Pellicle Formation Tool |  Pillory |  Pin beater |  Pointer stick
Portable battery booster |  Pressure vessels |  Print Quality Inspection Scope |  Pulling post
Purging compound dispenser |  Queue stanchion |  Quick release hook |  Roll Coating Paint Line
Rope pump |  Rotary drum bauxite washer |  Rotary impeller feeder |  Sand filter
Scale Breaker |  Schlenk flask |  Security drone |  Security token device
Shear Line |  Shock Absorber |  Sign language interpreter gloves |  Slab Tongs
Slush ice machines |  Soap scum remover |  Spin Welder |  Spoke cutting machine
Spot meter |  Springform pan |  Tabbing shears for composite test specimens |  Texture sprayer
Tower Climbing Harness |  Violin varnish brush |  Vixen Plate |  Wall Hooks for Art
Waste basket |  Water jet cutter for stone |  Whalebone Scraper |  Wire Mesh Cable Trays
Table 2: List of 100 Randomly Sampled Categories