11institutetext: Missouri S & T, Missouri MO 65401, USA,
11email<EMAIL_ADDRESS>22institutetext: Yale University, Yale, CO, USA
22email: {siddique9171, yclam} @gmail.com 33institutetext: Singapore Institute
for Clinical Sciences
33email<EMAIL_ADDRESS>44institutetext: Nanyang Technical
University
44email<EMAIL_ADDRESS><EMAIL_ADDRESS>55institutetext: National
University of Singapore
55email<EMAIL_ADDRESS>66institutetext: New York University, NY, USA
66email<EMAIL_ADDRESS>77institutetext: Jivox, USA
77email<EMAIL_ADDRESS>
# How is Fatherhood Framed Online in Singapore?

Tran Hien Van 44 Abhay Goyal 11 Muhammad Siddique 22 Lam Yin Cheung 22 Nimay
Parekh 77 Jonathan Y Huang 33 Keri McCrickerd 33 Edson C Tandoc Jr 44 Gerard
Chung 55 Navin Kumar 66

###### Abstract

The proliferation of discussion about fatherhood in Singapore attests to its
significance, indicating the need for an exploration of how fatherhood is
framed, aiding policy-making around fatherhood in Singapore. Sound and
holistic policy around fatherhood in Singapore may reduce stigma and
apprehension around being a parent, critical to improving the nation’s
flagging birth rate. We analyzed 15,705 articles and 56,221 posts to study how
fatherhood is framed in Singapore across a range of online platforms (news
outlets, parenting forums, Twitter). We used NLP techniques to understand
these differences. While fatherhood was framed in a range of ways on the
Singaporean online environment, it did not seem that fathers were framed as
central to the Singaporean family unit. A strength of our work is how the
different techniques we have applied validate each other.
Keywords: fatherhood, singapore, social media

##  1 Introduction

Fatherhood is now an unprecedentedly visible cultural phenomenon in Singapore.
This increased attention is related to the inaugural nationwide fatherhood
movement, Dads for Life, the continual development of parenting magazines and
the recent emergence of fatherhood blogs within the Singapore internet sphere.
In recent times, various fatherhood-related initiatives in Singapore have
collaborated with government agencies, business corporations, and community
organizations on initiatives to create awareness of the importance of the
father’s role, develop commitment to good fathering, and encourage fathers to
spend time with their children. In Singapore, the introduction of paternity
leave and encouragement for fathers to play a bigger role in childcare and
child-raising suggest that the government is sympathetic to the pursuit of
gender equality. However, there is a gap between the perception of the
importance of fathers and the actual involvement of fathers in their
children’s lives. In addition, the role of fathers continues to be recognized
primarily as that of a breadwinner. Yet fathers want to do more and experience
parenthood as a very fulfilling experience, to which they are highly committed
[3]. The proliferation of discussion about fatherhood in Singapore attests to
its significance as a commercial, ideological, and cultural subject,
indicating the need for an exploration of how fatherhood is framed, aiding
policy-making around fatherhood in Singapore. While there has been research
around how fatherhood is framed in the Singapore context, there is limited
analysis of how fatherhood is framed on social media, news outlets, or online
forums. Such platforms are where opinions or news on fatherhood are forwarded,
people get parenting information, or get quick answers to fatherhood
questions. Studying how fatherhood is framed in the online Singaporean context
is central to crafting progressive and effective policy around parenting in
Singapore, as well as managing the media landscape. Sound and holistic policy
around fatherhood in Singapore may reduce stigma and apprehension around being
a parent, critical to improving the nation’s flagging birth rate. Policies
developed in Singapore around fatherhood may then be implemented in nearby
East Asian countries, which have similarly low birth rates, to mitigate a
rapidly aging society and a shrinking taxpayer base. In this paper, we
demonstrate how fatherhood in Singapore is framed on multiple online platforms
(news outlets, parenting forums, Twitter). Our main research question (RQ) is
as follows: How is fatherhood in Singapore framed on various online platforms?
Our findings suggested that while fatherhood was framed in a multiplicity of
forms online, it did not seem that fathers were core to the family.

##  2 Related Work

Fatherhood Framing Online Work on fatherhood in Singapore is limited. Recent
work proposed the concept of Confucian masculinity to explain how the
depiction of active fatherhood reinforced the ubiquitous normal family that
upholds patriarchal ideology and perpetuates patriarchal power, obscuring the
contradictions of class, race, and sexuality that exist in Singapore [3].
Other work examined the fatherhood discourses in new dad ads; feature articles
from Today’s Parents, a parenting magazine; articles from Life Dads, a
government electronic newsletter on fatherhood; and blog entries from three
fatherhood blogs [4]. The study employed critical discourse analysis, and
proposed a Hegemonic Fatherhood Discourse Schema to postulate that the new
father/man and traditional father/man ideology is the hegemonic fatherhood in
Singapore, ultimately serving the interests of the Singapore state. While past
work detailed framing around fatherhood in Singapore, previous research did
not compare framing across online platforms, or provide an overview of
fatherhood framing to develop policy or informational tools. While there was
limited fatherhood research in the Singapore context, there was relatively
more research on fatherhood framing online in other contexts. For example,
recent work [5] used discussion threads from two Web-based parenting
communities, r/Daddit and r/PreDaddit from Reddit. Results demonstrated that
men used web-based communities to share the joys and challenges of the
fatherhood experience.

##  3 Data and Method

Data We first selected three content experts who had published at least ten
peer-reviewed articles in the last three years around fatherhood. We ensured
the content experts were either from Singapore or conducted research on
fatherhood/parenthood in Singapore. Given the wide disciplinary focus of
fatherhood research, we sought to select a range of experts across
disciplines. We recruited one expert from each of these disciplines: Public
policy, social work, computational social science. Selecting experts from a
range of fields allows results to be contextualized to fields where fatherhood
research is concentrated, allowing for findings to be drawn on by stakeholders
in public policy, social work, and computational social science. The context
experts separately developed lists of online platforms most relevant to
fatherhood in Singapore. Each expert developed a list of ten platforms
independently, and we selected only platforms common to all three experts’
lists. For each online platform, experts also provided up to 10 examples,
where applicable, of websites, or forums, and we selected examples common to
all experts’ lists. The final list of platforms is as follows: Singapore news
outlets (Straits Times, Channel NewsAsia, TODAYonline), parenting forums
(singaporemotherhood.com, singaporeparents.com.sg/forum,
forums.hardwarezone.com.sg/threads/welcome-to-hwzs-parenting-kids-early-
learning-forum.5684416, mummysg.com/forums), Twitter (filtering only posts
related to Singapore). Examples of platforms not selected: Facebook,
Instagram, Reddit, LinkedIn. We were not able to collect Facebook and
Instagram data as there was limited support for CrowdTangle, the main mode of
Facebook/Instagram data collection. Similarly, the pushshift.io Reddit API had
limited support and Reddit data collected was incomplete. LinkedIn had limited
fatherhood posts and posts were mostly centered on non-family content. To
capture fatherhood-related text on these platforms, we used queries based on a
related systematic review e.g., father* OR dad* OR patern* OR paternal OR
paternity OR stepdad* OR stepfather* OR step-dad* OR Step-father* OR papa. We
used only English-language keywords as most of discussion in the Singapore
internet environment is in English. English is also the major language of
communication in Singapore. For forums, we used automated scraping techniques
(Beautiful Soup) to obtain forum posts from 2010 to 2023, with the same set of
keywords. We ran a search for querying the keywords in the title of the forum
post or replies to the forum post. We collected all posts that contained these
keywords within the forum posts and replies. Regarding Twitter, we used the
Twitter API and the indicated keywords to collect tweets from 2011 to 2023.
Finally, for news articles, we used Nexis to obtain news archives from 1992 to
2023. To prepare the data for analysis, English stop words such as the, a, an
were removed, along with abbreviations, and terms were stemmed using Porter’s
stemming algorithm. Stemming converts words with the same stem or root (e.g.,
innovative and innovator) to a single word type (e.g., innovate). We organized
data into four streams for analysis: Twitter (tweets), news (news articles),
forums (forum posts).

Sentiment Sentiment analysis can aid us in comprehending how sentiment around
fatherhood is expressed in the online arena. As an example, forums may be more
likely to have lower sentiment compared to news. DistilBERT was used for
sentiment analysis. DistilBERT was used separately on data from each platform.
The model assigns sentiment based on each article or post. Sentiment is from a
-1 to 1 scale, where values $<$0 are negative sentiment, $>$0 are positive
sentiment, and close to 0 are neutral. To stay within the admitted input size
of the model, the text length (title + body text) was clipped to to 512
tokens.

Emotion Recognition Emotion recognition can help us understand how emotions
are expressed across various platforms, indicating differences in how
fatherhood is framed in Singapore. For example, forums may be more likely to
contain anger compared to news. We used DistilBERT for emotion recognition.
The model was applied separately on data from each platform. The model assigns
emotions (anger, fear, joy, love, sadness, surprise) based on each article or
post. To stay within the admitted input size of the model, we clipped the
length of the text (title + body text) to 512 tokens.

Platform | Data collected (e.g., N of posts, articles)
---|---
News outlets | 15,705 articles, 9,811,513 words
Twitter | 54,283 tweets, 900,939 words
Parenting forums | 969 threads, 425,966 words
Table 1: Data collected across online platforms

We provided an overview of the data in Table 1. Two reviewers independently
examined 10% of the articles or posts within each dataset to confirm salience
with our research question. The reviewers then discussed their findings and
highlighted items deemed relevant across both lists. We noted the following
relevance proportions: News outlets (82%), Twitter (90%), Parenting forums
(78%).

##  4 Results

Overview We first explored sample posts across platforms. News outlets
generally mentioned fatherhood in the context of providing demographic data
about interviewees, with excerpts such as So the 40-year-old eye specialist
and father of three had to wrap up his work at the hospital quickly, or when
interviewees were referring to their fathers with no specific reference to
fatherhood e.g., Mr Lee, whose father founded the clan association, rents out
its third floor to a small media firm. Broadly, news outlets did not seem to
focus on the experience of fatherhood, with the bulk of articles mentioning
fathers as a demographic indicator. Twitter posts focused on people recounting
incidents, often humorous or heart-warming, with their fathers e.g., My dad
was telling me something serious and he hit his leg against the table and I
burst out laughing so he had no choice but to laugh, Dad brought back homemade
fresh horfun (noodles) from the temple. It’s delicious. Twitter seemed to have
a greater focus on fathers playing a core function in the Singapore family
unit. Posts from forums were very diverse topically. Several posts were about
hiring a helper for a young child: My husband is totally against the idea of
employing a helper, as he does not like a stranger living with us; I am a
father of a newborn baby girl. I recently engaged a confinement lady by the
name of Auntie Judy. Such posts suggest the significant role domestic helpers
play in the Singaporean family, and how a portion of a father’s role is
perhaps to oversee the hiring of the domestic helper. Other posts were about
suspected infidelity e.g., So my Wife of 2 years has been cheating on me with
another male colleague, perhaps indicative of the strain parenting is related
to within some Singaporean families.

(a)

(b)

(c)

Figure 1: Word cloud visualizations for news (LABEL:fig:News), Twitter (3(b)),
forums (3(c)) based on keywords relevant to fatherhood in Singapore

We then provided word clouds in Figure 1 as an overview of the data. Across
all datasets, words such as time, work, now were prominent, perhaps indicative
of how work and likely limited time are central to fatherhood in Singapore.
Most common trigrams for news articles centered on leaders of Singapore, who
were father and son: Lee Kwan Yew and Lee Hsien Loong. This may indicate that
the mainstream news media discussion around fatherhood had little to do with
fathers’ role in a family, but simply around familial relationships within
major news stories. In 1992 - 2003, common trigrams in the news were engineer
success story and pressure parent counting. From 2004 - 2019, common trigrams
were two baby boy, first new baby, and first time parent. From 2020 - 2022,
common trigrams were generation grit family, and grit family love. Broadly,
news trigrams may detail how the initial focus was on children bringing pride
and wealth to their families, with a transition toward celebrating new births.
In more recent years, forums tended to focus on how the family unit could
overcome struggles. The most common trigrams in Twitter focused on celebrating
fathers through specific events such as Father’s Day and birthdays: happy
father’s day, happy birthday daddy. Such phrases indicated that Twitter may be
used to celebrate fathers, but only in relation to pre-defined events, instead
of fathers being celebrated for time put toward caregiving etc. Common
trigrams in 2011 - 2020 were love u dad, dad love love. 2021 onwards, popular
trigrams were feel fulfilling husband, and last nite daddy. Twitter data
demonstrated a shift from declaring love for one’s father, to fathers
indicating how they were fulfilled in their role. Unlike other datasets, there
appears to be a shift towards a more active form of fatherhood in Singapore,
where fathers describe pride in their role. Trigrams in forums centered on
perceived marital infidelity, such as wife unfaithful husband, and assisted
reproductive technologies, such as ivf mommy toben, and cousin egg donor.
Forums seemed to be platforms where people sought support around spousal
infidelity and assisted reproductive technologies, rather than discuss
fathers’ role in the family unit. The most common trigrams in forums changed
over time, with phrases such as gave birth daughter, and first time dad in
2010 - 2019, but with phrases such as happen file divorce, and judged urged
divorcing in 2020. In 2021, common trigrams were conceiving single women,
while in 2022, trigrams such as crave physical intimacy, and physicial
intimacy normal were popular. Forums, while initially around celebrating
birth, may have become places where people sought information around divorce,
assisted reproductive technologies, and physical intimacy. Broadly,
descriptive data indicated shifting framing around fatherhood, but a limited
focus on fathers as core to the Singapore family.

Sentiment

| News | Twitter | Forums
---|---|---|---
Positive | 53.7% | 57.0% | 27.2 %
Negative | 43.8% | 33.8% | 65.9 %
Neutral | 2.5% | 9.1% | 6.9 %
Table 2: Sentiment analysis breakdown for various platforms.

(a) News

(b) Twitter

(c) Forums

Figure 2: Sentiment analysis over time for various platforms.

We presented sentiment analysis results across each platform in Table 2. News
and Twitter had higher proportions of positive sentiment (53.7% and 57.0%
respectively) compared to forums (27.2%). Forums had the highest proportion of
negative sentiment (65.9%), compared to news and Twitter (43.8% and 33.8%
respectively). We then presented sentiment analysis results over time for each
platform in Figure 2. News data exhibited several fluctuations but had the
greatest rise in positive sentiment post-2009. The nationwide fatherhood
movement, Dads for Life, started in 2009, may explain the increase in positive
sentiment. Examples of news article content with positive sentiment were as
follows: A group of prominent figures from various organisations and
businesses have banded together to start up the Fathers Action Network. The
network aims to kick-start a movement called Dads for Life to get fathers more
involved with their families, especially in their childrens’ lives. This
follows a fatherhood perception survey conducted in April and May this year by
a Ministry. Most felt that being a father and raising children is one of the
most fulfilling experiences a man can have.; Work is work and family is
family. Our ultimate goal is still our family. Work is just a means to get the
money so we should be very clear about it. And that is the sort of spirit that
the Dads for Life movement wants to inspire. After 2017, positive sentiment
declined over time, and was overtaken by negative sentiment. Forums had
broadly negative sentiment 2015 onward, reaching a peak in 2017, followed by a
steady decline. Twitter exhibited mostly positive sentiment 2013 onward with a
steady decline after. We suggest that the high proportion of positive
sentiment in the news may be related to governmental initiatives and the high
proportion of negative sentiment in forums may be related to a more frank
discussion of the stresses of parenting.

Emotion Recognition

| News | Twitter | Forums
---|---|---|---
Anger | 32.7% | 28.5% | 25.5
Fear | 18.8% | 4.1% | 6.3
Joy | 61.3% | 56.6% | 44.2
Love | 34.2% | 2.4% | 4.1
Sadness | 2.0% | 7.6% | 18.9
Surprise | 11.4% | 0.8% | 1.0
Table 3: Emotion recognition breakdown for various platforms.

(a) News

(b) Twitter

(c) Forums

Figure 3: Emotion recognition over time for various platforms.

We presented emotion recognition results across each platform in Table 3. News
had the highest proportion of joyous (61.3%) and loving (34.2%) posts, perhaps
reflecting governmental initiatives around fatherhood. While Twitter and
forums had similar levels of joyous posts (56.6% and 44.2% respectively), they
were still not as high as news. Similarly, loving posts on Twitter and forums
(2.4% and 4.1% respectively) were far lower than news outlets. We suggest that
the emotion in the news reflects pro-fatherhood governmental initiatives, but
these do not always filter successfully to other media. We then presented
emotion recognition results over time for each platform in Figure 3. News data
exhibited several fluctuations but had the steepest rise post-2009. Dads for
Life, started in 2009, may explain the uptick in news articles, especially
around joy. Examples of news article content that were coded as joy: It’s a
happy Father’s Day for SAFRA, as it is set to receive funds from the ”Dads for
Life” movement to pump up father-friendly activities for its members over the
next two years.; He will be running alongside his daughter in the Dads For
Life 800m Father and Child Challenge, a new category in the annual SAFRA
Singapore Bay Run and Army Half-Marathon. Mr Shariff, who was born without
part of his left leg, said: I signed us up because I want to show her how
running can make her happy. Both Twitter and forum posts saw a sudden spike
post-2013 onward, mostly around joy. We suggest that the shift in emotion may
be due to a delayed reaction to Dads for Life. Broadly, we forward that the
2009 Dads for Life movement and other similar policies may have catalyzed
emotional reactions around fatherhood in the Singapore online arena. However,
the rises in emotion were not sustained and seemed to decline by 2023, perhaps
indicative that new policy levers may need to be rolled out.

##  5 Discussion

Our RQ was to explore how fatherhood in Singapore is framed on various online
platforms. A strength of our work is how the different techniques we applied
validate each other as well as reveal differences across platforms. While
fatherhood was framed in a range of ways on the Singaporean online
environment, it did not seem that fathers were framed as central to the
Singaporean family unit. Results also indicated that governmental initiatives
may have some effect on altering the framing of fatherhood, but are not
lasting in effect. The concordance in our results suggests the veracity of our
findings and we hope that results can add to research and policy around
fatherhood in Singapore. Our evidence adds to previous research, where we
provided data on how governmental initiatives may initially buttress framing
around fatherhood, but needs to be sustained to provide broad and lasting
support for fathers. Key to how fatherhood is framed in Singapore is the
inclusion of fathers’ viewpoints when writing news articles on fatherhood.
Where possible, fathers themselves should be consulted on articles about
fatherhood. For example, a panel staffed by fathers can comment on fatherhood-
related online news articles, providing suggestions on how articles can more
accurately represent fathers’ concerns [1, 2]. Our findings relied on the
validity of data collected with our search terms. We used a range of
established techniques to search for all articles/posts relevant to
fatherhood, and our data contained text aligned with how fatherhood is framed.
We were thus confident in the comprehensiveness of our data. We only used
English-language text but will include other languages in future work. Given
the token limits for the emotion recognition technique, we were not able to
use emotion recognition for the entirety of longer news articles. We note that
the recall of the search string was not tested. We note that our data may not
be generalizable to how fatherhood is framed globally. Our goal was not to
identify who was doing the framing around fatherhood e.g., family members or
government. Future studies will seek to identify which stakeholders were
likely involved in the framing.

## References

  * [1] Chen, K., Babaeianjelodar, M., Shi, Y., Janmohamed, K., Sarkar, R., Weber, I., Davidson, T., De Choudhury, M., Yadav, S., Khudabukhsh, A., et al.: Partisan us news media representations of syrian refugees. arXiv preprint arXiv:2206.09024 (2022)
  * [2] Kumar, N., Chen, K., Shi, Y., Altice, F.L.: Online platforms’ framing around vaping. Drug Testing and Analysis (2022)
  * [3] Lim, A.: Confucian masculinity: State advocacy of active fatherhood in singapore. Men and Masculinities 24(1), 46–63 (2021)
  * [4] See, E.K.: “Most Clearly, Dads are Best!”–a critical discourse analysis of fatherhood discourses in Singapore. Ph.D. thesis (2017)
  * [5] Teague, S.J., Shatte, A.B.: Exploring the transition to fatherhood: feasibility study using social media and machine learning. JMIR pediatrics and parenting 1(2), e12371 (2018)


# Momentum-SAM: Sharpness Aware Minimization without Computational Overhead

Marlon Becker Frederick Altrock Benjamin Risse
Department for Computer Science & Institute for Geoinformatics
University of Muenster, Germany
<EMAIL_ADDRESS>

###### Abstract

The recently proposed optimization algorithm for deep neural networks
Sharpness Aware Minimization (SAM) suggests perturbing parameters before
gradient calculation by a gradient ascent step to guide the optimization into
parameter space regions of flat loss. While significant generalization
improvements and thus reduction of overfitting could be demonstrated, the
computational costs are doubled due to the additionally needed gradient
calculation, making SAM unfeasible in case of limited computationally
capacities. Motivated by Nesterov Accelerated Gradient (NAG) we propose
Momentum-SAM (MSAM), which perturbs parameters in the direction of the
accumulated momentum vector to achieve low sharpness without significant
computational overhead or memory demands over SGD or Adam. We evaluate MSAM in
detail and reveal insights on separable mechanisms of NAG, SAM and MSAM
regarding training optimization and generalization. Code is available at
https://github.com/MarlonBecker/MSAM.

##  1 Introduction

While artificial neural networks (ANNs) are typically trained by Empirical
Risk Minimization (ERM), i.e., the minimization of a predefined loss function
on a finite set of training data, the actual purpose is to generalize over
this dataset and fit the model to the underlying data distribution. Due to
heavy overparameterization of state-of-the-art ANN models (Nakkiran et al.,
2021), the risk of assimilating the training data increases. As a consequence,
a fundamental challenge in designing network architectures and training
procedures is to ensure the objective of ERM to be an adequate proxy for
learning the underlying data distribution.
One strategy to tackle this problem is to exploit the properties of the loss
landscape of the parameter space on the training data. A strong link between
the sharpness in this loss landscape and the models generalization capability
has been proposed by Hochreiter & Schmidhuber (1994) and further analyzed in
the work of Keskar et al. (2017). Following these works, Foret et al. (2021)
proposed an algorithm to explicitly reduce the sharpness of loss minima and
thereby improve the generalization performance, named Sharpness Aware
Minimization (SAM). Built on top of gradient based optimizers such as SGD or
Adam (Kingma & Ba, 2015), SAM searches for a loss maximum in a limited
parameter vicinity for each optimization step and calculates the loss gradient
at this ascended parameter position. To construct a computationally feasible
training algorithm, SAM approximates the loss landscape linearly so that the
maximization is reduced to a single gradient ascent step. Moreover, this step
is performed on a single batch rather than the full training set.
Unfortunately, the ascent step requires an additional forward and backward
pass and therefore doubles the computational time, limiting the applications
of SAM severely. Even though the linear approximation of the loss landscape
poses a vast simplification and Foret et al. (2021) showed that searching for
the maximum with multiple iterations of projected gradient ascent steps indeed
yields higher maxima, these maxima, however, do not improve the
generalization, suggesting that finding the actual maximum in the local
vicinity is not pivotal. Instead, it appears to be sufficient to alter the
parameters to find an elevated point and perform the gradient calculation from
there. Following this reasoning, the ascent step can be understood as a
temporary parameter perturbation, revealing strong resemblance of the SAM
algorithm to extragradient methods (Korpelevich, 1976) and Nesterov
Accelerated Gradient (Nesterov, 1983; Sutskever et al., 2013) which both
calculate gradients at perturbed positions and were also discussed previously
in the context of sharpness and generalization (Lin et al., 2020a; Wen et al.,
2018).
Commonly, measures to address generalization issues are applied between the
data distribution and the full training dataset (Keskar et al., 2017; Li et
al., 2018). Since the generalization error is caused by the limited sample
size, further reducing the regarded sample size to single batches might impose
additional deteriorating effects. Thus motivated, we reconsider the effect of
momentum in batch-based gradient optimization algorithms as follows. The
momentum vector not only represents a trace over iterations in the loss
landscape and therefore accumulates the gradients at past parameter positions,
but also builds an exponential moving average over gradients of successive
batches. Hence, the resultant momentum vector can also be seen as an
approximation of the gradient of the loss on a larger subset - in the limiting
case on the full training dataset.
Building on these observations and the theoretical framework of SAM, which
assumes using the entire dataset for sharpness estimations, we present
Momentum-SAM (MSAM). MSAM aims to minimize the global sharpness without
imposing additional forward and backward pass computations by using the
momentum direction as an approximated, yet less stochastic, direction for
sharpness computations. In summary, our contribution is as follows:

  * •

We propose Momentum-SAM (MSAM), an algorithm to minimize training loss
sharpness without computational overhead over base optimizers such as SGD or
Adam.

  * •

The simplicity of our algorithm and the reduced computational costs enable the
usage of sharpness-aware minimization for a variety of different applications
without severely compromising the generalization capabilities and performance
improvements of SAM.

  * •

We discuss similarities and differences between MSAM and Nesterov Accelerated
Gradient (NAG) and reveal novel perspectives on SAM, MSAM, as well as on NAG.

  * •

We validate MSAM on multiple image classification benchmarks and compare MSAM
against related sharpness-aware approaches.

###  1.1 Related Work

Correlations between loss sharpness, generalization, and overfitting were
studied extensively (Hochreiter & Schmidhuber, 1994; Keskar et al., 2017; Lin
et al., 2020b; Yao et al., 2018; Li et al., 2018; Liu et al., 2020; Damian et
al., 2021), all linking flatter minima to better generalization, while Dinh et
al. (2017) showed that sharp minima can generalize too. While the above-
mentioned works focus on analysing loss sharpness, algorithms to explicitly
target sharpness reduction were suggested by Zheng et al. (2021); Wu et al.
(2020); Chaudhari et al. (2017) with SAM (Foret et al., 2021) being most
prevalent.
SAM relies on computing gradients at parameters distinct from the current
iterations position. This resembles extragradient methods (Korpelevich, 1976)
like Optimistic Mirror Descent (OMD) (Juditsky et al., 2011) or Nesterov
Accelerated Gradient (NAG) (Nesterov, 1983; Sutskever et al., 2013) which were
also applied to Deep Learning, either based on perturbations by last
iterations gradients (Daskalakis et al., 2018; Lin et al., 2020a) or random
perturbations (Wen et al., 2018).
Adaptive-SAM (ASAM) (Kwon et al., 2021) accommodates SAM by scaling the
perturbations relative to the weights norms to take scale invariance between
layers into account, resulting in a significant performance improvement over
SAM. Furthermore, Kim et al. (2022) refine ASAM by considering Fisher
information geometry of the parameter space. Also seeking to improve SAM, GSAM
(Zhuang et al., 2022) posit that minimizing the perturbed loss might not
guarantee a flatter loss and suggest using a combination of the SAM gradient
and the SGD gradients component orthogonal to the SAM gradient for the weight
updates.
Unlike the aforementioned methods, several algorithms were proposed to reduce
SAMs runtime, mostly sharing the idea of reducing the number of additional
forward/backward passes, in contrast to our approach which relies on finding
more efficient parameter perturbations. For example, Jiang et al. (2023) are
evaluating in each iteration if a perturbation calculation is to be performed.
LookSAM Liu et al. (2022) updates perturbations only each $k$-th iterations
and applies perturbation components orthogonal to SGD gradients in iterations
in between. Mi et al. (2022) are following an approach based on sparse matrix
operations and ESAM (Du et al., 2022b) combines parameter sparsification with
the idea to reduce the number of input samples for second forward/backward
passes. Similarly, Bahri et al. (2021) and Ni et al. (2022) calculate
perturbations on micro-batches. Not explicitly targeted at efficiency
optimization, Mueller & Hein (2022) show that only perturbing Batch Norm
layers even further improves SAM. SAF and its memory efficient version MESA
were proposed by Du et al. (2022a), focusing on storing past iterations
weights to minimize sharpness on the digits output instead of the loss
function. Perturbations in momentum direction after the momentum buffer update
resulting in better performance but no speedup where proposed by Li &
Giannakis (2023). Furthermore, several concepts were proposed to explain the
success of SAM and related approaches (Andriushchenko & Flammarion, 2022;
Möllenhoff & Khan, 2023).

##  2 Method

###  2.1 Notation

Given a finite training dataset
$\mathcal{S}\subset\mathcal{X}\times\mathcal{Y}$ where $\mathcal{X}$ is the
set of possible inputs and $\mathcal{Y}$ the set of possible targets drawn
from a joint distribution $\mathfrak{D}$, we study a model
$f_{\bm{w}}:\mathcal{X}\rightarrow\mathcal{Y}$ parameterized by
$\bm{w}\in\mathcal{W}$, an element-wise loss function
$l:\mathcal{W}\times\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}$, the
distribution loss
$L_{\mathfrak{D}}(\bm{w})=\mathbb{E}_{(x,y)\sim\mathfrak{D}}(l(\bm{w},x,y))$
and the empirical (training) loss
$L_{\mathcal{S}}(\bm{w})=1/|\mathcal{S}|\sum_{(x,y)\in\mathcal{S}}l(\bm{w},x,y)$.
If calculated on a single batch $\mathcal{B}\subset S$ we denote the loss as
$L_{\mathcal{B}}$. We denote the L2-norm by $||\cdot||$.

###  2.2 Sharpness Aware Minimization (SAM)

For many datasets modern neural network architectures and empirical risk
minimization algorithms like SGD or Adam (Kingma & Ba, 2015) effectively
minimize the approximation and optimization error (i.e. finding low
$L_{\mathcal{S}}(\bm{w})$), while reducing the generalization error
($L_{\mathfrak{D}}(\bm{w})-L_{\mathcal{S}}(\bm{w})$) remains a major
challenge. Following ideas of Hochreiter & Schmidhuber (1994), Keskar et al.
(2017) observed a link between sharpness of the minimized empirical loss
$L_{\mathcal{S}}(\bm{w^{\text{opt}}})$ with respect to the parameters and the
generalization error. Intuitively, this follows from the observation that
perturbations in inputs (cf. adversarial training (Goodfellow et al., 2015))
and perturbations in parameters have a similar effect on network outputs (due
to both being factors in matrix-vector products) and that the generalization
error is caused by the limitation to a smaller input subset which resembles an
input perturbation.
Without giving an explicit implementation, Keskar et al. (2017) sketches the
idea of avoiding sharp minima by replacing the empirical loss minimization
with a minimization of the highest loss value within a ball in parameter space
of fixed size $\rho$:

$\underset{\bm{w}}{\text{min}}~{}\underset{||\bm{\epsilon}||\leq\rho}{\text{max}}~{}L_{\mathcal{S}}(\bm{w}+\bm{\epsilon})$
(1)

Foret et al. (2021) propose a computationally feasible algorithm to
approximate this training objective via so-called Sharpness Aware Minimization
(SAM). SAM heavily reduces the computational costs of the inner maximization
routine of Eq. 1 by approximating the loss landscape in first order,
neglecting second order derivatives resulting from the min-max objective, and
performing the maximization on single batches (or per GPU in case of
$m$-sharpness). These simplifications result in adding one gradient ascent
step with fixed step length before the gradient calculation, i.e.,
reformulating the loss as

$L_{\mathcal{B}}^{\text{SAM}}(\bm{w})\coloneqq
L_{\mathcal{B}}(\bm{w}+\bm{\epsilon}^{\text{SAM}})~{}~{}\text{where}~{}~{}\bm{\epsilon}^{\text{SAM}}\coloneqq\rho\frac{\nabla
L_{\mathcal{B}}(\bm{w})}{||\nabla L_{\mathcal{B}}(\bm{w})||}.$ (2)

The parameters are temporarily perturbed by $\bm{\epsilon}^{\text{SAM}}$ in
the direction of the locally highest slope with the perturbation removed again
after gradient calculation. Thus, the parameters are not altered permanently.
While performance improvements could be achieved (Foret et al., 2021; Chen et
al., 2022) the computation of $\bm{\epsilon}^{\text{SAM}}$ demands an
additional backward pass and the computation of
$L_{\mathcal{B}}(\bm{w}+\bm{\epsilon}^{\text{SAM}})$ an additional forward
pass, resulting in roughly doubling the runtime of SAM compared to base
optimizer like SGD or Adam.
Minimizing Eq. 2 can also be interpreted as jointly minimizing the unperturbed
loss function $L_{\mathcal{B}}(\bm{w})$ and the sharpness of the loss
landscape defined by

$S_{\mathcal{B}}(\bm{w})\coloneqq
L_{\mathcal{B}}(\bm{w}+\bm{\epsilon})-L_{\mathcal{B}}(\bm{w}).$ (3)

###  2.3 Momentum and Nesterov Accelerated Gradient

Commonly, SGD is used with momentum, i.e., instead of updating parameters by
gradients directly ($\bm{w}_{t+1}=\bm{w}_{t}-\eta\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t})$ with learning rate $\eta$), an exponential
moving average of past gradients is used for the updates. Given the momentum
factor $\mu$ and the momentum vector $\bm{v}_{t+1}=\mu\bm{v}_{t}+\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t})$ the update rule becomes
$\bm{w}_{t+1}=\bm{w}_{t}-\eta\bm{v}_{t+1}$.
The momentum vector has two averaging effects. First, it averages the gradient
at different positions in the parameter space $\bm{w}_{t}$ and second, it
averages the gradient over multiple batches $\mathcal{B}_{t}$ which can be
interpreted as an increase of the effective batch size.
The update step consists of the momentum vector of the past iteration and the
present iterations gradient. While this gradient is calculated prior to the
momentum vector update in standard momentum training, NAG instead calculates
the gradient after the momentum vector step is performed. The update rule for
the momentum vector thus becomes $\bm{v}_{t+1}=\mu\bm{v}_{t}+\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t}-\eta\mu\bm{v}_{t})$. Analogously to Eq. 2, NAG
can be formulated in terms of a perturbed loss function as

$L_{\mathcal{B}}^{\text{NAG}}(\bm{w})\coloneqq
L_{\mathcal{B}}(\bm{w}+\bm{\epsilon}^{\text{NAG}})~{}~{}\text{where}~{}~{}\bm{\epsilon}^{\text{NAG}}\coloneqq-\eta\mu\bm{v}_{t}.$
(4)

Since the perturbation vector $\bm{\epsilon}^{\text{NAG}}$ neither depends on
the networks output nor its gradient at step $t$ no additional forward or
backward pass is needed.

Input: training data $\mathcal{S}$, momentum $\mu$, learning rate $\eta$,
perturbation strength $\rho$

Initialize: weights $\bm{\widetilde{w}}_{0}\leftarrow$random, momentum vector
$\bm{v}_{0}\leftarrow 0$

for  _$t\leftarrow 0$ to T_ do

sample batch $\mathcal{B}_{t}\subset\mathcal{S}$

$L_{\mathcal{B}_{t}}(\bm{\widetilde{w}}_{t})=1/|\mathcal{B}_{t}|\sum_{(x,y)\in\mathcal{B}_{t}}l(\bm{\widetilde{w}}_{t},x,y)$
// perturbed forward pass

$\bm{g}_{\text{MSAM}}=\nabla L_{\mathcal{B}_{t}}(\bm{\widetilde{w}}_{t})$ //
perturbed backward pass

$\bm{w}_{t}=\bm{\widetilde{w}}_{t}+{\color[rgb]{0,0.390625,0}\rho\frac{\bm{v}_{t}}{||\bm{v}_{t}||}}$
// remove last perturbation

$\bm{v}_{t+1}$ $=$ $\mu\bm{v}_{t}$ $+$ $\bm{g}_{\text{MSAM}}$ // update
momentum vector

$\bm{w}_{t+1}=\bm{w}_{t}-\eta$$\bm{v}_{t+1}$ // SGD step

$\bm{\widetilde{w}}_{t+1}=\bm{w}_{t+1}-$$\rho\frac{\bm{v}_{t+1}}{||\bm{v}_{t+1}||}$
// perturb for next iteration

end for

$\bm{w}_{T}=\bm{\widetilde{w}}_{T}+\rho\frac{\bm{v}_{T}}{||\bm{v}_{T}||}$ //
remove final perturbation

return _$\bm{w}_{T}$_

Algorithm 1 SGD with Momentum-SAM (MSAM; efficient implementation)

Figure 1: Schematic illustrations of optimization algorithms based on SGD. NAG
calculates gradients after updating parameters with the momentum vector. SAM
and MSAM calculate gradients at perturbed positions but remove perturbations
again before the parameter update step. See Alg. 1 for detailed description of
the efficient implementation of MSAM.

###  2.4 Momentum-SAM

Foret et al. (2021) show that performing multiple iterations of projected
gradient ascent in the inner maximization does result in parameters with
higher loss inside the $\rho$-ball (cf. Eq. 1). However, and
counterintuitively, this improved inner maximization does not yield a better
generalization of the model. We conclude that finding the exact (per batch)
local maximum is not pivotal to SAM. Inspired by NAG and given that the
theoretical framework of sharpness minimization is based on calculating the
sharpness on the full training dataset, we propose using the momentum vector
as the perturbation direction and call the resulting algorithm Momentum-SAM
(MSAM) (further perturbations are discussed in Appx. A.7). Following the above
notation, this yields the loss objective

$L_{\mathcal{B}}^{\text{MSAM}}(\bm{w})\coloneqq
L_{\mathcal{B}}(\bm{w}+\bm{\epsilon}^{\text{MSAM}})~{}~{}\text{where}~{}~{}\bm{\epsilon}^{\text{MSAM}}\coloneqq-\rho\frac{\bm{v}_{t}}{||\bm{v}_{t}||}.$
(5)

Contrary to SAM, we perturb in the negative direction. Since we use the
momentum vector before it is updated, a step in the negative direction of the
momentum has already been performed in the iteration before, thus, the loss is
expected to increase again for sufficiently large step sizes. The stepsize by
$\bm{\epsilon}^{\text{MSAM}}$ is typically at least one order of magnitude
higher than learning rate steps, so we overshoot local minima in momentum
direction and reach an increased perturbed loss offering a suitable direction
for sharpness estimation. We empirically show that the momentum descent step
actually results in an ascent on the per-batch loss in Sec. 4.1 and Sec. A.11.
For an efficient implementation, we shift the starting point of each iteration
to be the perturbed parameters
$\bm{\widetilde{w}}_{t}=\bm{w}_{t}-\rho\bm{v}_{t}/||\bm{v}_{t}||$ (in analogy
to common implementations of NAG) and remove the final perturbation after the
last iteration (see Alg. 1). All mentioned optimization strategies are
depicted in detail in Fig. 1. Since SGD with momentum as well as Adam store a
running mean of gradients, MSAM does not take up additional memory and comes
with negligible computational overhead.
Furthermore, we confirm that a similar theoretical generalization bound as
reported by Foret et al. (2021) also holds for directions of high curvature as
the momentum direction (see Appx. A.1).

##  3 Experimental Results

###  3.1 Speed and Accuracy for ResNets on CIFAR100

Table 1: Comparison against multiple (sharpness-aware) optimizers. Baseline optimizers are SGD for CIFAR100 and AdamW for ImageNet. Please see Appx. A.5 for experimental details. MSAM outperforms optimizers of equal speed (AdamW/SGD and NAG) and alternative approaches for faster sharpness reduction. Optimizer | CIFAR100 | ImageNet | Speed
---|---|---|---
WRN-28-10 | WRN-16-4 | ResNet-50 | ViT-S/32
SAM |  $84.16$$\pm 0.12$ |  $79.25$$\pm 0.10$ |  $83.36$$\pm 0.17$ | $69.1$ | 0.52
Baseline |  $81.51$$\pm 0.09$ |  $76.90$$\pm 0.15$ |  $81.46$$\pm 0.13$ | $67.0$ | $\bm{1.00}$
NAG |  $82.00$$\pm 0.11$ |  $77.09$$\pm 0.18$ |  $82.12$$\pm 0.12$ | $-$ | $\bm{0.99}$
LookSAM |  $\bm{83.31}$$\pm 0.12$ |  $\bm{79.00}$$\pm 0.08$ |  $82.24$$\pm 0.11$ | $68.0$ | 0.84
ESAM |  $82.71$$\pm 0.38$ |  $77.79$$\pm 0.11$ |  $80.49$$\pm 0.40$ | $66.1$ | 0.62
MESA |  $82.75$$\pm 0.08$ |  $78.32$$\pm 0.08$ |  $81.94$$\pm 0.26$ | $\bm{69.0}$ | 0.77
MSAM (ours) |  $\bm{83.21}$$\pm 0.07$ |  $\bm{79.11}$$\pm 0.09$ |  $\bm{82.65}$$\pm 0.12$ | $\bm{69.1}$ | $\bm{0.99}$

In Tab. 1, we show test accuracies for MSAM and related optimizers for
WideResNet-28-10, WideResNet-16-4 (Zagoruyko & Komodakis, 2016) and ResNet50
(He et al., 2016) on CIFAR100 (Krizhevsky & Hinton, 2009) and ImageNet-1k
(Deng et al., 2009) next to the training speed. We first tuned the learning
rate and weight decay for SGD/AdamW and then optimized $\rho$ for each model
(see Appx. A.4 for more details). Additionally, we conducted experiments with
related approaches which seek to make SAM more efficient, namely ESAM (Du et
al., 2022b), LookSAM (Liu et al., 2022) and MESA (Du et al., 2022a). While Du
et al. (2022a) also proposed a second optimizer (SAF), we decided to compare
against the memory-efficient version MESA (as recommended by the authors for
e.g. ImageNet). Note that LookSAM required tuning of an additional
hyperparameter. See Appx. A.5 for implementation details on the related
optimizers. Optimizers of the same speed as MSAM (i.e. SGD/AdamW and NAG) are
significantly outperformed. While SAM reaches slightly higher accuracies than
MSAM, twice as much runtime is needed. Accuracies of MSAM and LookSAM do not
differ significantly for WideResNets, however, MSAM performs better on
ResNet-50, is faster, and does not demand additional hyperparameter tuning.
For ESAM we observed only a minor speedup compared to SAM and the accuracies
of MSAM could not be reached. MESA yields similar results to MSAM for ViT on
ImageNet but performs worse on all models on CIFAR100 and is slower compared
to MSAM.

###  3.2 ResNet and ViT on ImageNet Results

Moreover, we test MSAM for ResNets (He et al., 2016) and further ViT variants
(Dosovitskiy et al., 2021) on ImageNet-1k (Deng et al., 2009) and report
results in Tab. 2. Due to limited computational resources, we only run single
iterations, but provide an estimate of the uncertainty by running 5 iterations
of baseline optimizers for the smallest models per category and calculate the
standard deviations. During the learning rate warm-up phase commonly used for
ViTs we set $\rho_{\text{MSAM}}=0$. SAM also benefits from this effect, but
less pronounced, so we kept SAM active during warm-up phase to stay consistent
with related work (see Appx. A.3 for detailed discussion). While performance
improvements are small for ResNets for MSAM and SAM, both optimizers achieve
clear improvements for ViTs. Even though slightly below SAMs performance for
most models, MSAM yields comparable results while being almost twice as fast.
In addition, we conducted experiments for ViT-S/32 on ImageNet when giving
MSAM the same computational budget as SAM (i.e. training for 180 epochs)
yielding a test accuracy of 70.1% and thus clearly outperforming SAMs 69.1%
(also see Appx. A.9).

Table 2: Test accuracies on ImageNet for baseline optimizers (SGD or AdamW), SAM and MSAM. Estimated uncertainties: ResNet: $\pm 0.08$, ViT (90 epochs): $\pm 0.17$, ViT (300 epochs): $\pm 0.24$. Improvements over baseline are given in green. MSAM yields results comparable to SAM for most models while being $\approx 2$ times faster in all our experiments. Model | Epochs | Baseline | SAM | MSAM
---|---|---|---|---
ResNet-50 | 100 | SGD | $76.3$ |  $76.6$$+0.3$ |  $76.5$$+0.2$
ResNet-101 | 100 | SGD | $77.9$ |  $78.7$$+0.8$ |  $78.2$$+0.3$
ViT-S/32 | 300 | AdamW | $67.2$ |  $71.4$$+4.2$ |  $70.5$$+3.3$
90 | AdamW | $67.0$ |  $69.1$$+2.1$ |  $69.1$$+2.1$
ViT-S/16 | 300 | AdamW | $73.0$ |  $78.2$$+5.2$ |  $75.8$$+2.8$
90 | AdamW | $72.6$ |  $75.8$$+3.2$ |  $74.9$$+2.3$
ViT-B/32 | 90 | AdamW | $66.9$ |  $70.4$$+3.5$ |  $70.2$$+3.3$
ViT-B/16 | 90 | AdamW | $73.0$ |  $77.7$$+4.7$ |  $75.7$$+2.7$

###  3.3 Combination with other SAM Variants

As shown by Kwon et al. (2021), weighting the perturbation components by the
parameters significantly improves SAM. Similarly, Mueller & Hein (2022) showed
that applying the perturbations only to the Batch Norm layers (Ioffe &
Szegedy, 2015) yields further enhancements. Both of these techniques can also
be applied to MSAM, yielding test results similar to SAM (see Tab. 3).

Table 3: Test accuracy for different variants of MSAM/SAM on CIFAR100. Adaptive refers to ASAM (Kwon et al., 2021) and BN-only to applying the perturbance only to Batch Norm layers (cf. Mueller & Hein (2022)). MSAM performs well with both variants. Optimizer | WRN-28-10 | WRN-16-4 | ResNet-50
---|---|---|---
SGD |  $81.51$$\pm 0.09$ |  $76.90$$\pm 0.15$ |  $81.46$$\pm 0.13$
vanilla | SAM |  $84.16$$\pm 0.12$ |  $79.25$$\pm 0.10$ |  $83.36$$\pm 0.17$
MSAM |  $83.21$$\pm 0.07$ |  $79.11$$\pm 0.09$ |  $82.65$$\pm 0.12$
adaptive | SAM |  $84.74$$\pm 0.13$ |  $79.96$$\pm 0.13$ |  $83.30$$\pm 0.06$
MSAM |  $84.15$$\pm 0.13$ |  $79.89$$\pm 0.09$ |  $83.48$$\pm 0.08$
BN-only | SAM |  $84.57$$\pm 0.07$ |  $79.73$$\pm 0.24$ |  $84.51$$\pm 0.17$
MSAM |  $83.62$$\pm 0.09$ |  $79.73$$\pm 0.14$ |  $83.49$$\pm 0.19$

##  4 Properties of MSAM

###  4.1 Optimization and Generalization Analysis

Instead of ascending along the positive gradient as in SAM, we propose
perturbing along the negative momentum vector (positive $\rho^{\text{MSAM}}$
in our notation) as it is also done by extragradient methods like Lin et al.
(2020a). We did not observe any increase in test performance when perturbing
in the positive momentum direction (negative $\rho^{\text{MSAM}}$). In fact,
negative $\rho$ values cause a rapid decrease in test accuracy, whereas
positive $\rho$ values cause a gain in test accuracy of more than $2\%$ for a
WideResNet-16-4 trained on CIFAR100 as depicted in Fig. 2A, while NAG only
provides minor improvements.
Fig. 2B shows the same data on logarithmic scale next to the test accuracy.
The ordinate limits are chosen such that baseline (SGD) accuracies as well as
maximal gains by MSAM align. NAG improves the training accuracy greatly,
especially compared to the gains in test accuracy. This underlines that NAG is
designed to foster the optimization procedure (ERM) but does not improve the
generalization capabilities of the model. Similarly, for MSAM the maximal test
accuracy is reached for high values of $\rho$ where the train accuracy dropped
far below the baseline, emphasising the effect of MSAM on the generalization
instead of optimization.
Furthermore, small negative values of $\rho$ induce a steep decrease in
training accuracy while the test accuracy is not significantly affected, but
drops for higher negative $\rho$. This might offer an explanation why MSAM
does not improve the test accuracy with negative $\rho$ values. A perturbation
in the positive momentum vector direction resembles a step back to past
iterations which might result in the gradient not encoding appropriate present
or future information about the optimization direction and thus seems to be
ill-suited to reduce the training loss effectively, counteracting the benefits
of the increased generalization (larger test-train gap). SAM might not suffer
from this effect, since the local (per batch) gradient does not encode much of
the general optimization direction (which is dominated by the momentum
vector), hence, the perturbed parameters disagree with parameters from
previous iterations.
Counterintuitively, the cosine similarity between the momentum vector
$\bm{v}_{t-1}$ and the gradient $g_{t}=\nabla
L_{\mathcal{B}_{t}}(\bm{{w}}_{t})$ is negative for most iterations (though
nears zero at the end of training; see Fig. 2C). Thus, the negative momentum
direction actually has a positive slope, so that perturbing in this direction
resembles an ascent on the per-batch loss, further supporting the analogy of
MSAM and SAM, and offering an explanation for the benefit of the negative sign
(see Appx. A.11 for an empirical validation). Since an update step in the
momentum direction was already performed, further moving in this direction
overshoots the local minima and thus causes the positive slope. This is also
in line with our theoretical considerations in Appx. A.1. Independently of
sharpness minimization, this observation might offer interesting insights to
SGD in general and we intend to follow this up in future work.

Figure 2: WideRestNet-16-4 on CIFAR100 A: Test accuracy for positive and
negative $\rho$ compared against SGD and NAG. B: Train and test accuracy on
logarithmic scale. C: Cosine similarity between momentum vector $\bm{v}_{t-1}$
and gradient $g_{t}=\nabla L_{\mathcal{B}_{t}}(\bm{w}_{t})$. Momentum vector
direction has mostly negative slope during training and approaches zero at the
end.

###  4.2 Similarity between SAM and MSAM

To support our hypotheses that MSAM yields a valid alternative to SAM, we
investigate the similarity between the resulting gradients. After searching
for the optimal value, we keep $\rho_{\text{SAM}}=0.3$ fixed, train a model
with SAM, and calculate the gradients

$\displaystyle\bm{g}_{\text{SGD}}$ $\displaystyle=\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t})$ (6) $\displaystyle\bm{g}_{\text{SAM}}$
$\displaystyle=\nabla L_{\mathcal{B}_{t}}(\bm{w}_{t}+\rho_{\text{SAM}}\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t})/||\nabla L_{\mathcal{B}_{t}}(\bm{w}_{t})||)$
(7) $\displaystyle\bm{g}_{\text{MSAM}}(\rho_{\text{MSAM}})$
$\displaystyle=\nabla
L_{\mathcal{B}_{t}}(\bm{w}_{t}-\rho_{\text{MSAM}}\bm{v}_{t}/||\bm{v}_{t}||)$
(8)

while we keep $\rho_{\text{MSAM}}$ as a free parameter. To eliminate gradient
directions which do not contribute in distinguishing between SAM and SGD
gradients, we first project $\bm{g}_{\text{MSAM}}$ into the plane spanned by
$\bm{g}_{\text{SAM}}$ and $\bm{g}_{\text{SGD}}$ and then calculate the angle
$\theta$ to $\bm{g}_{\text{SAM}}$ (see Fig. 3A). By repeating this every 50
iterations for various values $\rho_{\text{MSAM}}$ and calculating the value
of zero-crossing $\rho_{0}$, we determine when the maximal resemblance to SAM
is reached (see Fig. 3B). As shown in Fig. 3C, $\rho_{0}$ reaches values close
to the optimal regime of $\rho_{\text{MSAM}}^{\text{opt}}\approx 3$ (cf. Fig.
2A) for most epochs. While this correlation does not yield strict evidence it
offers additional empirical support for the similarity between SAM and MSAM
gradients.

Figure 3: A: Projecting $g_{\text{MSAM}}$ into the plane of $g_{\text{SAM}}$
and $g_{\text{SGD}}$ to measure SAM/MSAM similarity. B: Varying
$\rho_{\text{MSAM}}$ until maximal similarity is reached (i.e. $\theta=0$) and
determine $\rho_{0}$. C: $\rho_{\text{MSAM}}$ at maximal similarity $\rho_{0}$
is close to generalization optimality ($\rho_{\text{MSAM}}^{\text{opt}}$, cf.
Fig. 2A) for most epochs.

###  4.3 Loss Sharpness Analysis

Figure 4: Sharpness (Eq. 3) after full training (200 epochs) for
WideResNet-16-4 on CIFAR100 along different directions $\bm{\epsilon}$ scaled
by $\rho$. A: Gradient direction (as used as perturbation in SAM). B: Momentum
direction (as in MSAM). C: Random filter-normalized direction as in Li et al.
(2018). Vertical line at $\rho^{\text{opt}}$ marks values of optimal test
performance (cf. Fig. A.3A). MSAM and SAM are reducing their respective
optimization objective best while MSAM reaches the lowest sharpness along
random directions.

As mentioned above (and discussed by Foret et al. (2021)), the SAM
implementation performs the loss maximization step on a single data batch
instead of the full training set ($L_{\mathcal{B}}$ vs. $L_{\mathcal{S}}$). To
analyze the efficacy of SAMs sharpness minimization, we therefore compare the
sharpness (cf. Eq. 3) in the direction of local (per batch) gradients for
models after full training with SGD, SAM and MSAM as a function of the
perturbation scale $\rho$ in Fig. 4A. The minima in local gradient directions
are shifted from $\rho=0$, since parameters found after training are usually
not minima but saddle points (Dauphin et al., 2014). Compared to the other
optimizers, SAM successfully minimizes the sharpness, especially at optimal
$\rho_{\text{opt}}$ (as used during training).
The sharpness in momentum direction (Fig. 4B) represents the MSAM objective
(Eq. 5). Here we do not include the negative sign in the definition of
$\bm{\epsilon}$ (as in Eq. 5), hence $\rho_{\text{opt}}$ is negative. As
expected, MSAM reduces this sharpness best. In contrast to the definition
before, the sharpness is symmetric now for positive and negative signs. While
MSAM only minimizes the sharpness in the negative direction explicitly, the
positive direction is reduced jointly, further supporting the validity of
perturbations in the negative momentum direction.
In Fig. 4C, we choose filter-normalized random vectors as perturbations as in
Li et al. (2018). Since the loss landscape is rotational symmetric around the
origin for multiple directions (as used in the original paper), we confine out
analysis to one perturbation dimension. MSAM reaches the lowest sharpness,
while both MSAM and SAM, significantly flatten the loss. This might be caused
by MSAM approximating the maximization of $L_{\mathcal{S}}$ better due to the
momentum vector $\bm{v}_{t}$ being an aggregation of gradients of multiple
batches.
Interestingly, this relates to the findings of Foret et al. (2021) regarding
$m$-sharpness, where the authors performed the maximization on even smaller
data samples (fractions of batches per GPU in distributed training), yielding
even better generalization. In this sense MSAM, reduces $m$ even further over
ordinary SAM ($m=1$). In the same line of argument and despite being more
efficient, MSAM oftentimes does not improve generalization compared to SAM.
However, contradicting the general idea behind correlations of generalization
and sharpness, MSAM yields flatter minima (if defined as in Fig. 4C), hence,
indicating that explanations for the improved generalization of SAM/MSAM go
beyond the reduction in sharpness.
We additionally analyze the loss curvature for SGD, SAM and MSAM in Appx. A.2.

###  4.4 Normalization

To gain a better understanding of the relation between MSAM and NAG, we
conducted an ablation study by gradually altering the MSAM algorithm until it
matches NAG. Firstly, we drop the normalization of the perturbation
$\bm{\epsilon}$ (numerator in Eq. 5), then we reintroduce the learning rate
$\eta$ to scale $\bm{\epsilon}$, and finally set $\rho=1$ to arrive at NAG.
Train and test accuracies as functions of $\rho$ are shown in Fig. 5 for all
variants. Dropping the normalization only causes a shift of $\rho$ indicating
that changes of the momentum norm during training are negligible. However,
scaling by $\eta$ drastically impacts the performance. Since the model is
trained with a cosine learning rate scheduler (Loshchilov & Hutter, 2017),
$\rho$ decays jointly with $\eta$. The train accuracy is increased
significantly not only for $\rho=1$ (NAG), but even further for higher $\rho$,
while the test performance drops at the same time when compared to MSAM. Thus,
optimization is improved again while generalization is not, revealing
separable mechanisms for test performance improvements of MSAM and NAG. High
disturbances compared to the step size at the end of training appear to be
crucial for increased generalization. Extensively investigating the effect of
SAM/MSAM during different stages of training might offer potential to make SAM
even more effective and/or efficient (i.e. by scheduling $\rho$ to only apply
disturbances for selected episodes).

Figure 5: Test (A) and train (B) accuracy for WideResNet-16-4 on CIFAR100 for
different normalization schemes of MSAM in dependence on $\rho$. MSAM without
normalization works equally well. If the perturbation $\epsilon$ is scaled by
learning rate $\eta$ train performance (optimization) is increased while test
performance (generalization) benefits only marginally.

##  5 Conclusion

In this work we introduced Momentum-SAM (MSAM), an optimizer achieving
comparable results to the SAM optimizer while requiring no significant
computational or memory overhead over optimizers such as Adam or SGD, hence,
halving the computational load compared to SAM. On the one hand, this reduces
the major hindrance for a widespread application of SAM-like algorithms when
training resources are limited. On the other hand, we showed that
perturbations independent of local gradients (in particular the momentum
direction) can yield significant performance enhancements. Given the affinity
of MSAM and NAG, we demonstrated new perspectives on SAM/MSAM as well as on
NAG and underlined the importance as well as the need for future research of
scaling and scheduling of perturbations to further improve sharpness-aware
optimizers.

## Acknowledgements

This work was funded by the Deutsche Forschungsgemeinschaft (DFG CRC 1459;
Project-ID 433682494).

## References

  * Andriushchenko & Flammarion (2022) Maksym Andriushchenko and Nicolas Flammarion.  Towards understanding sharpness-aware minimization.  In _International Conference on Machine Learning_ , pp. 639–668, 2022.
  * Bahri et al. (2021) Dara Bahri, Hossein Mobahi, and Yi Tay.  Sharpness-aware minimization improves language model generalization.  In _Annual Meeting of the Association for Computational Linguistics_ , 2021.
  * Beyer et al. (2022) Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov.  Better plain vit baselines for imagenet-1k.  _CoRR_ , abs/2205.01580, 2022.
  * Bojar et al. (2016) Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation_ , pp. 131–198, Berlin, Germany, August 2016. Association for Computational Linguistics.
  * Chaudhari et al. (2017) Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina.  Entropy-SGD: Biasing gradient descent into wide valleys.  In _International Conference on Learning Representations_ , 2017.
  * Chen et al. (2022) Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.  When vision transformers outperform resnets without pre-training or strong data augmentations.  In _International Conference on Learning Representations_ , 2022.
  * Damian et al. (2021) Alex Damian, Tengyu Ma, and Jason D. Lee.  Label noise SGD provably prefers flat global minimizers.  In _Advances in Neural Information Processing Systems_ , pp. 27449–27461, 2021.
  * Daskalakis et al. (2018) Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng.  Training gans with optimism.  In _International Conference on Learning Representations_ , 2018.
  * Dauphin et al. (2014) Yann N. Dauphin, Razvan Pascanu, Çaglar Gülçehre, KyungHyun Cho, Surya Ganguli, and Yoshua Bengio.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.  In _Advances in Neural Information Processing Systems_ , pp. 2933–2941, 2014.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 248–255, 2009.
  * Dinh et al. (2017) Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.  Sharp minima can generalize for deep nets.  In _Proceedings of the 34th International Conference on Machine Learning_ , pp. 1019–1028, 2017.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Du et al. (2022a) Jiawei Du, Zhou Daquan, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou.  Sharpness-aware training for free.  In _Advances in Neural Information Processing Systems_ , 2022a.
  * Du et al. (2022b) Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan.  Efficient sharpness-aware minimization for improved training of neural networks.  In _International Conference on Learning Representations_ , 2022b.
  * Foret et al. (2021) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  In _International Conference on Learning Representations_ , 2021.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations_ , 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016.
  * Hochreiter & Schmidhuber (1994) Sepp Hochreiter and Jürgen Schmidhuber.  Simplifying neural nets by discovering flat minima.  In _Advances in Neural Information Processing Systems_ , 1994.
  * Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis R. Bach and David M. Blei (eds.), _Proceedings of the 32nd International Conference on Machine Learning_ , pp. 448–456, 2015.
  * Jiang et al. (2023) Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok.  An adaptive policy to employ sharpness-aware minimization.  In _International Conference on Learning Representations_ , 2023.
  * Juditsky et al. (2011) Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel.  Solving variational inequalities with stochastic mirror-prox algorithm.  _Stochastic Systems_ , 1(1):17–58, 2011.
  * Keskar et al. (2017) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _International Conference on Learning Representations_ , 2017.
  * Kim et al. (2022) Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales.  Fisher SAM: Information geometry and sharpness aware minimisation.  In _Proceedings of the 39th International Conference on Machine Learning_ , pp. 11148–11161, 2022.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Korpelevich (1976) G. M. Korpelevich.  The extragradient method for finding saddle points and other problems.  In _Ekonomika i Matematicheskie Metody_ , volume 12, pp. 747–756, 1976.
  * Krizhevsky & Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, Toronto, Ontario, 2009.
  * Kwon et al. (2021) Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi.  Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks.  In _Proceedings of the 38th International Conference on Machine Learning_ , pp. 5905–5914, 2021.
  * Li & Giannakis (2023) Bingcong Li and Georgios B. Giannakis.  Enhancing sharpness-aware optimization through variance suppression.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=Sf3t6Bth4P.
  * Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Lin et al. (2020a) Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi.  Extrapolation for large-batch training in deep learning.  In _Proceedings of the 37th International Conference on Machine Learning_ , pp. 6094–6104, 2020a.
  * Lin et al. (2020b) Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi.  Don’t use large mini-batches, use local SGD.  In _International Conference on Learning Representations_ , 2020b.
  * Liu et al. (2020) Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk.  On the loss landscape of adversarial training: Identifying challenges and how to overcome them.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Liu et al. (2022) Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You.  Towards efficient and scalable sharpness-aware minimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12360–12370, 2022.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In _International Conference on Learning Representations_ , 2017.
  * Mi et al. (2022) Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao.  Make sharpness-aware minimization stronger: A sparsified perturbation approach.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Möllenhoff & Khan (2023) Thomas Möllenhoff and Mohammad Emtiyaz Khan.  SAM as an optimal relaxation of bayes.  In _International Conference on Learning Representations_ , 2023.
  * Mueller & Hein (2022) Maximilian Mueller and Matthias Hein.  Perturbing batchnorm and only batchnorm benefits sharpness-aware minimization.  In _Has it Trained Yet? NeurIPS 2022 Workshop_ , 2022.
  * Nakkiran et al. (2021) Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.  Deep double descent: Where bigger models and more data hurt.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2021(12):124003, 2021.
  * Nesterov (1983) Y. Nesterov.  A method for solving a convex programming problem with convergence rate o(1/k2), 1983.
  * Ni et al. (2022) Renkun Ni, Ping yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein.  K-sam: Sharpness-aware minimization at the speed of sgd, 2022.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv e-prints_ , 2019.
  * Sutskever et al. (2013) Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.  On the importance of initialization and momentum in deep learning.  In _Proceedings of the 30th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, pp. 1139–1147, 2013.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1–9, 2015.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler.  Scale efficiently: Insights from pretraining and finetuning transformers.  In _International Conference on Learning Representations_ , 2022.
  * Wen et al. (2018) Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Yiran Chen, and Hai Li.  Smoothout: Smoothing out sharp minima for generalization in large-batch deep learning.  _CoRR_ , abs/1805.07898, 2018.
  * Wu et al. (2020) Dongxian Wu, Shu-Tao Xia, and Yisen Wang.  Adversarial weight perturbation helps robust generalization.  In _Advances in Neural Information Processing Systems_ , pp. 2958–2969, 2020.
  * Yao et al. (2018) Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney.  Hessian-based analysis of large batch training and robustness to adversaries.  In _Advances in Neural Information Processing Systems_ , pp. 4954–4964, 2018.
  * Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In _Proceedings of the British Machine Vision Conference_ , 2016.
  * Zheng et al. (2021) Yaowei Zheng, Richong Zhang, and Yongyi Mao.  Regularizing neural networks via adversarial model perturbation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8156–8165, 2021.
  * Zhuang et al. (2022) Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, sekhar tatikonda, James s Duncan, and Ting Liu.  Surrogate gap minimization improves sharpness-aware training.  In _International Eleventh Conference on Learning Representations_ , 2022.

##  Appendix A Appendix

###  A.1 Theoretical Analysis of MSAM

We build our analysis in analogy to Foret et al. (2021). While Foret et al.
(2021) proofed the existence of an upper generalization bound if the
parameters with the highest loss in a fixed $\rho$-ball are found, we show
that a similar bound can also be derived by simply assuming perturbations in
directions of high curvature. In practice this first assumption is fulfilled
by the momentum vector $\bm{v}_{t}$, since gradients are directions of high
curvature. Secondly, if a properly tuned learning rate is used, the slope in
momentum direction after the parameter update is either close to zero or even
negative caused by overshooting marginally.
We state these two assumptions in Setting 1. While we empirically validate the
first assumption in Appx. A.2, we already showed and discussed evidence for
the second assumption in the main text (cf. Fig. 2C).

###### Proposition 1

Let $\bm{\epsilon},\bm{v}\in\mathcal{W}$ with i.i.d. components
$\bm{\epsilon}_{i}\sim\mathcal{N}(0,\sigma)$ for some $\sigma>0$, then for any
$\rho>0$

$\mathbb{E}[\mathbf{1}_{\\{\|\bm{\epsilon}\|_{2}\leq\rho\\}}\bm{\epsilon}^{T}\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))\bm{\epsilon}]\leq\rho^{2}\kappa,$
(9)

where
$\kappa:=\frac{1}{|\mathcal{W}|}\mathrm{tr}[\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))]$.

#### Proof

W.L.O.G. we assume $\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))$ to be diagonal.
The claim then follows from the linearity of the expectation and symmetry.
$\square$

###### Setting 1

Let $\bm{w},\bm{v}\in\mathcal{W}$ with $\|\bm{v}\|_{2}=1$ such that:

  * •

$\bm{v}^{T}\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))\bm{v}>\kappa$,

  * •

$\nabla L_{S}(\bm{w})\cdot\bm{v}\leq 0$.

###### Lemma 2

Assume Setting 1 and let $\bm{\epsilon}\in\mathcal{W}$ with
$\bm{\epsilon}_{i}\sim\mathcal{N}(0,\sigma)$, then it holds for any $\rho>0$
that

$\mathbb{E}[\mathbf{1}_{\\{\|\epsilon\|_{2}\leq\rho\\}}L_{\mathcal{S}}(\bm{w}+\bm{\epsilon})]\leq
L_{\mathcal{S}}(\bm{w}-\rho\bm{v})+\mathcal{O}(\rho^{3}).$

#### Proof

Applying a Taylor expansion around $\bm{w}$ yields:

$\displaystyle\mathbb{E}[\mathbf{1}_{\\{\|\bm{\epsilon}\|_{2}\leq\rho\\}}L_{\mathcal{S}}(\bm{w}+\bm{\epsilon})]$
$\displaystyle\leq L_{\mathcal{S}}(\bm{w}-\rho\bm{v})$
$\displaystyle\iff\underbrace{\mathbb{E}[\mathbf{1}_{\\{\|\bm{\epsilon}\|_{2}\leq\rho\\}}\nabla
L_{S}(\bm{w})\cdot\bm{\epsilon}]}_{=0}+\underbrace{\mathbb{E}[\mathbf{1}_{\\{\|\bm{\epsilon}\|_{2}\leq\rho\\}}\bm{\epsilon}^{T}\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))\bm{\epsilon}]}_{\leq\rho^{2}\kappa}$
$\displaystyle\leq$ $\displaystyle\underbrace{-\rho\nabla
L_{\mathcal{S}}(\bm{w})\cdot\bm{v}}_{\geq 0}+$
$\displaystyle\rho^{2}\bm{v}^{T}\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))\bm{v}+\mathcal{O}(\rho^{3})$
$\displaystyle\impliedby\kappa$
$\displaystyle\leq\bm{v}^{T}\mathrm{Hess}(L_{S}(\bm{w}))\bm{v}+\mathcal{O}(\rho^{3})$

subtracting the $\mathcal{O}(\rho^{3})$-term from the initial inequality then
yields the claim. $\square$

###### Theorem 3

Assume Setting 1 then for any distribution $\mathfrak{D}$, with probability
$1-\delta$ over the choice of the training Set $\mathcal{S}\sim\mathfrak{D}$,

$L_{\mathfrak{D}}(\bm{w})\leq
L_{\mathcal{S}}(\bm{w}-\rho\bm{v})+\sqrt{\frac{\dim(\mathcal{W})\log\Big{(}1+\frac{\|\bm{w}\|_{2}^{2}}{\rho^{2}}\Big{(}1+\sqrt{\frac{\log(|\mathcal{S}|)}{\dim(\mathcal{W})}}\Big{)}^{2}\Big{)}+4\log\frac{|\mathcal{S}|}{\delta}+\mathcal{O}(1)}{|\mathcal{S}|-1}}+\mathcal{O}(\rho^{3})$

#### Proof

Using the bound from Lemma 2 we adapt the proof of Theorem 2 in Foret et al.
(2021) (i.e. Eq. 13 and following) to show the claim. $\square$

###  A.2 Curvature

We calculated the loss curvature in momentum direction, gradient direction and
in random direction in Fig. A.1 if training with SGD, SAM and MSAM for
WRN-16-4 on CIFAR100. For this we calculate
$\bm{\epsilon}^{T}\mathrm{Hess}(L_{\mathcal{S}}(\bm{w}))\bm{\epsilon}$ for
direction vectors $\bm{\epsilon}$ (normalized to $||\bm{\epsilon}||_{2}=1$)
every 50 optimizer steps.
The curvatures in momentum directions are larger than the curvature random
direction (which tends towards the mean curvature as amount of parameters
increase) for all optimizers and epochs, validating Setting 1 in Appx. A.1 and
thus the suitability of momentum directions for sharpness estimation
(especially compared to random perturbations; cf. Appx. A.7).
Additionally, the curvature in these directions offers a measure for the loss
sharpness. Since a local minimum of high curvature is approached, all three
curvatures increase at the end of the training for SGD. Similarly to Fig. 4,
SAM and MSAM are reducing the curvature best in their corresponding
perturbation direction and MSAM yields lower curvatures than SAM in random
directions.

Figure A.1: Curvature of random directions (RND), momentum (MOM) and gradient
(GRAD) for different optimizers.

###  A.3 Effects of MSAM/SAM During ViT’s Warm-up Phase

We further investigate the effect of SAM/MSAM during warm-up phase in Tab.
A.1. As described above, we do not apply MSAM during the warmup phase by
default (i.e. setting $\rho=0$) since if doing so, we observe a drop in test
accuracy from $69.1\%$ to $66.1\%$ which is below the AdamW baseline. We
assume fluctuations of the momentum vector, that determines the perturbation
direction for MSAM, to cause instabilities during the warmup phase. A similar
effect can be seen for SAM, however, it is less pronounced, so that applying
perturbations during the warm-up phase does not thwart SAM hugely. Since we
focused on proposing a computationally more efficient variant and not on
improving the generalization of SAM in this work, we thus decided to stay
consistent with related work and conduct our extensive experiments in Tab. 2
while applying SAM also during the warm-up phase. Nevertheless, we would
generally propose to apply SAM only after the warmup phase for ViT models to
further improve SAM. We think further investigating effects of
$\rho$-scheduling for SAM and MSAM is of high interest. E.g., Zhuang et al.
(2022) investigated to reduce $\rho$ during training (contrary to what our
findings suggest) by binding it to the learning rate scheduling for SAM and
they did not notice benefits. Despite the discussion in Sec. 4.4, we could not
observe analogous effects for ResNets (though we did not study these
extensively).

Table A.1: Impact of application of SAM/MSAM during warm-up phase. ViT S/32 on ImageNet. By default MSAM is applied after warmup phase only while SAM is always applied. AdamW | SAM | SAM (after warmup only) | MSAM | MSAM (during warmup)
---|---|---|---|---
$67.1$ | $69.2$ | $69.8$ | $69.1$ | $66.1$

###  A.4 Training and Implementation Details

If not stated differently, we calculate uncertainties of mean accuracies by
$68\%$ CI estimation assuming Student’s t-distribution.
We tuned weight decay and learning rates for our baseline models (SGD/AdamW)
and did not alter these parameters for the other used optimizing strategies.
We used basic augmentations (horizontal flipping, cutout and cropping) for
CIFAR100 trainings and normalized inputs to mean 0 and standard deviation 1.
For ImageNet trainings we used Inception-like preprocessing (Szegedy et al.,
2015) with 224x224 resolution, normalized inputs to mean 0 and std 1 and
clipped gradients L2-norms to 1.0. We used ViT variants proposed by Beyer et
al. (2022). A full implementation comprising all models and configuration
files is available at https://XXXXXXXX.

Table A.2: Training Hyperparameters | CIFAR100 | ImageNet
---|---|---
| WideResNets | ResNet50 | ResNets | ViTs
Base Optimizer | SGD | SGD | SGD | AdamW
Epochs | 200 | 200 | 100 | 90/300
Learning Rate | $0.5$ | $0.1$ | $1$ | 1e-3
LR-Scheduler | cos | cos | cos | cos + linear warm-up (8 epochs)
Label Smoothing | 0.1 | 0.1 | 0.1 | -
Batch Size | 256 | 256 | 1024 | 1024
Weight Decay | 5e-4 | 1e-3 | 1e-4 | 0.1

###  A.5 Details on Optimizer Comparison

We report experimental details on the results presented in Tab. 1 in this
section.
To calculate the speed, we conducted a full optimization on a single GPU for
each model and dataset combination, normalized the runtime by SGDs runtime and
report the average over all runs per combination.
We trained ViT-S/32 on ImageNet for 90 epochs for all models. Further
hyperparameters not specific to SAM variants are reported above in Appx. A.4.
Due to limited computational capacities and inline with related work, we did
not perform runs for multiple random seeds for ImageNet trainings. Thus, we
did not report standard deviations for these runs.
We adapted official implementations of ESAM (Du et al., 2022b) and MESA (Du et
al., 2022a) while no official implementation was available for LookSAM (Liu et
al., 2022).
For LookSAM, we fixed the trade-off parameter $k=5$ and conducted a thorough
search on the additional hyper parameter $\alpha$, since the value suggested
for ViTs by the original authors ($\alpha=0.7$) was not suitable for our
experiments (also see Fig. A.4). We decided to set $\alpha=0.1$, while runs
for $\alpha>0.3$ did not yield further performance increases. Full
hyperparameter search results are reported in Fig. A.2.
ESAM comprises two hyperparameter ($\gamma$ and $\beta$) that steer the
performance/runtime tradeoff which we set to match those of the original paper
(i.e. $\gamma=\beta=0.5$).
For MESA we tuned the regularization factor $\lambda$ instead of the
perturbation strength $\rho$.
Please also note the full $\rho$ scan results presented in the next section
(Fig. A.3 and Fig. A.4)

Figure A.2: Full results for $\rho$ and $\alpha$ search for lookSAM ($k=5$) on
CIFAR100. Shaded cone: $68\%$ CI. Dots: Random Seeds. A: WideResNet-16-4. B:
WideResNet-28-10. C: ResNet-50.

###  A.6 Full Hyper Parameter Search Results

We report our full $\rho$-hyperparameter search results in Fig. A.3, Fig. A.4
and Fig. A.5. In consistency with related work, we report results for best
$\rho$ only, in the main text. We sampled $\rho$ with approximately even
spacing on logarithmic scale with 6 datapoints per decade, i.e,
${\rho\in\\{...,0.1,0.15,0.22,0.34,0.5,0.67,1,1.5,...\\}}$, for experiments on
CIFAR100 and with 4 datapoints per decade, i.e,
${\rho\in\\{...,0.1,0.17,0.3,0.55,1,1.7,...\\}}$, on ImageNet for experiments
in Sec. 3. We used a slightly denser sampling for the visualizations in Sec.
4, but did not use those results for comparisons against baselines or other
methods.
While optimal values for $\rho$ vary slightly between models and datasets, we
do not observe higher susceptibility to changes in $\rho$ of MSAM compared to
SAM.
Over all models and datasets we find higher optimal $\rho$ values for MSAM
compared to SAM. Perturbation vectors are normalized (L2-norm), so we
conjecture components for parameters of less importance to be more pronounced
for momentum vectors compared to gradients on single batches. For ViT models,
we find optimal $\rho$ values to be higher compared to ResNets. If chosen even
higher, heavy instabilities occur during training, up to models not
converging, limiting performances especially for MSAM. Similar to the
observations during warm-up phase discussed above, this effect is more
pronounced for MSAM. Notably, MSAM looses most performance against SAM on the
biggest ViT models and if trained for 300 epochs, when highest $\rho$ values
are optimal for SAM. This suggests, that even better performances might be
achievable for MSAM if the instability problems are tackled. Strategies to do
so might include e.g. clipping $\bm{\epsilon}$ or scheduling of $\rho$, which
we intend to pursue in future work.

Figure A.3: Full results for $\rho$-search for different SAM/MSAM variants on
CIFAR100. AMSAM/ASAM refer to adaptive-MSAM/adaptive-SAM as in Kwon et al.
(2021). For LookSAM we set $k=5$ and report only the best performing value of
$\alpha=0.1$ (cf. Fig. A.2). For MESA: $\lambda$-search results plotted on
same axis. Shaded cone: $68\%$ CI. Dots: Random Seeds. A: WideResNet-16-4. B:
WideResNet-28-10. C: ResNet-50. Figure A.4: Full results for $\rho$-search for
different SAM/MSAM variants for ViT-S/32 trained for 90 epochs on ImageNet.
For MESA: $\lambda$-search results plotted on same axis.

Figure A.5: Full results for $\rho$-search for all models tested on ImageNet
(see Tab. 2). A: Vision Transformer (epochs in parentheses). B: ResNets.

###  A.7 Random and Last Gradient Perturbations

Instead of the momentum vector $v_{t}$ in MSAM, we also tried to use other
perturbations $\epsilon$ which are independent of the current gradient and
thus do not bring significant computational overhead, namely the last
iterations gradients $g_{t-1}$ (cf. Daskalakis et al. (2018); Lin et al.
(2020a)) with positive and negative sign as well as Gaussian random vectors
(cf. Wen et al. (2018)). For each variation, we tested absolute perturbations
(Fig. A.6A)

$\epsilon^{\text{ABS}}=\rho\frac{\delta}{||\delta||}$ (10)

and relative perturbations (Fig. A.6B)

$\epsilon^{\text{REL}}=\rho\frac{\delta|w|}{||\delta w||},$ (11)

with weights $w$ (multiplied element-wise) and, e.g., $\delta=-v_{t}$ for
MSAM.
MSAM provides the only perturbation reaching SAM-like performance without
inducing relevant computational overhead.

Figure A.6: Random perturbations and last gradient perturbations compared to
SAM and MSAM. A: Absolute perturbations, B: Relative perturbations, i.e.,
scaled by $|w_{i}|$ before normalization. All perturbations normalized by
L2-norm and scaled by $\rho$. WideResNet 16-4 trained on CIFAR100. MSAM always
better than other current gradient-independent perturbations.

###  A.8 Hyperparemeter Stability

To show the stability of MSAM and its hyper parameter $\rho$, we varied the
learning rate $\eta$ and the momentum factor $\mu$ when optimizing a WRN-16-4
on CIFAR100 for fixed $\rho=2.2$ and depict results in Fig. A.7. MSAM yields
stable performance increases compared to SGD and NAG over wide ranges of
hyperparameters. We made similar observations when comparing SGD and MSAM for
different number of epochs (cf. Fig. A.8).

Figure A.7: WRN-16-4 on CIFAR100 fixed $\rho=2.2$. A: Learning rate $\eta$
ablation (log-scale). B: Momentum factor $\mu$ ablation.

###  A.9 Comparison with Same Computational Budgets

We compare MSAM and SAM (and SGD and NAG) when given the same computational
budget for WRN-16-4 on CIFAR100 for a wider range of epochs (up to 1200) in
Fig. A.8. I.e., running SAM for half the number of epochs compared to other
optimizers, resulting in the same number of network passes for all optimizers.
MSAM performs similar to NAG (and SGD) for short training times, however, if
trained until convergence of SGD/NAG or even longer (overfitting occurs;
SGD/NAG results decrease again) MSAM reaches higher test accuracies and
overfitting is prevented. Due to the additional forward/backward passes SAM
performs worse compared to MSAM for limited computational budgets. For long
training times MSAM and SAM do not differ significantly.
We further support these observations by training a ViT-S/32 with MSAM with
doubled number of epochs (180) on ImageNet where we reach $70.1\%$ test
accuracy and thus clearly outperform SAMs $69.1\%$ (cf. Tab. 2).

Figure A.8: Comparing different optimizers when given the same computational
budget. WRN-16-4 on CIFAR100.

###  A.10 Machine Translation

Machine translation results (English to Romanian) on the WMT 2016 (Bojar et
al., 2016)dataset by finetuning a T5-tiny model (efficient version by Tay et
al. (2022)) from a publicly available checkpoint pretrained on the C4 dataset
(Raffel et al., 2019). We scanned $\rho\in\\{0.01,0.03,0.1,0.3,1\\}$ and found
MSAM and SAM both to perform best at $\rho=0.1$. The resulting BLEU scores are
shown in Tab. A.3. MSAM slightly outperforms SAM (as well as AdamW) while
requiring two times less computations.

Table A.3: BLEU scores for T5-tiny trained on English to Romanian translation on the WMT 2016 dataset. AdamW | SAM | MSAM
---|---|---
$23.35$ | $23.57$ | $23.64$

###  A.11 Loss Ascent in Momentum Direction

To validate that the perturbation of the loss results in a loss increase, we
show the perturbed and unperturbed loss during training for different learning
rates in Fig. A.9.

Figure A.9: Loss before $(L_{\mathcal{B}_{t}}(\bm{w}_{t}))$ and after
$(L_{\mathcal{B}_{t}}(\bm{w}_{t}-\rho\bm{v}_{t}/||\bm{v}_{t}||))$ perturbation
in momentum direction as done by MSAM ($\rho=3$) for WRN 16-4 on CIFAR100 for
different learning rates.