# Multilevel Markov Chain Monte Carlo for Bayesian inverse problems for Navier
Stokes equation with Lagrangian Observations

Yang Juntao Email address<EMAIL_ADDRESS>Nvidia AI Technology Center,
Singapore
Division of Mathematical Sciences, School of Physical and Mathematical
Sciences,
Nanyang Technological University, Singapore

##  1 Introduction

We consider the Bayesian inverse problem for recovering the initial velocity
and the random forcing of forward Navier-Stokes equation with noisy
observation on the position of tracers at some time moments. Data inversion
for Navier-Stokes equation is important for areas like weather forecasting,
ocean modeling and aerospace engineering. For weather forecasting and ocean
modeling, it is common to have positional data from tracers such as weather
balloons and drifters. However, sampling the posterior probability of Bayesian
inverse problem for the forward Navier-Stokes equation coupled with tracer
equation is highly expensive if not intractable. The Markov Chain Monte Carlo
(MCMC) sampling procedure requires a large number of realizations of the
forward equation to obtain a reasonable level of accuracy, hence leads to high
complexity.

For linear elliptic forward problems, in the case of the uniform prior
probability measure, Hoang, Schwab and Stuart [7] develop the Multilevel
Markov Chain Monte Carlo (MLMCMC) method that approximates the expectation
with respect to the posterior probability measure of quantities of interest by
solving the forward equation by finite elements (FEs) with different levels of
resolution. The number of MCMC samples is chosen judiciously according to the
FE resolution level. The method is essentially optimal. To obtain an
approximation for the posterior expectation of a quantity of interest within a
prescribed level of accuracy, the total number of degrees of freedom required
for solving all the realizations of the forward equation in the sampling
process is equivalent to that for solving only one realization of the forward
equation to obtain an equivalent level of FE accuracy. Comparing to the plain
MCMC procedure where a large number of realizations of the forward equation is
solved with an equally high level of accuracy, the computation time is
drastically reduced. The convergence of the method is rigorously proved.
Numerical experiments verify the theoretical convergence rate. We have early
applied MLMCMC method to Bayesian inverse problem for inferring the unknown
forcing and initial condition of the forward Navier-Stokes equation with noisy
Eulerian observations on the velocity. [11]. Rigorous theory has been
developed for the case of uniform prior where the forcing and the initial
condition depend linearly on a countable set of random variables which are
uniformly distributed in a compact interval.

In this paper, we extend our work to the Bayesian inverse problems for
inferring unknown forcing and initial condition of the forward Navier-Stokes
equation coupled with tracer equation with noisy Lagrangian observation on the
positions of the tracers. We consider the Navier-Stokes equations in the two
dimensional periodic torus with a tracer equation which is a simple ordinary
differential equation. We developed rigorously the theory for the case of the
uniform prior where the forcing and the initial condition depend linearly on a
countable set of random variables which are uniformly distributed in a compact
interval. Numerical experiment using the MLMCMC method produces approximations
for posterior expectation of quantities of interest which are in agreement
with the theoretical optimal convergence rate established.

##  2 Parametric Navier Stokes equation

Let $\mathbb{T}^{2}$ be the two dimensional unit torus in $\mathbb{R}^{2}$.
Let $[0,T]$ be the time interval of interest. We denote the following function
spaces $V:=[H^{1}(\mathbb{T}^{2})]^{2}$, $M=\\{p\in
L^{2}(\mathbb{T}^{2}),\int_{\mathbb{T}^{2}}p=0\\}$,
$\mathbf{H}=\\{u\in[L^{2}(\mathbb{T}^{2})]^{2},\>\nabla\cdot u=0\\}$ and
$\mathbf{H}^{s}=\\{u\in[H^{s}(\mathbb{T}^{2})]^{2},\>\nabla\cdot u=0\\}$ with
$\|\cdot\|$ the norm on $\mathbf{H}$ and $\|\cdot\|_{s}$ the norm on
$\mathbf{H}^{s}$. We denote by $\mathcal{H}=L^{2}(0,T;\mathbf{H})$ and
$\mathcal{H}^{s}=L^{2}(0,T;\mathbf{H}^{s})$ with $|\cdot|_{0}$ the norm in
$\mathcal{H}$ and $|\cdot|_{s}$ the norm in $\mathcal{H}^{s}$ We consider the
two dimensional Navier-Stokes equation in $\mathbb{T}^{2}$

$\begin{cases}\frac{\partial u}{\partial t}+u\cdot\nabla u-\nu\Delta u+\nabla
p=f,\quad&\forall(x,t)\in\mathbb{T}^{2}\times(0,T],\\\ \nabla\cdot
u=0,\quad&\forall(x,t)\in\mathbb{T}^{2}\times(0,T],\\\ u=u_{0},\quad&\forall
x\in\mathbb{T}^{2}\text{ and }t=0,\end{cases}$ (1)

where $u$ and $p$ are the velocity and the pressure, $\nu>0$ is the viscosity.
Let $v$ be a test function in space $V$ and $r$ be a scalar test function in
space $M$. We denote by $(\cdot,\cdot)$ the inner product in the space
$L^{2}(\mathbb{T}^{2})^{2}$, extended by density to the duality pairing
between $V^{\prime}$ and $V$. We have the following mixed weak formulation of
the Navier-Stokes equation,

$\begin{cases}\text{Find $(u(t),p(t))\in V\times M$ such that}\\\
\frac{d}{dt}(u,v)+(u\cdot\nabla u,v)+(\nu\nabla u,\nabla v)-(p,\nabla\cdot
v)=(f,v),&\quad\forall v\in V,\\\ (\nabla\cdot u,r)=0,&\quad\forall r\in M,\\\
u(0,\cdot)=u_{0}.\end{cases}$ (2)

The existence and uniqueness of a solution for the above two dimensional
problem is ensured by theorem 3.1 and theorem 3.2 in [10]. If
$u_{0}\in\mathbf{H}$ and $f\in L^{2}(0,T;V^{\prime})$, problem (2) possesses a
solution $u$ which belongs to $L^{2}(0,T;V)$. However, in this paper we need
more regularity for the solution to set up and establish the well-posedness of
the Bayesian inverse problem. We thus assume more regularity for the forcing
$f$ and the initial condition $u_{0}$. In particular, we assume a divergence
free initial velocity $u_{0}\in\mathbf{H}^{1}$ and $f\in
L^{2}(0,T;\mathbf{H})$ so that $u\in C(0,T;V)$ (see lemma 5.2 in [2]). By
lemma 5.2 in [2] we also have $u\in C(0,T;\mathbf{H}^{1})\cap
L^{2}(0,T;\mathbf{H}^{2})$ which satisfies

$\|u(t)\|_{1}^{2}+\int_{0}^{t}\|u(\tau)\|_{2}^{2}d\tau\leq
C(\|u_{0}\|_{1},|f|_{0}),$ (3)

where $C(\|u_{0}\|_{1},|f|_{0})$ is a constant. The Bayesian inverse problem
is to infer the unknown initial condition and random forcing given
observations on the velocity at a set of times $\\{\tau_{k}\\}_{k=1}^{K}$.
Furthermore, to obtain FE error estimates for the Navier-Stokes equation, we
need more regularity assumptions on $u_{0}$ and $f$ (see sections 1-3 of [4]
for example). Thus in section 5, we will assume more regularity for $u_{0}$
and $f$. Now we consider $J$ tracers $\\{z_{j}\\}_{j=1}^{J}$ which are
transported by the velocity field $u$ of the Navier-Stokes equation (1). The
trajectories of the tracers are governed by the following ODE equation,

$\frac{dz_{j}}{dt}=u(z_{j},t),\qquad z_{j}(0)=z_{j,0},$ (4)

where $\\{z_{j,0}\\}_{j=1}^{J}$ is a set of initial positions. We assume that
the initial velocity $u_{0}$ and the forcing $f$ are represented in the
parametric form as

$\displaystyle u_{0}(x)$ $\displaystyle=\bar{u}_{0}(x)+\sum_{i\geq
1}\zeta_{i}\phi_{i}(x),\quad$ $\displaystyle x\in\mathbb{T}^{2},$ (5)
$\displaystyle f(t,x)$ $\displaystyle=\bar{f}(t,x)+\sum_{i\geq
1}\xi_{i}\psi_{i}(t,x),\quad$ $\displaystyle x\in\mathbb{T}^{2},\;t\in(0,T],$

where $\zeta_{i}$ and $\xi_{i}$ are normalized such that
$\zeta_{i},\xi_{i}\in[-1,1]\text{ for }i=1,2,...$ and
$\phi_{i}\in\mathbf{H}^{1}$, $\psi_{i}\in L^{2}(0,T;\mathbf{H})$. We define
the sequences $\zeta=(\zeta_{1},\zeta_{2},...)\in[-1,1]^{\mathbb{N}}$ and
$\xi=(\xi_{1},\xi_{2},...)\in[-1,1]^{\mathbb{N}}$. To indicate the dependence
of $u_{0}$ on $\zeta$ and of $f$ on $\xi$, and of $u$ on $\zeta,\xi$, we write
them as $u_{0}(x,\zeta)$, $f(t,x,\xi)$, and $u(t,x,\zeta,\xi)$.
We make the following assumptions on the decay rate of the sequences
$\\{\phi_{i}\\}$ and $\\{\psi_{i}\\}$.

######  Assumption 2.1.

The functions $\bar{u}_{0}$, $\phi_{i}$ are in $\mathbf{H}^{1}$ and $\bar{f}$,
$\psi_{i}$ are in $\mathcal{H}$. There exist constants $s>1$ and $C>0$ such
that

$\forall i\in\mathbb{N}:\quad\|\phi_{i}\|_{1}\leq
Ci^{-s},\quad|\psi_{i}|_{0}\leq Ci^{-s}.$

With assumption 2.1, there exist finite positive constants ${u_{0}}_{\max}$
and $f_{\max}$ such that

$\displaystyle\|u_{0}(\zeta)\|_{1}\leq\|\bar{u}_{0}\|_{1}+\sum_{j=1}^{\infty}\|\phi_{i}\|_{1}\leq{u_{0}}_{\max},$
$\displaystyle|f(\xi)|_{0}\leq|\bar{f}|_{0}+\sum_{j=1}^{\infty}|\psi_{i}|_{0}\leq
f_{\max}.$

From assumption 2.1, we deduce that there is a constant $C>0$ such that for
all $I\in\mathbb{N}$,

$\sum_{i>I}\|\phi_{i}\|_{1}<CI^{-q},\quad\sum_{i>I}|\psi_{i}|_{0}<CI^{-q},$
(6)

where $q=s-1$. Next, we describe the prior measure. We denote the probability
space $U=[-1,1]^{\mathbb{N}}\bigotimes[-1,1]^{\mathbb{N}}$, the set of all
pairs $(\zeta,\xi)$ of sequences $\zeta=\\{\zeta_{i}\\}_{i\geq 1}$ of
coordinates $\zeta_{i}$ and $\xi=\\{\xi_{i}\\}_{i\geq 1}$ of coordinates
$\xi_{i}$, s.t. $\zeta_{i}\in[-1,1]$ and $\xi_{i}\in[-1,1]\text{ for all }i$.
We denote the product sigma algebra on the parameter domain $U$ by
$\Theta=\left(\bigotimes_{i=1}^{\infty}\mathcal{B}([-1,1])\right)\bigotimes\left(\bigotimes_{i=1}^{\infty}\mathcal{B}([-1,1])\right)$,
where $\mathcal{B}$ is the Borel sigma algebra. Due to assumption 2.1, the
series in (5) converge in $\mathbf{H}^{1}$ and $\mathcal{H}$. Assuming that
$\zeta_{i}$ and $\xi_{i}$ are uniformly distributed in $[-1,1]$, we define the
product probability on the measurable space $(U,\Theta)$ by

$\gamma=\left(\bigotimes_{i=1}^{\infty}\frac{d\zeta_{i}}{2}\right)\bigotimes\left(\bigotimes_{i=1}^{\infty}\frac{d\xi_{i}}{2}\right),$
(7)

where $d\zeta_{i}$ and $d\xi_{i}$ denote the Lebesgue measure in $\mathbb{R}$.

##  3 Bayesian inverse problem

In this section, we present the setting for the Bayesian inverse problem. We
consider general observations of the trajectories of drifting tracers in the
two dimensional velocity field governed by the two dimensional Navier-Stokes
equation. The trajectories $\\{z_{j}\\}_{j=1}^{J}$ are observed at a set of
times $\\{\tau_{k}\\}_{k=1}^{K}\subset(0,T]$. We define the forward
observation map $\mathcal{G}:U\rightarrow[\mathbb{R}^{2}]^{JK}$ for all
$(\zeta,\xi)\in U$ as

$\mathcal{G}(\zeta,\xi):=\\{z_{j}(\tau_{k})\\}_{j,k=1}^{J,K}.$ (8)

Let $\vartheta=\\{\sigma_{j,k}\\}_{j,k=1}^{J,K}$ be the observation noise.
$\sigma_{j,k}\in\mathbb{R}^{2}$ is assumed Gaussian and independent of the
parameters $\zeta$ and $\xi$. Thus the random variable $\vartheta$ has values
in $[\mathbb{R}^{2}]^{JK}$ and follows the normal distribution $N(0,\Sigma)$,
where $\Sigma$ is a known $2JK\times 2JK$ symmetric positive covariance
matrix. The noisy observation $y$ is

$y=\mathcal{G}(\zeta,\xi)+\vartheta.$

The posterior probability measure is the conditional probability of
$(\zeta,\xi)$ in $U$ given observation $y$. We define the mismatch function

$\Phi(\zeta,\xi;y)=\frac{1}{2}\|y-\mathcal{G}(\zeta,\xi)\|_{\Sigma}^{2},$ (9)

where
$\|\cdot\|_{\Sigma}^{2}=\langle\Sigma^{-1/2}\cdot,\Sigma^{-1/2}\cdot\rangle$
with $\langle\cdot,\cdot\rangle$ being the inner product in
$[\mathbb{R}^{2}]^{JK}$.

######  Lemma 3.1.

Under assumption 2.1, the forward map
$\mathcal{G}(\zeta,\xi):U\rightarrow[\mathbb{R}^{2}]^{JK}$ is continuous as a
mapping from the measurable space $(U,\Theta)$ to
$([\mathbb{R}^{2}]^{JK},\mathcal{B}([\mathbb{R}^{2}]^{JK}))$.

###### Proof.

Following from Lemma 3.13 from [2], we have

$\displaystyle|\mathcal{G}(\zeta,\xi)-\mathcal{G}(\zeta^{\prime},\xi^{\prime})|$
$\displaystyle\leq
C(\|u_{0}(\zeta)-v_{0}(\zeta^{\prime})\|_{1}+|f(\xi)-f(\xi^{\prime})|_{0})$
$\displaystyle\leq C\left(\sum_{i\geq
1}|\zeta_{i}-\zeta_{i}^{\prime}|\|\phi_{i}\|_{1}+\sum_{i\geq
1}|\xi_{i}-\xi_{i}^{\prime}||\psi_{i}|_{0}\right)$ $\displaystyle\leq
C(\|\zeta-\zeta^{\prime}\|_{l^{\infty}(\mathbb{N})}+\|\xi-\xi^{\prime}\|_{l^{\infty}(\mathbb{N}))}.$

Hence we have the continuity of $\mathcal{G}$ with respect to $\zeta,\xi$. ∎

With the continuity with respect to the random parameters proven, we have the
following theorem on the existence of Randon-Nikodym derivative as a result of
corollary 2.2. in [2].

######  Theorem 3.2.

The posterior probability measure $\gamma^{\delta}$ is absolutely continuous
with respect to the prior $\gamma$. The Radon-Nikodym derivative is given by

$\frac{d\gamma^{y}}{d\gamma}\propto\exp(-\Phi(\zeta,\xi;y))$ (10)

Next we consider the continuity of the posterior measure in the Hellinger
distance with respect to the observation data, which implies the well-
posedness of the posterior measure. The Hellinger distance is defined as

$d_{Hell}(\gamma^{\prime},\gamma^{\prime\prime})=\left(\frac{1}{2}\int_{U}(\sqrt{\frac{d\gamma^{\prime}}{d\gamma}}-\sqrt{\frac{d\gamma^{\prime\prime}}{d\gamma}})d\gamma\right)^{1/2},$
(11)

where $\gamma^{\prime}$ and $\gamma^{\prime\prime}$ are two measures on $U$,
which are absolutely continuous with respect to the measure $\lambda$. It is
shown in [7] that the Lipschitzness of the posterior measure with respect to
the Hellinger distance holds under general conditions.

######  Proposition 3.3.

The measure $\gamma^{y}$ depends locally Lipschitz continuously on the data
$y$ with respect to the Hellinger metric: for every $r>0$ and
$y,y^{\prime}\in\mathbb{R}^{d}$ such that for
$|y|_{\Sigma},|y^{\prime}|_{\Sigma}\leq r$, there exists $C=C(r)>0$ such that

$d_{Hell}(\gamma^{y},\gamma^{y^{\prime}})\leq C(r)|y-y^{\prime}|_{\Sigma}.$
(12)

The proof of the proposition is similar to the one for Proposition 3.3. in
[11].

##  4 Posterior approximation by finite truncation of the forcing and the
initial condition

We consider the approximation of the forward equation by truncating the series
expansion (5) for the forcing and the initial velocity after $I$ terms. Let

$u_{0}^{I}(x,\zeta)=\bar{u}_{0}(x)+\sum_{i=1}^{I}\phi_{i}(x)\zeta_{i},\quad
f^{I}(t,x,\xi)=\bar{f}(t,x)+\sum_{i=1}^{I}\psi_{i}(t,x)\xi_{i}.$ (13)

We consider the truncated problem,

$\begin{cases}\text{Find }u^{I}(t)\in V,p^{I}(t)\in M\text{ such that }\\\
\frac{d}{dt}(u^{I},v)+(u^{I}\cdot\nabla u^{I},v)+(\nu\nabla u^{I},\nabla
v)-(p^{I},\nabla\cdot v)=(f^{I},v),&\quad\forall v\in V,\\\ (\nabla\cdot
u^{I},r)=0,&\quad\forall r\in M,\\\ u^{I}(0,\cdot)=u_{0}^{I},\end{cases}$ (14)

with the tracer trajectory

$\frac{dz_{j}^{I}}{dt}=u(z_{j}^{I},t),\qquad z_{j}^{I}(0)=z_{j,0}.$ (15)

######  Proposition 4.1.

Under assumption 2.1, the truncated forward map
$\mathcal{G}^{I}(\zeta,\xi):=\\{z_{j}^{I}(\tau_{k})\\}_{j,k=1}^{J,K}$
satisfies the estimate,

$|\mathcal{G}(\zeta,\xi)-\mathcal{G}^{I}(\zeta,\xi)|\leq CI^{-q}.$ (16)

###### Proof.

With $u_{0}\in\mathbf{H}^{1}$ and $f\in\mathcal{H}^{1}$, we have

$|\mathcal{G}(\zeta,\xi)-\mathcal{G}^{I}(\zeta,\xi)|\leq
C(\|u_{0}\|_{1},\|u_{0}^{I}\|_{1},|f|_{1},|f^{I}|_{1})(\|u_{0}-u_{0}^{I}\|_{s}^{2}+|f-f^{I}|_{0}^{2})^{\frac{1}{2}}.$

The preceding conclusion is by Lemma 3.13 in [2]. Due to assumption 2.1 and
$\|u_{0}-u_{0}^{I}\|_{1}^{2}\leq CI^{-2q},|f-f^{I}|_{0}^{2}\leq CI^{-2q}$.
Hence, we conclude that

$|\mathcal{G}(\zeta,\xi)-\mathcal{G}^{I}(\zeta,\xi)|\leq CI^{-q},$

where $C$ is a constant uniformly bounded with respect to $(\zeta,\xi)\in U$.
∎

Then we define the approximated posterior measure $\gamma^{I,y}$ as,

$\frac{d\gamma^{I,y}}{d\gamma}\propto\exp(-\Phi^{I}(\zeta,\xi;y)),$

where $\Phi^{I}(\zeta,\xi;y)$ is the potential function

$\Phi^{I}(\zeta,\xi;y)=\frac{1}{2}|y-\mathcal{G}^{I}(\zeta,\xi)|_{\Sigma}^{2}.$
(17)

The measure $\gamma^{I,y}$ is an approximation of the Bayesian posterior. Next
we show the error estimate for the approximation of the posterior measure by
the solution of the truncated equation in the Hellinger metric.

######  Proposition 4.2.

Under assumption 2.1 and proposition 4.1, there is a constant $C(y)>0$ such
that for every $I$

$d_{Hell}(\gamma^{y},\gamma^{I,y})\leq C(y)I^{-q}.$ (18)

##  5 FE Approximation of the truncated problem

We describe the FE approximation of solution $u^{I},p^{I}$ of (14) with the
truncated forcing and initial condition in (13). In the two dimensional
periodic unit torus $\mathbb{T}^{2}$, we define the following nested family
$\\{\mathcal{T}^{l}\\}_{l=0}^{\infty}$ of simplicial partition of
$\mathbb{T}^{2}$. The domain $\mathbb{T}^{2}$ is first subdivided into a
regular family $\mathcal{T}^{0}$ of simplices $T$ which are periodically
distributed; then for $l\geq 1$, each simplex in $\mathcal{T}^{l}$ is obtained
by subdividing each simplex in $\mathcal{T}^{l-1}$ into $4$ congruent
triangles. Hence the mesh size
$h_{l}=\max\\{\text{diam($T$)}:T\in\mathcal{T}^{l}\\}$ of $\mathcal{T}^{l}$ is
$h_{l}=2^{-l}h_{0}$. We define the following nested hierarchical family of
spaces of $\mathbb{P}_{1}$-iso-$\mathbb{P}_{2}/\mathbb{P}_{1}$ FE spaces on
$\mathcal{T}^{l}$ as

$\displaystyle V^{l}$ $\displaystyle=\\{u\in
V:u|_{T}\in[\mathcal{P}_{1}(T)]^{2}\quad\forall T\in\mathcal{T}^{l+1}\\},$
$\displaystyle M^{l}$ $\displaystyle=\\{p\in
M:p|_{T}\in\mathcal{P}_{1}(T)\quad\forall T\in\mathcal{T}^{l}\\},$

where $\mathcal{P}_{1}(T)$ denotes the set of linear functions in the simplex
$T\in\mathcal{T}^{l}\text{ or }T\in\mathcal{T}^{l+1}$. With the FE
approximation space defined, we consider the FE approximation of the truncated
problem

$\begin{cases}\text{Seek }u^{I,l}(t)\in V^{l}\text{ and }p^{I,l}(t)\in
M^{l}\text{ such that}\\\ (\frac{du^{I,l}}{dt},v^{l})+(\nu\nabla
u^{I,l},\nabla v^{l})+((u^{I,l}\cdot\nabla)u^{I,l},v^{l})+(\nabla
p,v^{l})=(f^{I},v^{l}),&\forall v^{l}\in V^{l},\\\ (\nabla\cdot
u^{I,l},r^{l})=0,&\forall r^{l}\in M^{l},\\\ \end{cases}$ (19)

with $I,l\in\mathbb{N}$. We choose simplicial partition with
$\mathbb{P}_{1}$-iso-$\mathbb{P}_{2}/\mathbb{P}_{1}$ FE spaces in the
numerical analysis, but the analysis is also valid for nested hyperrectangular
partition with $\mathbb{Q}_{1}$-iso-$\mathbb{Q}_{2}/\mathbb{Q}_{1}$ FE pair.
The wellposedness of the two dimensional problem for approximating the Navier-
Stokes equation with $\mathbb{P}_{1}$-iso-$\mathbb{P}_{2}/\mathbb{P}_{1}$
element is well known (see [10, 4]). More references can be found in [1, 3,
8]. To solve the time continuous FE approximation problem, we consider the
Implicit/Explicit (IMEX) Euler scheme (see [4]). However, other time schemes
can be employed, e.g. implicit Euler and implicit Crank-Nicholson schemes [5,
8]. In the multi-level setup, we recursively bisect the time discretization at
level $l$ to get time discretization at level $l+1$. For $n=1,2,...,2^{l}$, we
define $t_{n}=n\Delta t_{l}$, where $\Delta t_{l}=T2^{-l}$. We denote by
$u^{I,l,n},p^{I,l,n}$ the solution at time step $t_{n}$. Hence under the
implicit/explicit Euler scheme, we solve the following saddle point problem
for each time step.

$\begin{cases}\text{Seek }u^{I,l,n+1}\in V^{l}\text{ and }p^{I,l,n+1}\in
M^{l}\text{ such that}\\\ (\frac{u^{I,l,n+1}}{\Delta t},v^{l})+(\nu\nabla
u^{I,l,n+1},\nabla v^{l})+((u^{I,l,n}\cdot\nabla)u^{I,l,n+1},v^{l})\\\
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad+(\nabla
p^{I,l,n+1},v^{l})=(f^{J}+\frac{u^{I,l,n}}{\Delta t},v^{l}),&\forall v^{l}\in
V^{l},\\\ (\nabla\cdot u^{I,l,n+1},r^{l})=0,&\forall r^{l}\in
M^{l}.\end{cases}$ (20)

Let the operator $a(u^{I,l,n};u^{I,l,n+1},v^{l})=(\frac{u^{I,l,n+1}}{\Delta
t},v^{l})+(\nu\nabla u^{I,l,n+1},\nabla
v^{l})+((u^{I,l,n}\cdot\nabla)u^{I,l,n+1},v^{l})$ and operator
$b(u^{I,l,n+1},r^{l})=-(\nabla\cdot u^{I,l,n+1},r^{l})$. We have the following
saddle point problem,

$\begin{cases}\text{Seek }u^{I,l,n+1}\in V^{l}\text{ and }p^{I,l,n+1}\in
M^{l}\text{ such that}\\\
a(u^{I,l,n};u^{I,l,n+1},v^{l})+b(v^{l},p^{I,l,n+1})=(f^{J}+\frac{u^{I,l,n}}{\Delta
t},v^{l}),&\forall v^{l}\in V^{l}\\\ b(u^{I,l,n+1},r^{l})=0,&\forall r^{l}\in
M^{l}.\end{cases}$ (21)

To solve Stokes like equations, FE spaces have to satisfy the inf-sup
condition uniformly with respect to $l$. Ern and Guermond [1] provide a
comprehensive list of FE spaces that satisfy the inf-sup condition. Bubble
element, Taylor-Hood element $(\mathbb{P}_{2}/\mathbb{P}_{1})$ and
$\mathbb{P}_{1}$-iso-$\mathbb{P}_{2}/\mathbb{P}_{1}$ element are some commonly
known elements that satisfy the inf-sup conditions and widely adopted for
solving fluid mechanics problems. More detailed error analysis of the
implicit/explicit Euler scheme can be found in He’s paper [4]; and analysis
for second order time scheme can be found in [5]. We choose the
implicit/explicit Euler scheme in our analysis. To further estimate the FE
approximation error with the implicit/explicit Euler scheme, we make the
following regularity assumption.

######  Assumption 5.1.

We assume $\bar{u}_{0}\in D(A)$, $\bar{f},\bar{f}_{t},\bar{f}_{tt}\in
L^{\infty}(0,T;[L^{2}(\mathbb{T})]^{2})$ , and $\phi_{i}\in D(A)$ and
$\psi^{i},\psi^{i}_{t},\psi^{i}_{tt}\in
L^{\infty}(0,T;[L^{2}(\mathbb{T})]^{2})$ for all $i\in\mathbb{N}$ such that

$\|A\bar{u}_{0}\|_{L^{2}(D)}+\sum_{i\geq 1}\|A\phi_{i}\|_{L^{2}(D)}\leq C,$
(22)

and

$\displaystyle\sup_{0\leq t\leq T}\\{\|\bar{f}(t)$
$\displaystyle\|_{L^{2}(\mathbb{T}^{2})}+\|\bar{f}_{t}(t)\|_{L^{2}(\mathbb{T}^{2})}+\|\bar{f}_{tt}(t)\|_{L^{2}(\mathbb{T}^{2})}\\}$
(23) $\displaystyle+\sup_{0\leq t\leq T}\\{\sum_{i\geq
1}(\|\psi^{i}(t)\|_{L^{2}(\mathbb{T}^{2})}+\|\psi^{i}_{t}(t)\|_{L^{2}(\mathbb{T}^{2})}+\|\psi^{i}_{tt}(t)\|_{L^{2}(\mathbb{T}^{2})})\\}\leq
C,$

where $C$ is a positive constant and $A=-P\Delta$ is the Stokes operator; $P$
is the orthogonal projection from $L^{2}\rightarrow\mathbf{H}$.

With assumption 5.1, we deduce

$\|Au_{0}\|_{L^{2}(\mathbb{T}^{2})}+\sup_{0\leq t\leq
T}\\{\|f(t)\|_{L^{2}(\mathbb{T}^{2})}+\|f_{t}(t)\|_{L^{2}(\mathbb{T}^{2})}+\|f_{tt}(t)\|_{L^{2}(\mathbb{T}^{2})}\\}\leq
C.$ (24)

With the regularity assumption, we have the following error estimate for the
fully discretized Navier-Stokes equation with Euler implicit/explict method
from He [4]. There are two positive constants $K_{1}$ and $K_{2}$ such that,

$\displaystyle\left\|u^{J}(t_{n})-u^{J,l,n}\right\|_{V}\leq
K_{1}\left(h_{l}+\Delta t_{l}\right),$ (25)
$\displaystyle\left\|p^{J}(t_{n})-p^{J,l,n}\right\|_{L^{2}(\mathbb{T}^{2})}\leq
K_{2}\left(h_{l}+\Delta t_{l}\right),$

where $h_{l}=O(2^{-l})$ and $\Delta t_{l}=O(2^{-l})$ for all $n=1,...,2^{l}$.
In particular $K_{1}$ and $K_{2}$ are independent from $\zeta$ and $\xi$.
Hence we have the following error bound.

######  Proposition 5.2.

Consider the FE approximation of the truncated mixed problem, with the error
estimate (25). There is a constant $C>0$ such that for every
$J,l\in\mathbb{N}$ and for every $(\zeta,\xi)\in U$, the following error bound
holds

$\displaystyle\|u(t_{n},\cdot,\zeta,\xi)-u^{J,l,n}(\cdot,\zeta,\xi)\|_{V}$
$\displaystyle\leq C(2^{-l}+J^{-q}).$ (26)

###### Proof.

The proposition can be proved by considering the truncation error and the FE
approximation error,

$\displaystyle\|u(t_{n},\cdot,\zeta,\xi)$
$\displaystyle-u^{J,l,n}(\cdot,\zeta,\xi)\|_{V}$
$\displaystyle\leq\|u(t_{n},\cdot,\zeta,\xi)-u^{J}(t_{n},\cdot,\zeta,\xi)\|_{V}+\|u^{J}(t_{n},\cdot,\zeta,\xi)-u^{J,l,n}(\cdot,\zeta,\xi)\|_{V}$
$\displaystyle\leq C(2^{-l}+J^{-q}).$

∎

To solve the ODE equation of the tracer trajectories, we use the backward
Euler method, which leads to the following discretized equation,

$z_{j}^{n+1}=z_{j}^{n}+\Delta t\cdot u^{I,l,n+1}(z_{j}^{n}),$ (27)

where $z_{j}^{0}=z_{j}(t=0)$ and $\Delta t$ is the time step size.

######  Lemma 5.3.

$|z_{j}(t_{n})-z_{j}^{n}|\leq C(t_{n})\cdot\Delta t$ (28)

###### Proof.

The euler method is given as in (27). We consider the following Taylor
expansion,

$z_{j}(t_{n+1})=z_{j}(t_{n})+\Delta t\cdot z_{j}^{\prime}(t_{n})+(\Delta
t)^{2}\cdot z_{j}^{\prime\prime}(t_{n})+....$

where we have $z_{j}^{\prime\prime}(t_{n})=\frac{\partial u}{\partial
t}(t_{n},z_{j}(t_{n}))+\frac{\partial u}{\partial
z_{j}}(t_{n},z_{j}(t_{n}))u(t_{n},z_{j}(t_{n}))$ hence
$|z_{j}^{\prime\prime}(t_{n})|$ is bounded. Now we have the Taylor
approximation,

$z_{j}(t_{n+1})=z_{j}(t_{n})+\Delta t\cdot
u(t_{n},z_{j}(t_{n}))+\mathcal{O}((\Delta t)^{2}).$ (29)

Let $e_{n}=z_{j}(t_{n+1})-z_{j}^{n+1}$. By comparing equation (27) and
equation (29), we have

$\displaystyle e_{n+1}$ $\displaystyle=e_{n}+\Delta
t\cdot(u_{h}(t_{n},z_{j}^{n})-u(t_{n},z_{j}(t_{n})))+\mathcal{O}((\Delta
t)^{2})$ $\displaystyle=e_{n}+\Delta
t\cdot(u_{h}(t_{n},z_{j}^{n})-u(t_{n},z_{j}^{n})+u(t_{n},z_{j}^{n})-u(t_{n},z_{j}(t_{n})))+\mathcal{O}((\Delta
t)^{2})$ $\displaystyle=e_{n}+\Delta
t\cdot(u_{h}(t_{n},z_{n})-u(t_{n},z_{n}))+\Delta
t\cdot(u(t_{n},z(t_{n})+e_{n})-u(t_{n},z(t_{n})))+\mathcal{O}((\Delta t)^{2})$
$\displaystyle\leq(1+\Delta t\cdot\lambda)|e_{n}|+\Delta
t\cdot|u_{h}(t_{n},z_{n})-u(t_{n},z_{n})|_{L_{\infty}}+c(\Delta t)^{2}$
$\displaystyle\leq(1+\Delta t\cdot\lambda)|e_{n}|+\Delta
t\cdot|u_{h}(t_{n},z_{n})-u(t_{n},z_{n})|_{1+s}+c(\Delta t)^{2}$

From this we can have the following,

$\displaystyle|e_{n+1}|\leq(1+\Delta t\cdot\lambda)|e_{n}|+c(\Delta t)^{2}.$

with $h$ being linearly proportional to $\Delta t$. From deduction, we can get

$|e_{n}|\leq\frac{c}{\lambda}h((1+h\lambda)^{n}-1),$ (30)

and subsquently we have

$|e_{n}|\leq\frac{c}{\lambda}(e^{\lambda t_{n}}-1)\cdot\Delta t.$ (31)

∎

##  6 Multilevel MCMC

We follow the Multilevel Markov Chain Monte Carlo method first developed in
[7] for elliptic equation and subsequently applied to the Navier Stokes
equations with Eulerian observations in [12]. We denote the posterior
expectation of $\ell(u(t,\cdot,\zeta,\xi))$, where $\ell$, where $\ell$ is a
bounded linear map on $V$. Similar to the setup in the above mentioned work,
we choose $J$ s.t. $J^{-q}=\mathcal{O}(2^{-l})$ and denote $u^{J_{L},l}$ as
$u^{l}$, $\theta^{J_{l},l}$ as $\gamma^{J_{l},l,\delta}$ as $\gamma^{l}$. We
also have the hierarchy of approximations $\gamma^{l}$ of the posterior
measure $\gamma^{\delta}$. The MLMCMC estiamtor $E_{L}^{MLMCMC}[\ell(u]$ of
$\mathbb{E}[\ell(u]$ is

$\displaystyle E_{L}^{MLMCMC}[\ell(u)]$ (32)
$\displaystyle=\sum_{l=1}^{L}\sum_{l^{\prime}=1}^{L^{\prime}(l)}E_{M_{ll^{\prime}}}^{\gamma^{l}}[(1-\exp(\Phi^{l}(\cdot,\cdot;\delta)-\Phi^{l-1}(\cdot,\cdot;\delta)))(\ell(u^{l^{\prime}})-\ell(u^{l^{\prime}-1}))]$
$\displaystyle+\sum_{l=1}^{L}\sum_{l^{\prime}=1}^{L^{\prime}(l)}E_{M_{ll^{\prime}}}^{\gamma^{l}}[\exp(\Phi^{l}(\cdot,\cdot;\delta)-\Phi^{l-1}(\cdot,\cdot;\delta))-1]\cdot
E_{M_{ll^{\prime}}}^{\gamma^{l-1}}[\ell(u^{l^{\prime}})-\ell(u^{l^{\prime}-1})]$
$\displaystyle+\sum_{l=1}^{L}E_{M_{l0}}^{\gamma^{l}}[(1-\exp(\Phi^{l}(\cdot,\cdot;\delta)-\Phi^{l-1}(\cdot,\cdot;\delta)))(\ell(u^{0}))]$
$\displaystyle+\sum_{l=1}^{L}E_{M_{l0}}^{\gamma^{l}}[\exp(\Phi^{l}(\cdot,\cdot;\delta)-\Phi^{l-1}(\cdot,\cdot;\delta))-1]\cdot
E_{M_{l0}}^{\gamma^{l-1}}[\ell(u^{0})]$
$\displaystyle+\sum_{l^{\prime}=1}^{L^{\prime}(0)}E_{M_{0l^{\prime}}}^{\gamma^{0}}[\ell(u^{l^{\prime}})-\ell(u^{l^{\prime}-1})]+E_{M_{00}}^{\gamma^{0}}[\ell(u^{0})]$

where $L^{\prime}(l)$ is chosen below; and $E_{M_{ll^{\prime}}}^{\gamma^{l}}$
denotes the MCMC sample average of the Markov chain generated by MCMC sampling
procedure with the acceptance probability

$\alpha^{l}((\zeta,\xi),(\zeta^{\prime},\xi^{\prime}))=1\wedge\exp(\Phi^{l}(\zeta,\xi;\delta)-\Phi^{l}(\zeta^{\prime},\xi^{\prime};\delta)),\quad(\zeta,\xi),(\zeta^{\prime},\xi^{\prime})\in
U,$

for the independence sampler and the pCN sampler we employ in the numerical
implementation, with $M_{ll^{\prime}}$ samples.

With the following multi-level sampling numbers,

$L^{\prime}(l):=L-l,\quad\text{ and }\quad
M_{ll^{\prime}}:=2^{2(L-(l+l^{\prime}))},$ (33)

we have the following error estimate,

$\mathbf{E}[|\mathbb{E}^{\gamma^{\delta}}[\ell(u)]-{E}_{L}^{MLMCMC}[\ell(u)]|]\leq
C(\delta)L^{2}2^{-L},$ (34)

where $\mathbf{E}$ denotes the expectation in the probability space of all the
Markov chains in the MLMCMC sampling procedure. To reduce the effect of the
$L^{2}$ multiplying factor in (34), we can slightly enlarge the sample size
$M_{ll^{\prime}}$ as

$M_{ll^{\prime}}=(l+l^{\prime})^{a}2^{2(L-(l+l^{\prime}))}$ (35)

for $a>0$. We have the following result from [6].

Table 1: Total MLMCMC error with different sample size choices for uniform prior $a$ | $M_{ll^{\prime}},l,l^{\prime}>1$ | $M_{l0}=M_{0l}$ | $M_{00}$ | Total error
---|---|---|---|---
0 | $2^{2\left(L-\left(l+l^{\prime}\right)\right)}$ | $2^{2(L-l)}/L^{2}$ | $2^{2L}/L^{4}$ | $O\left(L^{2}2^{-L}\right)$
2 | $\left(l+l^{\prime}\right)^{2}2^{2\left(L-\left(l+l^{\prime}\right)\right)}$ | $2^{2(L-l)}$ | $2^{2L}/L^{2}$ | $O\left(L\log L2^{-L}\right)$
3 | $\left(l+l^{\prime}\right)^{3}2^{2\left(L-\left(l+l^{\prime}\right)\right)}$ | $l2^{2(L-l)}$ | $2^{2L}/L$ | $O\left(L^{1/2}2^{-L}\right)$
4 | $\left(l+l^{\prime}\right)^{4}2^{2\left(L-\left(l+l^{\prime}\right)\right)}$ | $l^{2}2^{2(L-l)}$ | $2^{2L}/\left(\log L^{2}\right)$ | $O\left(\log L2^{-L}\right)$

The number of degrees of freedom for each time step is $O(2^{2l})$ for the FE
resolution mesh size $O(2^{-l})$. For $O(2^{l})$ time steps, the total number
of degrees of freedom required for the forward solver is $O(2^{3l})$. Thus the
total number of degrees of freedom required for computing the MLMCMC estimator
with the finest resolution level $O(2^{-L})$ is

$\lesssim\sum_{l=1}^{L}\sum_{l^{\prime}=1}^{L-l}(l+l^{\prime})^{a}2^{2(L-(l+l^{\prime}))}(2^{3l}+2^{3l^{\prime}})+\sum_{l=1}^{L}l^{a}2^{2(L-l)}2^{3l}+\sum_{l^{\prime}=1}^{L}{l^{\prime}}^{a}2^{3l^{\prime}}+2^{2L}\lesssim
L^{a}2^{3L}.$ (36)

##  7 Numerical experiments

In this section we implement the FE-MLMCMC method for approximating quantities
of interest for Bayesian inverse problems for Navier-Stokes equation analyzed
in the preceding sections. We employ the
$\mathbb{Q}_{1}$-iso-$\mathbb{Q}_{2}/\mathbb{Q}_{1}$ elements and Euler
implicit/explicit time scheme for solving the forward Navier-Stokes equation.
By lemma 4.27 of [1], the
$\mathbb{Q}_{1}$-iso-$\mathbb{Q}_{2}$/$\mathbb{Q}_{1}$ elements satisfy the
inf-sup condition. At each time step we solve a saddle point system. We use
the iterative FGMRES method with Schur complement precondition to solve the
linear system efficiently. To illustrate the theoretical result, we consider
the case where the forcing depends on one random variable as in this case we
can compute a reference posterior expectation highly accurately, though we
stress that the numerical method works for the case where the forcing and the
initial condition depend on many random variables as shown theoretically
above. In the first experiment, we consider the inverse problem of Navier-
Stokes equation in the $(0,1)\times(0,1)$ square domain with the periodic
boundary condition. For $\xi\sim U(0,1.0)$, we consider the following model
problem

$\begin{cases}\frac{\partial u}{\partial t}+u\cdot\nabla u-\nu\Delta u+\nabla
p=f,&\quad\text{for }x\in\mathbb{T}^{2},\\\ \nabla\cdot u=0,&\quad\text{for
}x\in\mathbb{T}^{2},\\\ u(x,0)=0;\end{cases}$ (37)

with the periodic boundary condition and the random forcing

$f=\begin{cases}\xi(\cos(2\pi x_{1})\sin(2\pi x_{2})+1.0)\exp(t),\\\
-\xi(\sin(2\pi x_{1})\cos(2\pi x_{2})+1.0)\exp(t).\end{cases}$ (38)

The forward observation functional is
$\\{z_{i}(t=0.5)\\}_{i=1}^{5},\\{z_{i}(t=1.0)\\}_{i=1}^{5}$. The quantity of
interest is

$100\int_{D}\left(x_{1}^{1/2}x_{2}^{1/2}\frac{\partial u_{1}(x,1.0)}{\partial
x_{2}}-x_{1}^{1/2}x_{2}^{1/2}\frac{\partial u_{2}(x,1.0)}{\partial
x_{1}}\right)dx.$ (39)

With the Gaussian prior, we generate a random realization of the solution by
solving the forward problem with a randomly generated $\xi$. Then a random
observation is obtained with a randomly generated Gaussian noise from
$N(0,1.0)$ to the forward functional. The observed tracer location is
$z_{1}=\\{(0.640,0.795),(0.172,0.195),\\\
(1.134,0.079),(0.173,-0.027),(0.846,0.411)\\}$ and
$z_{2}=\\{(1.112,0.323),(0.659,-0.303),(1.628,-0.429),\\\
(0.656,-0.524),(1.353,-0.089)\\}$. The initial tracer locations are
$\\{(0.491,0.908),(0.012,0.312),(0.975,0.203),\\\
(0.035,0.111),(0.725,0.575)\\}$. The reference posterior expectation of the
quantity of interest is $-1.0924$ which is computed by Gauss-Legendre
quadrature with the highly accurate Fourier spectral forward solver with a
total of $128\times 128$ spectral points and 4th order accurate Runge-Kutta
explicit time stepping with 0.0001 time step size. Detailed implementation of
the Fourier spectral method for Navier-Stokes equation can be found in [9].

L | 1 | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---|---
error (a=2) | 0.663275 | 0.338971 | 0.252836 | 0.1122 | 0.062769 | 0.031369
error (a=3) | 0.663275 | 0.397023 | 0.190823 | 0.078067 | 0.032926 |
Table 2: Error Table Figure 1: MLMCMC error for 2D Navier-Stokes equation with
uniform prior, a=2 Figure 2: MLMCMC error for 2D Navier-Stokes equation with
uniform prior, a=3

## References

  * [1] A. Ern and J.L. Guermond, Theory and Practice of Finite Elements, Springer, 2004.
  * [2] S. L. Cotter, M. Dashti, J. C. Robinson, and A. M. Stuart, Bayesian inverse problems for functions and applications to fluid mechanics, Inverse Problems, (2009).
  * [3] V. Girault and P.-A. Raviart, Finite Element Methods for Navier-Stokes Equations, Springer, 1986.
  * [4] Y. He, The Euler implicit/explicit scheme for the 2D time-dependent Navier-Stokes equations with smooth or non-smooth initial data, Mathematics of Computation, 77 (2008), pp. 2097–2124.
  * [5] J. G. Heywood and R. Rannacher, Finite-element approximation of the nonstationary navier-stokes problem part IV. Error analysis for second-order time discretization, SIAM Journal on Numerical Analysis, 27 (1990), pp. 353–384.
  * [6] V. H. Hoang, J. H. Quek, and C. Schwab, Analysis of a multilevel Markov chain Monte Carlo finite element method for Bayesian inversion of log-normal diffusions, Inverse Problems, 36 (2020), pp. 0–46.
  * [7] V. H. Hoang, C. Schwab, and A. M. Stuart, Complexity analysis of accelerated MCMC methods for Bayesian inversion, Inverse Problems, 29 (2013).
  * [8] V. John, Finite Element Methods for Incompressible Flow Problems, vol. 51, Springer, 2016.
  * [9] R. Peyret, Spectral Methods for Incompressible Flows, Springer, 2002\.
  * [10] R. Temam, Navier-Stokes equations: theory and numerical analysis., North-Holland Publishing Company, 1978.
  * [11] J. Yang and V. H. Hoang, Multilevel markov chain monte carlo for bayesian inverse problem for navier-stokes equation, Inverse Problems and Imaging, 0 (2022), pp. –.
  * [12] J. Yang and V. H. Hoang, Multilevel Markov chain Monte Carlo for Bayesian inverse problem for Navier-Stokes equation, Inverse Probl. Imaging, 17 (2023), pp. 106–135.


# See More Details: Efficient Image Super-Resolution by Experts Mining

Eduard Zamfir  Zongwei Wu  Nancy Mehta  Yulun Zhang  Radu Timofte

# See More Details: Efficient Image Super-Resolution by Experts Mining
– Appendix –

Eduard Zamfir  Zongwei Wu  Nancy Mehta  Yulun Zhang  Radu Timofte

###### Abstract

Reconstructing high-resolution (HR) images from low-resolution (LR) inputs
poses a significant challenge in image super-resolution (SR). While recent
approaches have demonstrated the efficacy of intricate operations customized
for various objectives, the straightforward stacking of these disparate
operations can result in a substantial computational burden, hampering their
practical utility. In response, we introduce SeemoRe, an efficient SR model
employing expert mining. Our approach strategically incorporates experts at
different levels, adopting a collaborative methodology. At the macro scale,
our experts address rank-wise and spatial-wise informative features, providing
a holistic understanding. Subsequently, the model delves into the subtleties
of rank choice by leveraging a mixture of low-rank experts. By tapping into
experts specialized in distinct key factors crucial for accurate SR, our model
excels in uncovering intricate intra-feature details. This collaborative
approach is reminiscent of the concept of “see more”, allowing our model to
achieve an optimal performance with minimal computational costs in efficient
settings. The source will be publicly made available at
https://github.com/eduardzamfir/seemoredetails

Mixture of Experts, Efficiency, Image Super-Resolution

##  1 Introduction

$0$$20$$40$$60$$80$$100$$120$$140$$160$$180$$200$$220$$240$$260$$280$$38.7$$38.8$$38.9$$39$$39.1$$39.2$$39.3$$39.4$$39.5$+$0.18$dB-$51\%$+$0.16$dB-$17\%$
better CNN Transformers ShuffleMixerIMDNPANDDistill-SRSAFMNSeemoRe-TSeemoRe-
BSeemoRe-LSRFormerSwinIRGMACSPSNR (dB)$\times 2$ SR on Manga109 Figure 1:
Model complexity trade-off. Visualization of PSNR, GMACS, and parameter counts
on Manga109 dataset for $\times$2 task. Our proposed SeemoRe excels the state-
of-the-art CNN-based and lightweight Transformer-based SR models. Marker size
indicates parameter counts w.r.t SwinIR-Light (Liu et al., 2021).

Single image super-resolution (SR) is a long-standing low-level vision
endeavour that pursues the reconstruction of a high-resolution (HR) image from
its degraded low-resolution (LR) counterpart. This challenging task has
garnered considerable attention owing to the expeditious development of ultra-
high definition devices and video streaming applications (Khani et al., 2021;
Zhang et al., 2021a). Foreseeing the resource constraints, it is of
substantial desire to design an efficient SR model for gauging the HR images
to be perfectly visualized on these devices or platforms. Identifying the most
plausible candidates for missing HR pixels poses a particular challenge for
SR. In the absence of external priors, the primary approaches for SR involves
exploring the intricate relationships among the neighboring pixels for
reconstruction. Recent SR models exemplify this through methods such as (a)
attention (Liang et al., 2021; Zhou et al., 2023; Chen et al., 2023), (b)
feature mixing (Hou et al., 2022; Sun et al., 2023), and (c) global-local
context modeling (Wang et al., 2023; Sun et al., 2022), yielding remarkable
accuracy.

Unlike other approaches in this work, we aim to avoid complex and disconnected
blocks focusing on specific factors, opting instead for a unified learning
module specialized for all aspects. However, an additional challenge arises
due to the efficiency requirement, rendering implicit learning through a vast
number of parameters unfeasible, especially in the context of devices with
limited resources.

To achieve such an efficient unification, we introduce SeemoRe, which
leverages the synergy of different experts to maximize intra-feature
intertwining, collaboratively learning a cohesive relation across LR pixels.
Our motivation stems from the observation that image features often display
diverse patterns and structures. Attempting to capture and model all these
patterns with a single, monolithic model can be challenging. Collaborative
experts, on the other hand, enable the network to specialize in different
regions or aspects of the input space, enhancing its adaptability to various
patterns and facilitating the modeling of LR-HR dependencies, akin to “See
More”.

Technically, our network is composed of stacked residual groups (RGs) for
dynamically selecting the pivotal features via experts, focusing on two
different aspects. At the macro level, each RG embodies two successive expert
blocks: (a) Rank modulating expert (RME), expertized in dealing with the most
informative features through low-rank modulation, and (b) Spatial modulating
expert (SME), expertized in efficient spatial enhancement. At the micro level,
we devise a Mixture of Low-Rank Expertise (MoRE) as the foundational component
within RME to dynamically select the best and most suitable rank for different
inputs and at different network depths while implicitly modeling the global
contextual relationships. Furthermore, we design a Spatial Enhancement
Expertise (SEE) as an efficient alternative to complex self-attention within
SME for distinctly improving the spatial-wise local aggregation capabilities.
Such a combination efficiently modulates the mutual dependencies within the
feature attributes, enabling our model to extract high-level information,
which is a key aspect of SR. By explicitly mining experts at different
granularity for different expertise, our network navigates the intricacies
between spatial and channel features, maximizing their synergistic
contribution and thus accurately and efficiently reconstructing more details.

As shown in Figure 1, our network significantly outperforms the state-of-the-
art (SOTA) efficient models such as DDistill-SR (Wang et al., 2022) or SAFMN
(Sun et al., 2023) by a considerable margin, while utilizing only half or even
less of the GMACS. Although our model is specifically designed for efficient
SR, its scalability is evident as our larger model surpasses the SOTA
lightweight transformer in performance while incurring lower computational
costs. Overall, our key contributions are threefold:

  * •

We propose SeemoRe which matches the versatility of Transformer-based methods
and the efficiency of CNN-based methods.

  * •

A Rank modulating expert (RME) is proposed to probe into the intricate inter-
dependencies among the relevant feature projections in an efficient manner.

  * •

A Spatial modulating expert (SME) is proposed to integrate the complementary
features extracted by SME by encoding the local contextual information.

##  2 Related Works

CNN-based SR. In recent years, CNN-based techniques have outperformed
traditional interpolation algorithms (Duchon, 1979) by learning a non-linear
mapping between the input and target in an end-to-end training manner. The
seminal SRCNN (Dong et al., 2014) introduced a three-layer convolutional
approach for image super-resolution, later extended by works such as (Lim et
al., 2017; Zhang et al., 2018b; Hui et al., 2019; Liang et al., 2021). VDSR
(Kim et al., 2016) and EDSR (Lim et al., 2017) deepen networks using residual
learning principles, with EDSR streamlining residual blocks for deeper
training. Conversely, RCAN (Zhang et al., 2018a) introduces a novel residual-
in-residual architecture for models exceeding 400 layers. While various
spatial and channel attention mechanisms aim to enhance image reconstruction
quality, CNN-based techniques still struggle to effectively utilize shared
information across both dimensions. In this work, we aim to explore the
interdependencies among the features in a computationally efficient way.

Transformer-based SR. Thanks to its remarkable performance in high-level tasks
(Dosovitskiy et al., 2021), the Transformer architecture has found its way
into low-level vision tasks, such as image SR. Contemporary Transformer-based
approaches aim to alleviate the computational load by confining self-attention
to local regions and incorporating a higher degree of locality bias into their
network design. SwinIR (Liang et al., 2021) incorporates local window self-
attention and a shift mechanism inspired by the Swin Transformer design(Liu et
al., 2021). Meanwhile, others like ELAN (Zhang et al., 2022) or ESRT (Lu et
al., 2022) reduce the feature dimensions by splitting or down-scaling to
enhance the computational efficiency. Omni-SR (Wang et al., 2023) models
pixel-interactions across different axes, creating universal correlations.
SRFormer (Zhou et al., 2023) optimizes the computational efficiency by
employing large window self-attention through the permutation of self-
attention mechanisms. However, transformer-based methods typically demand
significantly higher computational resources, even with smaller model
capacities.

Figure 2: Architectute Overview. SeemoRe refines the feature representations
via stacked Residual groups (RGs). Each RG consists of a Rank Modulating Exert
(RME) and a Spatial Modulating Expert (SME). RME leverages the Mixeture of Low
Rank (MoRE) to refine the global texture, while SME employs spatial
enhancement experts (SEE) to supplement RME with the spatial cues.

Efficiency in SR. In recent years, the pursuit of efficient SR techniques has
gained significant momentum (Li et al., 2022; Ignatov et al., 2023; Li et al.,
2023; Conde et al., 2023). Consequently, researchers have introduced
streamlined neural architectures (Ignatov et al., 2021), network compression
(Wang et al., 2022), reparameterization (Zhang et al., 2021b), and other
training strategies to cater to the demand for efficiency. Initially,
efficient SR methods utilized group convolutions and cascaded block designs to
boost efficiency (Ahn et al., 2018; Hui et al., 2019). Subsequent advancements
introduced convolution-based spatial or channel enhancement modules (Liu et
al., 2020b). More recently, ShuffleMixer (Sun et al., 2022) integrates large
kernel convolutions and feature shuffling, improving both computational
efficiency and high-resolution reconstruction. SAFMN (Sun et al., 2023)
improves the efficiency by collecting non-local features using a shallow
pyramid. Despite improvements in several efficiency aspects brought up by the
aforementioned approaches, there is still scope for a better trade-off between
model efficiency and the restoration performance.

##  3 Methodology

In this section, we unveil the fundamental components of our proposed model
tailored for efficient super-resolution. As demonstrated in Figure 2, our
overall pipeline embodies a sequence of $N$ residual groups (RGs) and an
upsampler layer. The initial step involves applying a 3$\times$3 convolution
layer to generate the shallow features from the input low-resolution (LR)
image. Subsequently, multiple stacked RGs are deployed to refine the deep
features, easing the reconstruction of high-resolution (HR) images while
maintaining efficiency. Each RG consists of a Rank modulating expert (RME) and
a Spatial modulating expert (SME). Lastly, a global residual connection links
the shallow features to the output of the deep features for capturing the
high-frequency details and an up-sampler layer ( 3$\times$3 and pixel-shuffle
(Shi et al., 2016)) is deployed for faster reconstruction.

###  3.1 Rank Modulating Expert

Unlike large kernel convolution (Hou et al., 2022) or self-attention (Vaswani
et al., 2017) that rely upon resource-intensive matrix operations for
modelling the LR-HR dependencies, we opt for modulating the most relevant
interactions in low-rank in our quest for efficiency. Our proposed Rank
modulating expert (RME) (see Figure 2) explores a Transformer alike
architecture using Mixture of Low-Rank Expertise (MoRE) for modelling the
relevant global informative features efficiently and a GatedFFN (Chen et al.,
2023) for refined contextual feature aggregation.

Mixture of Low-Rank Expertise. As illustrated in Figure 3, from a layer
normalised input tensor $\mathbf{x}\in\mathbb{R}^{H\times W\times C}$, we use
a 3$\times$3 convolution for feature projection and then we split along the
channel dimension to create two distinct views
$\mathbf{x}_{a}~{}\text{and}~{}\mathbf{x}_{b}\in\mathbb{R}^{H\times W\times
C}$. To efficiently aggregate the pixel-wise cross-channel context, we
leverage a recursive strided convolution $t$ times followed by a refinement
and upsampling step, resulting in the construction of the feature pyramid
denoted as $\mathbf{\hat{x}}_{b}\in\mathbb{R}^{H\times W\times C}$. The
process is formulated as follows:

$\displaystyle|p|_{\downarrow h\times w}$
$\displaystyle=\text{DConv}^{s}_{k\times k}(...(\text{DConv}^{s}_{k\times
k}(\mathbf{x}_{b}))$ (1) $\displaystyle\mathbf{\hat{x}}_{b}$
$\displaystyle=|~{}\mathbf{W}_{C\rightarrow C}(\text{DConv}_{3\times
3}(|p|_{\downarrow h\times w}))~{}|_{\uparrow H\times W},$ (2)

where $\text{DConv}_{k\times k}$ denotes a depth-wise convolution with kernel
size $k$ and stride $s$, $\mathbf{W}_{C\rightarrow C}$ denotes a linear layer,
$p$ represents the contextual feature pyramid. Simultaneously, a parallel
depth-wise convolution extracts the local spatial context
$\mathbf{\hat{x}}_{a}$ before feeding both the extracted feature maps into the
mixture of low-rank expertise. This branched parallel design approach is
chosen purposefully. In general, the downsampling of the feature maps impacts
the reconstruction performance of SR methods. Therefore, we maintain the same
resolution for general feature extraction while incorporating an additional
path to capture global contextual cues efficiently, thereby circumventing any
information loss.

To further delve into the intricacies of the inter-dependencies among the
extracted features for reducing complexity, we deploy low-rank decomposition
for the inputs while modeling the global contextual relationships. As
demonstrated in Figure 3, a single low-rank expert ($\mathcal{E}$), takes as
input the spatial features, $\mathbf{\hat{x}}_{a}$, and encoded pixel-wise
contextual cues, $\mathbf{\hat{x}}_{b}$ and is formulated as:

$\displaystyle\mathcal{E}_{i}$ $\displaystyle=\mathbf{W}^{3}_{R_{i}\rightarrow
C}(\mathbf{W}^{1}_{C\rightarrow
R_{i}}\mathbf{\hat{x}}_{a}\odot\mathbf{W}^{2}_{C\rightarrow
R_{i}}\mathbf{\hat{x}}_{b}),$ (3)

where the linear layers denoted as $\mathbf{W}_{C\rightarrow R_{i}}$, compress
the encoded features along the channel dimension to their low-rank
approximation $R_{i}$, where $i\in\\{1,...,n\\}$. After adeptly modulating the
spatial cues through element-wise multiplication with the contextual cues in
low-dimensional space, another linear layer $\mathbf{W}^{3}_{R_{i}\rightarrow
C}$ extends the features back to the original dimension $C$ to extract the
relevant channel-wise spatial content. Thereby, implicitly mixing the crucial
spatial and channel dependencies in an efficient way.

However, manually determining the optimal low-rank ($R$) may not fully
leverage all the inherent information for modulation, leading to underutilized
model capacity. Thus, we employ a dynamic approach using a mixture of
different low rank experts, with a routing network ($\mathcal{G}$) that
systematically explores the search space to identify the ideal low-rank expert
based on the input and network depth. Following (Shazeer et al., 2017), the
final output $\mathbf{y}$ of the mixture of low-rank experts is as follows:

$\displaystyle\mathbf{y}$
$\displaystyle=\sum_{i}^{n}\mathcal{G}(\mathbf{\hat{x}}_{a})\mathcal{E}_{i}(\mathbf{\hat{x}}_{a},\mathbf{\hat{x}}_{b})+\mathbf{\hat{x}}_{a},$
(4)

where $\mathcal{G}(\cdot)$ and $\mathcal{E}_{i}(\cdot)$ denote the learned
routing function and the output of the $i$-th expert, respectively. The
sparsity inherent in the router function $\mathcal{G}(\cdot)$ optimizes
computation by assigning greater weights to the top-$k$ low-rank experts.
While at training time, our method learns from different experts, during
inference, only the selected top-$k$ expert is utilized for computation,
further enhancing the efficiency. More specifically, the inference complexity
is not proportional to the number of experts. The design choices contributing
towards the selection of the number of low-rank experts ($\mathcal{E}_{i}$)
and the rank dimension ($R$) for memory efficient reconstruction is
illustrated in Table 5 of the ablation study. We also provide the pseudocode
for the proposed MoRE block in Algorithm 1.

Figure 3: Illustration of the proposed Mixture of Low-Rank Expertise (MoRE) as
core block of the RME. Algorithm 1 Mixture of Low-Rank Experise

1: Input: Input feature $\mathbf{\hat{x}}_{a}$, semantic cues
$\mathbf{\hat{x}}_{b}$

2: Parameters: $n$ Experts $\mathcal{E}$, Router $\mathcal{G}$, Low-Rank
dimensions $R_{i}=2^{i+1}$ with $i\in\\{1,...,n\\}$, top-k expert $k$

3: Compute router outputs: $\mathbf{g}=\mathcal{G}(\mathbf{\hat{x}}_{a})$

4: Normalize weights: $\mathbf{w}=\text{Softmax}(\mathbf{g})$

5: Select top-k expert: $w_{k}=\text{topk}(\mathbf{w},k)$

6: if training then

7: for each $e\in\mathcal{E}$ do

8: $\mathbf{y}^{i}_{e}=\mathbf{W}^{3}_{R_{i}\rightarrow
C}(\mathbf{W}^{1}_{C\rightarrow
R_{i}}\mathbf{\hat{x}}_{a}\odot\mathbf{W}^{2}_{C\rightarrow
R_{i}}\mathbf{\hat{x}}_{b})$

9: end for

10: Compute final output:
$\mathbf{y}=\sum_{i=1}^{n}w_{i}\cdot\mathbf{y}^{i}_{e}$

11: else

12: Compute final output: $\mathbf{y}=w_{k}\cdot\mathbf{y}^{k}_{e}$

13: end if

14: Output: Final output $\mathbf{y}$

###  3.2 Spatial Modulating Expert

We observe that the rank modulating expert is more dedicated towards
investigating the global channel-wise contextual information, and its
effectiveness would be complemented by the spatial-wise local information.
Inspired by the previous work in classification (Yang et al., 2022; Hou et
al., 2022), we design a spatial modulating expert (SME) (see Figure 2)
comprising of a spatial enhancement expertise (SEE) block that efficiently
captures the spatial-wise coupling followed by a GatedFFN (Chen et al., 2023)
for feature refinement.

#### Spatial Enhancement Expertise.

While the vanilla self-attention (SA) mechanism (Vaswani et al., 2017) creates
connections among all the input pixels, effectively capturing the relevant
context, its quadratic computational complexity with image size poses
limitations, particularly in high-resolution scenarios like image SR. Thus,
our spatial enhancement expertise simplifies the computation of the similarity
matrix $\mathbf{A}$ between keys $\mathbf{K}$ and queries $\mathbf{Q}$ by
utilizing a striped depth-wise convolution with a large kernel, sequentially
convolving the feature maps with $\mathbf{k_{1}}\in\mathbb{R}^{[1,k]}$
followed by $\mathbf{k_{2}}\in\mathbb{R}^{[k,1]}$. Specifically, we compute
the locally enhanced spatial-wise features as follows:

$\displaystyle\mathbf{x}_{out}$ $\displaystyle=\text{DConv}^{s}_{k\times
k}(\mathbf{W}^{4}_{C\rightarrow
C}\mathbf{x}_{in})\odot\mathbf{W}^{5}_{C\rightarrow C}\mathbf{x}_{in},$ (5)

where $\odot$ is the Hadamard product, $\mathbf{W}^{4}_{C\rightarrow C}$ and
$\mathbf{W}^{5}_{C\rightarrow C}$ are linear (project) layers,
$\text{DConv}^{s}_{k\times k}$ denotes the striped depth-wise convolution, and
$\mathbf{x}_{in}$ is the layer normalised output of the RME. The use of a
large-kernel convolution facilitates a localized correlation among the pixels
within the $k\times k$ window, emulating the window-based SA layers frequently
employed in image restoration (Liu et al., 2021; Zamir et al., 2022; Chen et
al., 2023), all the while preserving the efficiency benefits associated with
convolutional layers as demonstrated in Table 4(a).

##  4 Experiments

Table 1: Comparison to efficient SR models. PSNR (dB $\uparrow$) and SSIM ($\uparrow$) metrics are reported on the Y-channel. Best and second best performances are highlighted. GMACS (G) are computed by upscaling to a $1280\times 720$ HR image. SeemoRe-T achieves state-of-the-art performance across all benchmarks with the lowest parameter count and computational demand. ‘-’ represents unreported results. |  Method | Params | GMACS | Set5 | Set14 | BSD100 | Urban100 | Manga109
---|---|---|---|---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
$\times 2$ |  Bicubic | - | - | 33.66 | .9299 | 30.24 | .8688 | 29.56 | .8431 | 26.88 | .8403 | 30.80 | .9339
CARN-M (Ahn et al., 2018) | 412K | 91 | 37.53 | .9583 | 33.26 | .9141 | 31.92 | .8960 | 31.23 | .9193 | - | -
IMDN (Hui et al., 2019) | 694K | 159 | 38.00 | .9605 | 33.63 | .9177 | 32.19 | .8996 | 32.17 | .9283 | 38.88 | .9774
PAN (Zhao et al., 2020) | 261K | 71 | 38.00 | .9605 | 33.59 | .9181 | 32.18 | .8997 | 32.01 | .9273 | 38.70 | .9773
DRSAN (Park et al., 2021) | 370K | 86 | 37.99 | .9606 | 33.57 | .9177 | 32.16 | .8999 | 32.10 | .9279 | - | -
DDistill-SR (Wang et al., 2022) | 414K | 128 | 38.03 | .9606 | 33.61 | .9182 | 32.19 | .9000 | 32.18 | .9286 | 38.94 | .9777
ShuffleMixer (Sun et al., 2022) | 394K | 91 | 38.01 | .9606 | 33.63 | .9180 | 32.17 | .8995 | 31.89 | .9257 | 38.83 | .9774
SAFMN (Sun et al., 2023) | 228K | 52 | 38.00 | .9605 | 33.54 | .9177 | 32.16 | .8995 | 31.84 | .9256 | 38.71 | .9771
SeemoRe-T (ours) | 220K | 45 | 38.06 | .9608 | 33.65 | .9186 | 32.23 | .9004 | 32.22 | .9286 | 39.01 | .9777
$\times 3$ |  Bicubic | - | - | 30.39 | .8682 | 27.55 | .7742 | 27.21 | .7385 | 24.46 | .7349 | 26.95 | .8556
CARN-M (Ahn et al., 2018) | 415K | 46 | 33.99 | .9236 | 30.08 | .8367 | 28.91 | .8000 | 27.55 | .8385 | - | -
IMDN (Hui et al., 2019) | 703K | 72 | 34.36 | .9270 | 30.32 | .8417 | 29.09 | .8046 | 28.17 | .8519 | 33.61 | .9445
PAN (Zhao et al., 2020) | 261K | 39 | 34.40 | .9271 | 30.36 | .8423 | 29.11 | .8050 | 28.11 | .8511 | 33.61 | .9448
DRSAN (Park et al., 2021) | 410K | 43 | 34.41 | .9272 | 30.27 | .8413 | 29.08 | .8056 | 28.19 | .8529 | - | -
DDistill-SR (Wang et al., 2022) | 414K | 57 | 34.37 | .9275 | 30.34 | .8420 | 29.11 | .8053 | 28.19 | .8528 | 33.69 | .9451
ShuffleMixer (Sun et al., 2022) | 415K | 42 | 34.40 | .9272 | 30.37 | .8423 | 29.12 | .8051 | 28.08 | .8498 | 33.69 | .9448
SAFMN (Sun et al., 2023) | 233K | 23 | 34.34 | .9267 | 30.33 | .8418 | 29.08 | .8048 | 27.95 | .8474 | 33.52 | .9437
SeemoRe-T (ours) | 225K | 20 | 34.46 | .9276 | 30.44 | .8445 | 29.15 | .8063 | 28.27 | .8538 | 33.92 | .9460
$\times 4$ |  Bicubic | - | - | 28.42 | .8104 | 26.00 | .7027 | 25.96 | .6675 | 23.14 | .6577 | 24.89 | .7866
CARN-M (Ahn et al., 2018) | 415K | 33 | 31.92 | .8903 | 28.42 | .7762 | 27.44 | .7304 | 25.62 | .7694 | - | -
IMDN (Hui et al., 2019) | 715K | 41 | 32.21 | .8948 | 28.58 | .7811 | 27.56 | .7353 | 26.04 | .7838 | 30.46 | .9075
PAN (Zhao et al., 2020) | 272K | 28 | 32.13 | .8948 | 28.61 | .7822 | 27.59 | .7363 | 26.11 | .7854 | 30.51 | .9095
DRSAN (Park et al., 2021) | 410K | 31 | 32.15 | .8935 | 28.54 | .7813 | 27.54 | .7364 | 26.06 | .7858 | - | -
DDistill-SR (Wang et al., 2022) | 434K | 33 | 32.23 | .8960 | 28.62 | .7823 | 27.58 | .7365 | 26.20 | .7891 | 30.48 | .9090
ShuffleMixer (Sun et al., 2022) | 411K | 28 | 32.21 | .8953 | 28.66 | .7827 | 27.61 | .7366 | 26.08 | .7835 | 30.65 | .9093
SAFMN (Sun et al., 2023) | 240K | 14 | 32.18 | .8948 | 28.60 | .7813 | 27.58 | .7359 | 25.97 | .7809 | 30.43 | .9063
SeemoRe-T (ours) | 232K | 12 | 32.31 | .8965 | 28.72 | .7840 | 27.65 | .7384 | 26.23 | .7883 | 30.82 | .9107
Table 2: Comparison to lightweight SR Transformers. SeemoRe-L outperforms or achieves comparable performance to compared Transformers while being more efficient. Best and second best performances are highlighted. $\times 3$ results are in the Supplemental.  |  Method | Params | GMACS | Set5 | Set14 | BSD100 | Urban100 | Manga109
---|---|---|---|---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
$\times 2$ |  Bicubic | - | - | 33.66 | .9299 | 30.24 | .8688 | 29.56 | .8431 | 26.88 | .8403 | 30.80 | .9339
SwinIR-Light (Liang et al., 2021) | 910K | 244 | 38.14 | .9611 | 33.86 | .9206 | 32.31 | .9012 | 32.76 | .9340 | 39.12 | .9783
ELAN-Light (Zhang et al., 2022) | 621K | 201 | 38.17 | .9611 | 33.94 | .9207 | 32.30 | .9012 | 32.76 | .9340 | 39.11 | .9782
SRFormer-Light (Zhou et al., 2023) | 853K | 236 | 38.23 | .9613 | 33.94 | .9209 | 32.36 | .9019 | 32.91 | .9353 | 39.28 | .9785
DAT-Light (Chen et al., 2023) | 553K | 194 | 38.24 | .9614 | 34.01 | .9214 | 32.34 | .9019 | 32.89 | .9346 | 39.49 | .9788
SeemoRe-L (ours) | 931K | 197 | 38.27 | .9616 | 34.01 | .9210 | 32.35 | .9018 | 32.87 | .9344 | 39.49 | .9790
$\times 4$ |  Bicubic | - | - | 28.42 | .8104 | 26.00 | .7027 | 25.96 | .6675 | 23.14 | .6577 | 24.89 | .7866
SwinIR-Light (Liang et al., 2021) | 897K | 64 | 32.44 | .8976 | 28.77 | .7858 | 27.69 | .7406 | 26.47 | .7980 | 30.91 | .9151
ELAN-Light (Zhang et al., 2022) | 601K | 54 | 32.43 | .8975 | 28.78 | .7858 | 27.69 | .7406 | 26.54 | .7982 | 30.92 | .9150
SRFormer-Light (Zhou et al., 2023) | 873K | 63 | 32.51 | .8988 | 28.82 | .7872 | 27.73 | .7422 | 26.67 | .8032 | 31.17 | .9165
DAT-Light (Chen et al., 2023) | 573K | 50 | 32.57 | .8991 | 28.87 | .7879 | 27.74 | .7428 | 26.64 | .8033 | 31.37 | .9178
SeemoRe-L (ours) | 969K | 50 | 32.51 | .8990 | 28.92 | .7888 | 27.78 | .7428 | 26.79 | .8046 | 31.48 | .9181
|  |
---|---|---
Urban100: img60 ($\times 4$) |  Urban100: img73 ($\times 4$) |  Urban100: img11 ($\times 4$)
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
HR Crop |  DDistill-SR |  ShuffleMixer |  SwinIR-Light |  DAT-Light |  SeemoRe-L
Figure 4: Visual comparison of SeemoRe with state-of-the-art methods on
challenging cases for $\times 4$ SR from the Urban100 benchmark. Table 3:
Complexity Analysis. Runtime (ms, $\downarrow$) and memory consumption (M,
$\downarrow$) averaged across 200 samples.

Method | Input | Scale | Time | GPU Memory
---|---|---|---|---
DAT-Light | $[320,180]$ | $\times 4$ | 210.12 | 8715.1
SwinIR-Light | 131.25 | 6175.3
ShuffleMixer | 7.40 | 1380.7
DDistill-SR | 11.20 | 2822.1
SeemoRe-T (ours) | 5.66 | 2744.9
(a) Runtime and memory consumption.

Scale |  Method | Params. | GMACS | Urban100 | Manga109
---|---|---|---|---|---
$\times 2$ |  SwinIR$\ast$ | 191K | 48 | 31.56 | 38.07
SRFormer$\ast$ | 188K | 49 | 31.60 | 38.59
SeemoRe-T | 220K | 45 | 32.22 | 39.01
(b) PSNR (dB $\uparrow$) on the Y-Channel. $\ast$ denotes retrained models.

Datasets and Evaluation. Following the SR literature (Liang et al., 2021; Chen
et al., 2023), we utilize DIV2K (Agustsson & Timofte, 2017) and Flickr2K (Lim
et al., 2017) datasets for training. We produce LR images using bicubic
downscaling of HR images. When testing our method, we assess its performance
on canonical benchmark datasets for SR - Set5 (Bevilacqua et al., 2012), Set14
(Zeyde et al., 2010), BSD100 (Martin et al., 2001), Urban100 (Huang et al.,
2015) and Manga109 (Matsui et al., 2017). We calculate PSNR and SSIM results
on the Y-channel from the YCbCr color space.

Implementation Details. We augment our training data with randomly extracted
$64\times 64$-sized crops, with random rotation, horizontal and vertical
flipping. Similar to (Sun et al., 2022, 2023), we minimize the L1-Norm between
SR output and HR ground truth in the pixel and frequency domain using Adam
(Kingma & Ba, 2017) optimizer for $500$K iterations with a batch size of $32$
and initial learning rate of $1\times 10^{-3}$ halving it at following
milestones: [$250K$,$400K$,$450K$,$475K$]. All experiments are conducted with
the PyTorch framework on NVIDIA RTX 4090 GPUs. We design our smallest model
(SeemoRe-T) with 6 RGs. The feature dimension and channel expansion factor in
GatedFFN are set to $36$ and $2$, respectively. For all MoRE sub-modules, we
select an exponential growth of the channel dimensionality and choose in total
of $3$ experts. The kernel size in SEE is set to $11\times 11$. More details
can be found in the supplemental, c.f. Table 6.

###  4.1 Comparison to State-of-the-Art Methods

We present quantitative results for $\times 2$, $\times 3$, and $\times 4$
image SR, comparing against current efficient state-of-the-art models in Table
1, including CARN-M (Ahn et al., 2018), IMDN (Hui et al., 2019), PAN (Zhao et
al., 2020), DRSAN (Park et al., 2021), DDistill-SR (Wang et al., 2022),
ShuffleMixer (Sun et al., 2022), and SAFMN (Sun et al., 2023). Additionally,
we evaluate against lightweight variants of popular Transformer-based SR
models such as SwinIR (Liu et al., 2021), ELAN (Zhang et al., 2022), and
SRFormer (Zhou et al., 2023) in Table 2. Our proposed SeemoRe-T stands out as
the most efficient method, consistently surpassing all other methods across
all benchmarks and scale factors. For instance as clear from Table 1, on the
Urban100 and Manga109 benchmarks ($\times 2$), SeemoRe-T outperforms SAFMN
(Sun et al., 2023) by $0.41$dB and $0.30$dB, respectively. Furthermore, with
$47\%$ fewer parameters and $65\%$ fewer GMACS than DDistill-SR (Wang et al.,
2022), SeemoRe-T achieves on average $0.12$ dB higher PSNR results across all
benchmarks ($\times 4$). Scaling our method up to a comparable size with
lightweight Transformers yields comparable or superior results. As
demonstrated in Table 2, our SeemoRe-L outperforms SwinIR-Light and SRFormer-
Light on Manga109 ($\times 4$) by $0.57$dB and $0.31$dB, while requiring fewer
GMACS.

Visual Results. We show visual comparisons ($\times 4$) in Figure 4. In some
challenging scenarios, the previous methods may suffer blurring artifacts,
distortions, or inaccurate texture restoration. Contrary to others, our
SeemoRe alleviates the blurring artifacts better and maintains structural
fidelity. For instance, in image img60 and img73 from Urban100, certain
methods like DDistill-SR, SwinIR-Light and DAT-Light fail to accurately
reconstruct shadow patterns or window struts, whereas our method exhibits
strong recovery of fine details. These visual comparisons highlight SeemoRe’s
ability to reconstruct high-quality images by effectively leveraging local and
contextual information. Coupled with quantitative comparisons, these findings
underscore the effectiveness of our method. More visual results can be found
in the Supplementary material.

###  4.2 Model Complexity Trade-Off

In the vision domain, scalability becomes more paramount. We strive to expand
the limits of our SeemoRe framework, optimizing for both reconstruction
fidelity and efficiency. The framework provides three complexity scales—tiny
(T), base (B), and large (L)—with progressively improved reconstruction
performance, c.f Figure 1. In Table 3, we present comparisons of memory usage
and running time, demonstrating that our SeemoRe-T outperforms representative
state-of-the-art methods. By using the low-rank feature modulation and
simultaneous aggregation of the channel-spatial dependencies, the GPU
consumption of our SeemoRe-T is $3\%$ less than DDistill-SR, while being $2$
times faster. Additionally, Table 3 highlights the significant efficiency
advantage of SeemoRe over lightweight Transformers. Further results are
provided in the Supplemental. To further underscore our method’s capability,
we align SwinIR-Light and SRFormer-Light with a size and computational demand
similar to ours, followed by retraining these downsized networks using our
schedule. The results presented in Table 3(b) highlight that SeemoRe-T
significantly outperforms both Transformer-based models by a considerable
margin.

Table 4: Ablation on Blocks. GMACS ($\downarrow$) are computed by upscaling to
a $1280\times 720$ HR image.

Method | RME | SME | Params. | GMACS | Urban100 | Manga109
---|---|---|---|---|---|---
Baseline | - | - | 232K | 52 | 31.73 | 38.63
SeemoRe-T | $\checkmark$ | - | 249K | 48 | 31.99 | 38.75
- | $\checkmark$ | 238K | 53 | 32.05 | 38.96
$\checkmark$ | $\checkmark$ | 220K | 45 | 32.22 | 39.01
(a) Contribution of components.

Method |  Kernel $k$ | Params. | GMACS | Urban100 | Manga109
---|---|---|---|---|---
SeemoRe-T | 3 | 217K | 44 | 32.04 | 38.84
5 | 218K | 44 | 32.12 | 38.72
7 | 219K | 45 | 32.10 | 38.96
11 | 220K | 45 | 32.22 | 39.01
(b) Kernel size ($k$) variation.

Table 5: Ablation on MoRE. Exponential growth yields best performance in terms of parameter counts and PSNR. $\\#\mathcal{E}$ denotes the number of experts, and Dim. is the dimensionality of the rank. Method | $\\#\mathcal{E}$ | Dim. | Params. | Urban100 | Manga109
---|---|---|---|---|---
SeemoRe-T |  |  |  |  |
Growth |  $i+2$ | 6 | 2,3,4,5,6,7 | 229K | 32.12 | 39.01
$2*i+2$ | 4 | 2,4,6,8 | 224K | 32.11 | 38.98
$2^{i}$ | 4 | 2,4,8,16 | 231K | 32.21 | 39.02
$\\#\mathcal{E}$ |  $2^{i}$ | $3$ | ${2,4,8}$ | 220K | 32.22 | 39.01
| $2$ | ${2,4}$ | 214K | 32.19 | 39.00
| $1$ | ${2}$ | 211K | 32.16 | 38.92

###  4.3 Ablation Study

We conduct detailed studies on the components within our approach. All
experiments are conducted on the $\times 2$ setting.

Architecture Contribution. As reported in Table 4(a), we evaluate the
effectiveness of our proposed key architectural components by comparing them
with a baseline model consisting solely of depthwise and pointwise
convolutions, more details in Supplemental. After adding the proposed modules
into the baseline model, their is a notable and persistent improvement in the
results. The incorporation of RME or SME results in improvements of $0.26$ dB
or $0.32$ dB on Urban100 over the baseline, respectively. Although both
modules individually outperform the baseline with only a marginal increase in
parameters, alternating the insertion of both the modules within each RG fully
unleashes the model’s capabilities while enhancing the overall efficiency.
Overall, our SeemoRe-T obtains a compelling gain of $0.49$ dB and $0.38$ dB on
Urban100 and Manga109, respectively.

$0$$50$$100$$\\#$Images$\times 2$
Urban100$\mathcal{E}_{1}$$\mathcal{E}_{2}$$\mathcal{E}_{3}$

$\times 2$ Manga109

123456$0$$50$$100$Residual Groups$\\#$Images$\times 4$ Urban100

123456Residual Groups$\times 4$ Manga109

(a)

Urban100: img10$\mathcal{E}_{1}$$\mathcal{E}_{2}$$\mathcal{E}_{3}$Expert

Manga109:
img78$1$$2$$3$$4$$5$$6$$\mathcal{E}_{1}$$\mathcal{E}_{2}$$\mathcal{E}_{3}$Residual
GroupsExpert

(b)

Figure 5: Low-Rank Analysis. (a) We plot the decisions made by the routing
function for SeemoRe-T over the depth of the network.(b) We visualize the low-
rank features of SeemoRe-T for $\times 4$ SR given example images from
Urban100 and Manga109. InputBefore MoREAfter MoREAfter SEE Figure 6: Feature
Visualization. We present visualizations of feature maps before and after our
proposed modules. Clearly, our MoRE block notably enhances activation
sharpness via contextual feature modulation. Moreover, our SEE module improves
learned representations by integrating spatial cues effectively.

Design choices of SME. The main component of SME module, SEE deploys striped
convolutions with large-kernel sizes to effectively module the spatial cues.
Table 4(b) demonstrates that deploying large kernel sizes improves the overall
performance of the model. In particular, the PSNR shows a notable gain of 0.18
dB on Urban100 dataset when increasing the kernel size from 3$\times$3 to
11$\times$11 (keeping other settings intact), with only 3K increase in
parameters. It clearly proves that such a design benefits in the efficient use
of the relevant information to augment the restoration of sharp regions
spatially.

Design choices of RME. We motivate our design choices for the MoRE module in
RME by varying the growth function and the number of experts as depicted in
Table 5. When pursuing a dynamic solution for determining the optimal low-rank
dimensionality, it becomes necessary to design the corresponding search space.
First, we present results for $\times 2$ upscaling on Urban100 and Manga109
using different growth functions. Based on the observed outcomes, it is
evident that an exponentially increasing low-rank dimensionality yields the
best performance with marginal increase in the parameters. Hence, we opt to
retain this search space design in all further experimentation. Next, we
analyze the reconstruction quality based on the number of experts in each MoRE
module, while exponentially increasing the low-rank dimensionality. Based on
these experiments, we assert that the efficient results are obtained when we
have three, as our total number of experts.

##  5 Discussion on Experts

Our model integrates experts at varying levels, each specializing in crucial
factors for SR. In this section, we aim to elucidate their expertise.

Mixture of Low-Rank Experts. The decision-making process of the router at
different network depths is illustrated in Figure 5(a). Notably, earlier
blocks showcase a diverse range of rank choices (
$\mathcal{E}_{1}$,$\mathcal{E}_{2}$,$\mathcal{E}_{3}$), while deeper layers
tend to favor lower ranks ($\mathcal{E}_{1}$) (Please note that for every
$\mathcal{E}_{i}$, the corresponding rank dimension is $2^{i}$). This
phenomenon can be attributed to the hierarchical feature learning nature of
deep neural networks, aligning with our expectations. In fact, earlier layers
typically capture low-level details and, at times, unwanted noise while
reconstructing details in the input LR image, thus resulting in wide
variations in the rank choices. In contrast, deeper layers focus on the main
structures and key features required for SR. Hence, higher ranks at deeper
layers are less favored, as they may introduce redundancy or noise that does
not significantly contribute to the overall quality of the reconstructed
image. This design aspect provides our method with the flexibility to adapt to
the complexity of the task, a capability that, to the best of our knowledge,
has not yet been explored in the image reconstruction community. In Figure
5(b), we further visualize the routing decisions and the corresponding low-
rank feature maps for two exemplary input images. It is noteworthy that each
individual rank carries distinct information while being mutually
complementary. As the model depth increases, the network becomes proficient in
restructuring these representations.

How important are MoRE and SEE? To substantiate the significance of the
proposed MoRE and SEE modules, we analyze the feature maps before and after
integrating both blocks into the RME module as depicted in Figure 6. This
analysis vividly showcases the advantages of leveraging MoRE for contextual
information mining within RME. Notably, the activations exhibit reduced noise
and enhanced sharpness. Additionally, we observe a synergistic interaction
between MoRE and SEE at marked locations (indicated by red arrows): MoRE
effectively refines global textures by filtering out noise, while SEE
supplements over-filtered regions with critical local details.

##  6 Conclusion

We propose a novel ConvNet, named SeemoRe, for efficient and accurate image
SR. Our SeemoRe excels in modeling local and contextual information,
surpassing both previous CNN-based and lightweight Transformer approaches in
terms of efficiency and reconstruction fidelity. Unlike other approaches, we
empirically demonstrate both the scalability of efficiency and reconstruction
performance. In our approach, we intricately design the rank modulation expert
to discern the most pivotal features, enhancing this compressed representation
with valuable contextual cues. Our spatial enhancement expert efficiently
integrates local spatial-wise information, unlocking the full potential of our
architecture. This novel approach optimally exploits the low information
regime in the input image, enhancing detail reconstruction while improving
efficiency. Extensive experiments on image SR demonstrate that our proposed
SeemoRe achieves consistent superior performance over recent state-of-the-art
efficient methods on all considered SR benchmarks, while even being on par
with the lightweight Transformers in terms of reconstruction fidelity.

##  7 Impact Statements

This paper presents work whose goal is to advance the field of Machine
Learning, specifically efficient image super-resolution. There are many
potential societal consequences of our work, none which we feel must be
specifically highlighted here. However, applying super-resolution methods in
AI-assisted software raises ethical concerns about privacy invasion and
increased surveillance capabilities. Adherence to transparency,
accountability, and privacy rights is crucial to mitigate potential harm and
ensure responsible deployment in alignment with societal values.

## References

  * Agustsson & Timofte (2017) Agustsson, E. and Timofte, R.  NTIRE 2017 challenge on single image super-resolution: Dataset and study.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 126–135, 2017.
  * Ahn et al. (2018) Ahn, N., Kang, B., and Sohn, K.-A.  Fast, accurate, and lightweight super-resolution with cascading residual network.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 252–268, 2018.
  * Bevilacqua et al. (2012) Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi-Morel, M. L.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In _Proceedings of the British Machine Vision Conference_ , 2012.
  * Chen et al. (2023) Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., and Yu, F.  Dual aggregation transformer for image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Conde et al. (2023) Conde, M. V., Zamfir, E., and Timofte, R.  Efficient deep models for real-time 4k image super-resolution. ntire 2023 benchmark and report.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pp. 1495–1521, June 2023.
  * Dong et al. (2014) Dong, C., Loy, C. C., He, K., and Tang, X.  Learning a deep convolutional network for image super-resolution.  In _Proceeding of the European Conference on Computer Vision_ , pp. 184–199. Springer, 2014.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Duchon (1979) Duchon, C. E.  Lanczos filtering in one and two dimensions.  _Journal of Applied Meteorology and Climatology_ , 1979.
  * Hou et al. (2022) Hou, Q., Lu, C.-Z., Cheng, M.-M., and Feng, J.  Conv2former: A simple transformer-style convnet for visual recognition.  _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Huang et al. (2015) Huang, J.-B., Singh, A., and Ahuja, N.  Single image super-resolution from transformed self-exemplars.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5197–5206, 2015.
  * Hui et al. (2019) Hui, Z., Gao, X., Yang, Y., and Wang, X.  Lightweight image super-resolution with information multi-distillation network.  In _Proceedings of the ACM International Conference on Multimedia_ , pp. 2024–2032, 2019.
  * Ignatov et al. (2021) Ignatov, A., Timofte, R., Denna, M., and Younes, A.  Real-time quantized image super-resolution on mobile npus, mobile ai 2021 challenge: Report.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2525–2534, 2021.
  * Ignatov et al. (2023) Ignatov, A., Timofte, R., Denna, M., Younes, A., Gankhuyag, G., Huh, J., Kim, M. K., Yoon, K., Moon, H.-C., Lee, S., et al.  Efficient and accurate quantized image super-resolution on mobile npus, mobile ai & aim 2022 challenge: report.  In _Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III_ , pp. 92–129. Springer, 2023.
  * Khani et al. (2021) Khani, M., Sivaraman, V., and Alizadeh, M.  Efficient video compression via content-adaptive super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4521–4530, 2021.
  * Kim et al. (2016) Kim, J., Lee, J. K., and Lee, K. M.  Deeply-recursive convolutional network for image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1637–1645, 2016.
  * Kingma & Ba (2017) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization, 2017.
  * Kong et al. (2022) Kong, F., Li, M., Liu, S., Liu, D., He, J., Bai, Y., Chen, F., and Fu, L.  Residual local feature network for efficient super-resolution.  In _Proceedings of the European Conference on Computer Vision Workshops_ , 2022.
  * Li et al. (2022) Li, Y., Zhang, K., Timofte, R., Van Gool, L., Kong, F., Li, M., Liu, S., Du, Z., Liu, D., Zhou, C., et al.  Ntire 2022 challenge on efficient super-resolution: Methods and results.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1062–1102, 2022.
  * Li et al. (2023) Li, Y., Zhang, Y., Van Gool, L., Timofte, R., et al.  NTIRE 2023 challenge on efficient super-resolution: Methods and results.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2023.
  * Liang et al. (2021) Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Lim et al. (2017) Lim, B., Son, S., Kim, H., Nah, S., and Lee, K. M.  Enhanced deep residual networks for single image super-resolution.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017.
  * Liu et al. (2020a) Liu, J., Tang, J., and Wu, G.  Residual feature distillation network for lightweight image super-resolution.  In _Proceedings of the European Conference on Computer Vision Workshops_ , 2020a.
  * Liu et al. (2020b) Liu, J., Zhang, W., Tang, Y., Tang, J., and Wu, G.  Residual feature aggregation network for image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 2359–2368, 2020b.
  * Liu et al. (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2021.
  * Lu et al. (2022) Lu, Z., Li, J., Liu, H., Huang, C., Zhang, L., and Zeng, T.  Transformer for single image super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2022.
  * Martin et al. (2001) Martin, D., Fowlkes, C., Tal, D., and Malik, J.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In _Proceedings of the IEEE International Conference on Computer Vision_ , volume 2, pp. 416–423. IEEE, 2001.
  * Matsui et al. (2017) Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., and Aizawa, K.  Sketch-based manga retrieval using manga109 dataset.  _Multimedia Tools and Applications_ , 2017.
  * Park et al. (2021) Park, K., Soh, J. W., and Cho, N. I.  Dynamic residual self-attention network for lightweight single image super-resolution.  _IEEE Transactions on Multimedia_ , 2021.
  * Shazeer et al. (2017) Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Shi et al. (2016) Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., and Wang, Z.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1874–1883, 2016.
  * Sun et al. (2022) Sun, L., Pan, J., and Tang, J.  Shufflemixer: An efficient convnet for image super-resolution.  _Advances in Neural Information Processing Systems_ , 35:17314–17326, 2022.
  * Sun et al. (2023) Sun, L., Dong, J., Tang, J., and Pan, J.  Spatially-adaptive feature modulation for efficient image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2023) Wang, H., Chen, X., Ni, B., Liu, Y., and jinfan, L.  Omni aggregation networks for lightweight image super-resolution.  In _Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Wang et al. (2022) Wang, Y., Su, T., Li, Y., Cao, J., Wang, G., and Liu, X.  Ddistill-sr: Reparameterized dynamic distillation network for lightweight image super-resolution.  _IEEE Transactions on Multimedia_ , 2022.
  * Yang et al. (2022) Yang, J., Li, C., and Gao, J.  Focal modulation networks.  arXiv, 2022.
  * Zamir et al. (2022) Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., and Yang, M.-H.  Restormer: Efficient transformer for high-resolution image restoration.  In _CVPR_ , 2022.
  * Zeyde et al. (2010) Zeyde, R., Elad, M., and Protter, M.  On single image scale-up using sparse-representations.  In _Proceedings of International Conference on Curves and Surfaces_ , pp. 711–730. Springer, 2010.
  * Zhang et al. (2021a) Zhang, K., Li, D., Luo, W., Ren, W., Stenger, B., Liu, W., Li, H., and Yang, M.-H.  Benchmarking ultra-high-definition image super-resolution.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 14769–14778, 2021a.
  * Zhang et al. (2021b) Zhang, X., Zeng, H., and Zhang, L.  Edge-oriented convolution block for real-time super resolution on mobile devices.  In _Proceedings of the ACM International Conference on Multimedia_ , 2021b.
  * Zhang et al. (2022) Zhang, X., Zeng, H., Guo, S., and Zhang, L.  Efficient long-range attention network for image super-resolution.  In _Proceedings of the European Conference on Computer Vision_ , 2022.
  * Zhang et al. (2018a) Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y.  Image super-resolution using very deep residual channel attention networks.  In _Proceedings of the European conference on computer vision_ , 2018a.
  * Zhang et al. (2018b) Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y.  Residual dense network for image super-resolution.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018b.
  * Zhao et al. (2020) Zhao, H., Kong, X., He, J., Qiao, Y., and Dong, C.  Efficient image super-resolution using pixel attention.  In _Proceedings of the European Conference on Computer Vision Workshops_ , pp. 56–72, 2020.
  * Zhou et al. (2023) Zhou, Y., Li, Z., Guo, C.-L., Bai, S., Cheng, M.-M., and Hou, Q.  Srformer: Permuted self-attention for single image super-resolution.  _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.

Table 6: Implementation Details. Parameter | SeemoRe-T | SeemoRe-B | SeemoRe-L
---|---|---|---
Num. RGs | 6 | 8 | 16
Channel dimension | 36 | 48 | 48
MLP-Ratio | 2
LR dimensionality growth | exponential
Num. Experts $\mathcal{E}$ | 3
Top-$k$ experts | 1
SFM kernel size | 11
Recursion steps | 2 | 2 | 1
Training Dataset | DIV2K + Flickr2K
Optimizer | Adam
Batch size | 32
Total Num. Iterations | $500k$
FFT Loss weight | 0.1
LR-Rate | 1e.3
LR-Decay Rate | 0.5
LR-Decay Milestones |  [$250K$,$400K$,$450K$,$475K$]
Table 7: Comparison to lightweight SR Transformers. Extension of Table 2. PSNR (dB $\uparrow$) and SSIM ($\uparrow$) metrics are reported on the Y-channel. GMACS are computed by upscaling to a $1280\times 720$ HR image.  |  Method | Params | GMACS | SET5 | SET14 | BSD100 | Urban100 | Manga109
---|---|---|---|---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
$\times 3$ |  Bicubic | - | - | 30.39 | .8682 | 27.55 | .7742 | 27.21 | .7385 | 24.46 | .7349 | 26.95 | .8556
SwinIR-Light (Liang et al., 2021) | 918K | 111 | 34.62 | .9289 | 30.54 | .8463 | 29.20 | .8082 | 28.66 | .8624 | 33.98 | .9478
ELAN-Light (Zhang et al., 2022) | 629K | 90 | 34.61 | .9288 | 30.55 | .8463 | 29.21 | .8081 | 28.69 | .8624 | 34.00 | .9478
SRFormer-Light (Zhou et al., 2023) | 861K | 105 | 34.67 | .9296 | 30.57 | .8469 | 29.26 | .8099 | 28.81 | .8655 | 34.19 | .9489
DAT-Light (Chen et al., 2023) | 629K | 89 | 34.76 | .9299 | 30.63 | .8474 | 29.29 | .8103 | 28.89 | .8666 | 34.55 | .9501
SeemoRe-L (ours) | 959K | 87 | 34.72 | 0.9297 | 30.60 | .8469 | 29.29 | .8101 | 28.86 | .8653 | 34.53 | .9496
Table 8: Optimization function. SeemoRe-T was trained on DIV2K and Flickr2K. We report PSNR (dB $\uparrow$) on the Y-Channel for $\times 2$ upscaling. Method | L1 | FFT | BSD100 | Urban100 | Manga109
---|---|---|---|---|---
SeemoRe-T | $\checkmark$ | 1.0 | - | 0.0 | 32.21 | 32.14 | 38.90
$\checkmark$ | 1.0 | $\checkmark$ | 0.1 | 32.23 | 32.22 | 39.01
$\checkmark$ | 1.0 | $\checkmark$ | 0.2 | 32.22 | 32.16 | 39.02
Table 9: Model size. PSNR (dB $\uparrow$) is reported on the Y-channel. GMACS are computed by upscaling to a $1280\times 720$ HR image. N and C denote number of RGs and channel features, respectively. Method | Config | Params. | GMACS | Urban100 | Manga109
---|---|---|---|---|---
SeemoRe-T | N:6 | C:36 | 220K | 45 | 32.22 | 39.01
SeemoRe-B | N:8 | C:48 | 490K | 101 | 32.52 | 39.30
SeemoRe-L | N:16 | C:48 | 931K | 197 | 32.87 | 39.49

##  Appendix A Further Implementation Details

Table 6 outlines the architectural configurations and training settings
employed to achieve the reported results in this study. Throughout all our
experiments, we maintained a fixed random seed for reproducibility purposes.
We based our implementation on the public PyTorch-based
BasicSR111https://github.com/XPixelGroup/BasicSR framework for architecture
development and training. We use
fvcore222https://github.com/facebookresearch/fvcore Python package for
computing GMACS and parameter counts.

#### Baseline for Architecture Contribution.

Here we provide more details about the baseline method for the ablation in
Table 4(a). To ensure roughly equivalent parameter counts and computational
complexity, we adopt $5$ RGs with a channel dimensionality of $48$. Within
each RG, we integrate simple convolutional operators from our RME submodule
without the MoRe module, while the SME module is simplified to a pointwise
convolution.

##  Appendix B More Ablations

#### Optimization function.

In Table 8, we explore the impact of using the L1-Norm in FFT space to compare
the model output with high-quality GT images. Compared to utilizing only the
traditional L1 loss in RGB space, we observe an average performance
improvement of $0.09$ dB on Urban100 and Manga109 datasets while using the
combined losses. We acknowledge that only a few previous methods incorporate
the same FFT loss (Sun et al., 2022, 2023); however, other efficient image
super-resolution methods either employ a more intricate training schedule with
multiple stages (Liu et al., 2020a; Kong et al., 2022) or utilize large-scale
models for knowledge distillation (Wang et al., 2022).

#### Scaling the model size.

In Table 9, we detail the architecture, efficiency, and PSNR results across
different model sizes on Urban100 and Manga109 datasets. Starting with
SeemoRe-T, which has 220K parameters and 45 GMACS, each subsequent complexity
stage doubles these figures. Notably, all model stages achieve state-of-the-
art performance within their weight classes, with SeemoRe-L matching or even
surpassing recent lightweight Transformer-based SR models.

#### Future work and limitations.

The proposed approach, employing a mixture of experts for feature modulation,
is versatile for tasks with limited input information, such as low-light
enhancement and denoising. Additionally, SeemoRe’s efficient design makes it a
valuable solution for dynamic and resource-intensive environments. Expanding
the number of experts in our network’s low-rank aspect poses challenges due to
rapid feature dimensionality growth. Thus, our approach is currently limited
to a small number of experts, contrasting with other fields leveraging larger
expert ensembles.

|  |
---|---|---
Manga109: img25 ($\times 4$) |  Manga109: img99 ($\times 4$) |  Manga109: img59 ($\times 4$)
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
HR Crop |  DDistill-SR |  ShuffleMixer |  SwinIR-Light |  DAT-Light |  SeemoRe-L
Figure 7: Visual comparison of SeemoRe with state-of-the-art methods on challenging cases for $\times 4$ SR from the Manga109 benchmark. |  |
---|---|---
Urban100: img04 ($\times 4$) |  Urban100: img72 ($\times 4$) |  Urban100: img92 ($\times 4$)
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
HR Crop |  DDistill-SR |  ShuffleMixer |  SwinIR-Light |  DAT-Light |  SeemoRe-L
Figure 8: Visual comparison of SeemoRe with state-of-the-art methods on
challenging cases for $\times 4$ SR from the Urban100 benchmark.

##  Appendix C Visual Results.

We provide additional visual comparisons ($\times$4) in Figure 7 for the
Manga109 benchmark and in Figure 8 for the Urban100 benchmark. Our SeemoRe
framework consistently produces visually pleasing results, even on artistic
images. In contrast to previous methods which exhibit flawed texture and
character reconstruction, our proposed approach effectively reconstructs
missing details, as illustrated in Figure 7, across all exemplary images
considered. More concretely, when examining the example image img25 our
SeemoRe network proficiently reconstructs the capital letter “I” within the
text prompt “COMIC,” whereas SwinIR-Light and DAT-L encounter difficulty in
producing any readable output. Additionally, in example image img04 our model
significantly outperforms others in reconstructing the pattern with higher
fidelity. Moreover, our model’s reconstruction of img92 in Figure 8
demonstrates reduced blurring and more distinct edges, enhancing overall
visibility.