# Object 6D pose estimation meets zero-shot learning

Andrea Caraffa1<EMAIL_ADDRESS>Davide Boscaini1<EMAIL_ADDRESS>Amir
Hamza1,2<EMAIL_ADDRESS>Fabio Poiesi1<EMAIL_ADDRESS>
1Fondazione Bruno Kessler, Trento, Italy 2University of Trento, Italy

###### Abstract

Object 6D pose estimation methods can achieve high accuracy when trained and
tested on the same objects. However, estimating the pose of objects that are
absent at training time is still a challenge. In this work, we advance the
state-of-the-art in zero-shot object 6D pose estimation by proposing the first
method that fuses the contribution of pre-trained geometric and vision
foundation models. Unlike state-of-the-art approaches that train their
pipeline on data specifically crafted for the 6D pose estimation task, our
method does not require task-specific finetuning. Instead, our method, which
we name PoMZ, combines geometric descriptors learned from point cloud data
with visual features learned from large-scale web images to produce
distinctive 3D point-level descriptors. By applying an off-the-shelf
registration algorithm, like RANSAC, PoMZ outperforms all state-of-the-art
zero-shot object 6D pose estimation approaches. We extensively evaluate PoMZ
across the seven core datasets of the BOP Benchmark, encompassing over a
hundred objects and 20 thousand images captured in diverse scenarios. PoMZ
ranks first in the BOP Benchmark under the category Task 4: 6D localization of
unseen objects. We will release the source code publicly.

\begin{overpic}[width=496.85625pt]{main/images/teaser.png}
\put(0.4,1.8){\footnotesize\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{{RGB}}}
\put(12.8,12.9){\footnotesize\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{{Depth}}}
\put(-1.2,6.0){\footnotesize\rotatebox{90.0}{scene}}
\put(17.5,14.5){\footnotesize\rotatebox{90.0}{object}}
\put(36.5,-1.8){\footnotesize visual features} \put(51.5,-1.8){\footnotesize
geometric features} \put(67.5,-1.8){\footnotesize fused features}
\put(84.0,3.0){\footnotesize object pose in the scene}
\put(48.3,15.6){$\oplus$} \put(48.3,4.3){$\oplus$} \put(63.8,15.6){$=$}
\put(63.8,4.3){$=$}
\put(79.6,6.9){\small\rotatebox{90.0}{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{registration}}}
\end{overpic}

Figure 1:  Our approach, PoMZ, is designed for zero-shot object 6D pose
estimation, which entails estimating the pose of an unseen 3D object within a
3D scene. We begin by extracting features from the input object (first row)
and scene (second row). Then, we perform registration (gray block on the
right) to estimate the rotation and translation of the object within the
scene. PoMZ is the first method that successfully builds features for
registration based on the fusion of visual and geometric features extracted
from distinct and pre-trained networks. We extract visual features from a 2D
vision foundation model, while geometric features from a deep neural network
designed for 3D descriptors, each pre-trained with data different from that of
the target scenario. We depict the raw geometric content of point clouds (XYZ
coordinates) in yellow, while their photometric content (RGB colors) is
illustrated in green-brown hues. The colors in the middle section represent
PCA-processed visual, geometric, and our fused features, respectively. The
symbol $\oplus$ indicates feature concatenation.

##  1 Introduction

Object 6D pose estimation aims to find the rotation and translation of an
object within a 3D scene. This task is key for robotic manipulation [54, 26],
localization [4], autonomous driving [18], and augmented reality [36].
Learning-based approaches [44, 37], which utilize the object during both
training and inference phases, are currently the best performing ones. Recent
research trends have begun to challenge this premise, estimating poses of
objects unseen during training, i.e. a zero-shot learning scenario.

Determining the object 6D pose comes with well-known challenges, such as
occlusions, variations in illumination, texture-less or symmetrical objects,
cluttered scenes, and shape variability [48, 41]. In addition to these, the
context of zero-shot scenarios also requires solutions capable of generalizing
across different domains [6, 40]. There exist instance-level pose estimators
trained on specific objects and tested on scenes containing the same objects
[15, 8]. These methods often achieve precision but present limited
generalization abilities. Conversely, category-level pose estimators are
trained on a variety of objects of the same category (e.g. cups, laptops) and
tested in scenes that contain objects of the same category, but not of the
objects seen during training [49, 55, 7, 45]. Category-level pose estimators
can generalize to unseen instances of the same object, with the limitation of
being sensitive to shape and appearance variations. Category-level estimators
can be further classified into CAD-model-free and CAD-model-based approaches.
CAD-model-free approaches are either one-shot or few-shot learning methods
that build a sparse representation of an object given its scan as prior and
estimates the pose transformation in a scene though a 2D-3D matching network.
CAD-model-based approaches utilize a CAD model for each object category. These
methods demand no training, positioning them as a unique case within zero-shot
learning. ZeroPose [6] and MegaPose [22] are examples based on this
assumption. ZeroPose utilizes a hierarchical geometric structure matching
approach, correlating point features from segmented masks and depth
information with an object’s CAD model. However, it mainly focuses on
geometric features, overlooking critical visual and photometric details that
are essential for capturing an object’s real-world texture and appearance.
MegaPose adopts a render-and-compare technique to promote adaptability to
unseen objects. This approach demands an extensive and specialized 6D pose
estimation dataset, encompassing two million images across 20 thousand unique
objects.

In this work, we harness the synergy between 3D local deep descriptors, such
as GeDi [31], and 2D vision foundation models, such as DINOv2 [29], to create
new distinctive features for point cloud registration. 3D descriptors (or
geometric features) are optimized to encode local geometric structures,
falling short in capturing texture nuances. Conversely, visual features are
optimized to encode photometric patterns, falling short in capturing geometric
nuances. Given that both domains have now models that can generalize across a
variety of scenarios, we introduce PoMZ 111Project webpage:
davideboscaini.github.io/PoMZ_website. as an effective solution that blends
the two domains for zero-shot 6D pose estimation. Specifically, we mask the
object in the RGBD image using CAD-based novel object segmentation [27]. For
the CAD model, we extract geometric and visual features using the point cloud
and rendered views, respectively. For the RGBD image, we extract geometric and
visual features using the masked depth and the masked RGB, respectively.
Geometric features are estimated at multiple scales, and combined with visual
features via normalization. We estimate the 6D pose using RANSAC. In the case
of symmetric objects, we introduce a symmetry-aware refinement tasked to
minimize the patch-level similarity in the visual feature space to determine a
better pose. We extensively evaluate PoMZ across the seven core datasets of
BOP Benchmark222bop.felk.cvut.cz/leaderboards/pose-estimation-unseen-
bop23/core-datasets, accessed Nov. 2023., which include LM-O [3], T-LESS [16],
TUD-L [17], IC-BIN [11], ITODD [12], HB [20], and YCB-V [46]. Together, these
datasets encompass over a hundred objects and 20 thousand images, captured
across a wide range of scenarios. PoMZ ranks first in the BOP Benchmark under
the category Task 4: 6D localization of unseen objects. In summary, our
contributions are:

  * •

PoMZ is the first method that fuses the contribution of pre-trained geometric
and vision foundation models;

  * •

PoMZ is versatile and can be extended with features extraction networks that
may emerge in future;

  * •

We establish a new state-of-the-art benchmark in 6D pose estimation of unseen
objects.

##  2 Related work

3D deep descriptors are typically used for generic point cloud registration
problems [9, 31]. Corsetti et al. [10] showed that these descriptors can be
adapted to the challenge of object 6D pose estimation. Deep descriptors can be
based on the Local Reference Frame (LRF), which transforms a local set of
points (patch) into a canonical representation to achieve rotation and
translation invariance [47]. Canonicalized points are then processed by a
neural network to produce a compact descriptor as output. Methods in this
category include G3DOA [50], which utilizes multi-scale cylindrical
convolutions; WSDesc [24], based on voxelization layers that learn optimal
voxel neighborhood size; Li et al. [23], which employs multi-view
differentiable rendering; and GeDi [31], which processes canonicalized points
through a PointNet++ [32] network. Descriptors can aslo be computed without
relying on LRF. Methods in this category include SpinNet [1], which processes
points projected on a cylindrical kernel using 3D cylindrical convolutions;
FCGF [9], based on sparse convolutions for learning point-level 3D
descriptors; and PREDATOR [19], which uses an attention mechanism to handle
low-overlap point clouds.

These descriptors are tailored for registering point clouds of similar types,
dealing with structures of points that differ significantly from those found
in object 6D pose estimation benchmarks. In our work, we have successfully
employed these types of descriptors to establish new state-of-the-art results
in object 6D pose estimation.

2D vision foundation models, including CLIP [33], DINOv2 [29], ImageBind [13],
and Segment Anything Model (SAM) [21], can be employed across a range of tasks
and domains. CLIP [33] connects visual concepts with textual descriptions
using a ViT as image encoder and a GPT-like transformer as textual encoder,
respectively. DINOv2 [29] adopts self-supervised training on large-scale
unlabeled data, introducing a patch-level objective to capture fine-grained
image details. ImageBind [13] integrates data from six different modalities
into a shared feature space. This integration is achieved with image-paired
data, thereby eliminating the need for combinations of all modalities. SAM
[21] is specifically designed for object segmentation based on visual prompts
such as a single point, a set of points, or a bounding box. SAM can
effectively segment unseen object types in a variety on unseen contexts.

Although CLIP has been previously employed for zero-shot point cloud
understanding [53, 30], where visual features are transferred to points for
downstream tasks, to the best of our knowledge, there are no existing object
6D pose estimation methods that transfer analogous visual features to point
clouds in a similar manner. In this work, we successfully achieve this using
DINOv2 visual features to boost geometric features for registration, and for
effectively disambiguating object poses of symmetric objects.

Zero-shot object 6D pose estimation methods, while less numerous compared to
fully supervised [25, 15, 43] or few-/one-shot learning approaches [14, 38,
5], are gaining attention. Zero-shot methods include Zephyr [28], MegaPose
[22], ZeroPose [6], and ZS6D [2]. In the initial step, zero-shot image
segmentation techniques like SAM [21] can be employed to locate the object
within the image. Zephyr [28] generates candidate poses for an object and
selects the best one based on point-level differences in color and geometric
domains. MegaPose [22] estimates object poses by rendering multiple views of
the CAD model and matching these with the masked image to obtain a coarse
pose. ZeroPose [6] uses multi-resolution geometric features to match regions
between the scene’s point cloud and the CAD model. ZS6D [2] matches DINOv2
[29] visual features against a database of features from pre-rendered object
views to select the most similar one, followed by a RANSAC-based PnP algorithm
for final pose estimation.

Unlike existing zero-shot 6D pose estimation approaches, PoMZ employs deep
neural networks for feature extraction that have been trained for various and
different generic downstream tasks. PoMZ can utilize point cloud descriptors
that were trained for point cloud registration, such as GeDi [31], and image
feature extraction networks that were self-supervised on large-scale internet
data, such as DINOv2 [29]. PoMZ is designed to be adaptable, allowing for the
incorporation of new state-of-the-art feature representation methods as they
become available in the future, without requiring re-training on specific
data, all while maintaining the core methodology.

##  3 Our approach

\begin{overpic}[width=496.85625pt]{main/images/block_diagram.pdf}
\end{overpic} Figure 2:  Overview of PoMZ. Left-hand side, we create rendered
images from the input model of the query object to extract visual features,
which we then back project to the object point cloud. Concurrently, we extract
geometric features, which we then fuse with the visual features. Similarly,
right-hand side, we compute visual and geometric features of the target object
imaged in the input crop, and fuse the two as before. Although the query and
target objects are from two different modalities (a 3D model the former, and a
RGBD image the latter), we employ the same vision and geometric encoders to
compute their features. Lastly, we input the fused features to a registration
algorithm based on feature matching to estimate the object 6D pose. The color
heatmaps used for the features are generated by applying a t-SNE [42]
dimensionality reduction. The symbol $\oplus$ indicates feature concatenation.

###  3.1 Overview

Given a 3D model of an object and an RGBD image containing the object as
input, we first compute their geometric and visual features, and then estimate
the object 6D pose through registration (Fig. 2). For the 3D model, pixel-
level visual features are computed from rendered viewpoints of the 3D model
and then back projected to the 3D model. Geometric features are computed from
the point cloud of the 3D model (Sec. 3.3). For the RGBD image, pixel-level
visual features are computed from the RGB information, and transferred to
depth, which is then converted into a point cloud through the camera intrinsic
parameters (Sec. 3.4). Geometric and visual features of both the object and
the RGBD image are fused together (Sec. 3.5), and then used to estimate the 6D
pose through RANSAC (Sec. 3.6). Estimated poses of symmetric objects are
eventually corrected using a symmetry-aware refinement step (Sec. 3.7).

###  3.2 Problem formulation

Given a query object $O$ captured from a camera viewpoint in a scene $S$, we
aim to find the 6D pose of $O$ with respect to the camera reference frame. We
name $O$ in $S$ as target object. $O$ can be modeled as either a CAD model or
a triangular mesh reconstructed from multiple views. We produce a point cloud
representation $\mathcal{P}_{O}\in\mathbb{R}^{Q_{O}\times 3}$ by sampling
$Q_{O}$ points from the faces of $O$’s model. $\mathcal{P}_{O}$ is a set of
coordinate points. Let $\textbf{I}_{S}\in\mathbb{R}^{H\times W\times 4}$ be a
tensor of the capture of $S$, where
$\textbf{I}_{S}=[\textbf{I}_{S}^{RGB}\mid\textbf{I}_{S}^{D}]$,
$\textbf{I}_{S}^{RGB}\in\mathbb{R}^{H\times W\times 3}$ is the RGB image,
$\textbf{I}_{S}^{D}\in\mathbb{R}^{H\times W}$ is the depth map, $H$ and $W$
are the height and width of the tensor, and $\mid$ is the operator that
concatenates tensors along their last dimension. We compute visual and
geometric features for both the query and target objects using deep neural
networks. Specifically, we process images with a vision encoder
$\Phi_{\Theta}$ to get visual features, and point clouds with a geometric
encoder $\Psi_{\Omega}$ to get geometric features. $\Phi_{\Theta}$ and
$\Psi_{\Omega}$ are frozen. PoMZ processes a single instance of the target
object in $S$ at a time.

###  3.3 Query object processing

To compute the visual features $\mathcal{F}_{O}^{V}$, we begin by rendering
$R$ RGBD images of the textured 3D model of $O$ from different viewpoints,
i.e. $\\{\textbf{I}_{O,i}\in\mathbb{R}^{H\times W\times 4}\\}_{i=1}^{R}$,
where $i$ is a viewpoint. We also render the associated object segmentation
masks $\\{\textbf{M}_{O,i}\in{[0,1]}^{H\times W}\\}_{i=1}^{R}$. For each
rendered image, we crop $\textbf{I}_{O,i}$ based on the foreground pixels of
$\textbf{M}_{O,i}$, obtaining
$\tilde{\textbf{I}}_{O,i}=[\tilde{\textbf{I}}_{O,i}^{RGB}\mid\tilde{\textbf{I}}_{O,i}^{D}]$,
where $\tilde{\textbf{I}}_{O,i}^{RGB}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times 3}$ is the RGB information,
$\tilde{\textbf{I}}_{O,i}^{D}\in\mathbb{R}^{H^{\prime}\times W^{\prime}}$ is
the depth information, and $H^{\prime}\times W^{\prime}$ is the size of the
crop. $\tilde{\textbf{I}}_{O,i}^{RGB}$ is processed by $\Phi_{\Theta}$ to
generate the visual features
$\textbf{F}_{O,i}^{V}\in\mathbb{R}^{H^{\prime\prime}\times
W^{\prime\prime}\times F_{V}}$, where $H^{\prime\prime}\times
W^{\prime\prime}$ and $F_{V}$ are the output feature map size and dimension,
respectively. If $\Phi_{\Theta}$ is a Transformer, $H^{\prime\prime}\times
W^{\prime\prime}(\neq H^{\prime}\times W^{\prime}$) corresponds to the number
of image patches (16$\times$16 in DINOv2 [29]). We use bilinear interpolation
to convert $\textbf{F}_{O,i}^{V}$ to the pixel-level features
$\tilde{\textbf{F}}_{O,i}^{V}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times
F_{V}}$. $\tilde{\textbf{F}}_{O,i}^{V}$ are then back projected to the 3D
space, and associated with $\mathcal{P}_{O}$’s points as follows. To find the
correspondences between the pixels of $\tilde{\textbf{I}}_{O,i}^{RGB}$ and the
points of $\mathcal{P}_{O}$, we convert $\tilde{\textbf{I}}_{O,i}^{D}$ into a
viewpoint-dependent point cloud $\mathcal{P}_{O,i}\in\mathbb{R}^{Q_{O,i}\times
3}$ using renderer’s camera intrinsic parameters, and employ nearest neighbor
search between the points of $\mathcal{P}_{O}$ and $\mathcal{P}_{O,i}$. Let
$\mathcal{F}_{{O,i}}^{V}=\\{\textbf{f}_{O,i,j}^{V}\\}_{j=1}^{Q_{O,i}}$ be the
set of visual features extracted from the $i$th rendered image, where
$\textbf{f}_{O,i,j}^{V}\in\mathbb{R}^{F_{V}}$ is the feature corresponding to
the $j$th point of $\mathcal{P}_{O}$. Lastly, we aggregate
$\\{\mathcal{F}_{{O,i}}^{V}\\}_{i=1}^{R}$ into a single set
$\mathcal{F}_{O}^{V}=\\{\textbf{f}_{O,j}^{V}\\}_{j=1}^{Q_{O}}$ by averaging
the contribution of each viewpoint as
$\textbf{f}_{O,j}^{V}=\sum_{i=1}^{R}\textbf{f}_{O,i,j}^{V}/R$.

To compute the geometric features $\mathcal{F}_{O}^{G}$, we process
$\mathcal{P}_{O}$ using $\Psi_{\Omega}$ to obtain the point-level features
$\textbf{f}_{O,j}^{G}\in\mathbb{R}^{F_{G}}$ of dimension $F_{G}$, each
associated to one of the $Q_{O}$ points of $\mathcal{P}_{O}$, i.e.
$\mathcal{F}_{O}^{G}=\Psi_{\Omega}(\mathcal{P}_{O})=\\{\textbf{f}_{O,j}^{G}\\}_{j=1}^{Q_{O}}$.

###  3.4 Target object processing

Localization and masking. To localize $O$ within $S$, we use CNOS [27]. CNOS
performs zero-shot object segmentation by comparing region proposals generated
using SAM [21] (or FastSAM [51]) within $\textbf{I}_{S}^{RGB}$ against object
templates derived from rendering $O$’s textured 3D model. It assigns to each
proposals a score based on its visual feature similarity against the object
templates. CNOS produces the set $\\{(\textbf{M}_{O,S,i},c_{i})\\}_{i=1}^{M}$
as output, where $\textbf{M}_{O,S,i}\in[0,1]^{H\times W}$ is a binary
segmentation mask localizing $O$ in $S$, and $c_{i}$ is the respective
confidence score. The common practice is to select the mask
$\textbf{M}_{O,S,j}$ with the highest confidence score
$c_{j}=\max_{i=1,\dots,M}c_{i}$. However, due to CNOS not being well
calibrated, we noticed that more accurate segmentation masks might have lower
confidence scores. Hence, we retain the $1\leq\tilde{M}\leq M$ most confident
masks, obtaining the set
$\mathcal{M}_{O,S}=\\{(\textbf{M}_{O,S,i})\\}_{i=1}^{\tilde{M}}$. When no CNOS
masks are available for $O$ in $\textbf{I}_{S}$, we consider the whole scene,
i.e. $\mathcal{M}_{O,S}=\\{\mathbf{1}^{H\times W}\\}$. For each mask in
$\mathcal{M}_{O,S}$, we extract the region $S^{\prime}\subseteq S$ in
$\textbf{I}_{S}$ based on $\textbf{M}_{O,S,i}$, obtaining the cropped RGBD
image
$\textbf{I}_{S^{\prime}}=[\textbf{I}_{S^{\prime}}^{RGB}\mid\textbf{I}_{S^{\prime}}^{D}]$.

Feature extraction. To compute the visual features
$\mathcal{F}_{S^{\prime}}^{V}$, we process $\textbf{I}_{S^{\prime}}^{RGB}$
directly with $\Phi_{\Theta}$ to obtain visual features
$\textbf{F}_{S^{\prime}}^{V}\in\mathbb{R}^{H^{\prime\prime}\times
W^{\prime\prime}\times F_{V}}$. As for the query object, we upsample the
patch-level features $\textbf{F}_{S^{\prime}}^{V}$ via bilinear interpolation
to obtain pixel-level features
$\tilde{\textbf{F}}_{S^{\prime}}^{V}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times F_{V}}$. Associating $\tilde{\textbf{F}}_{S^{\prime}}^{V}$
with the points of $\mathcal{P}_{S^{\prime}}$ is more direct than the process
required for the query object. First, we convert $\textbf{I}_{S^{\prime}}^{D}$
into a point cloud
$\mathcal{P}_{S^{\prime}}\in\mathbb{R}^{Q_{S^{\prime}}\times 3}$ using
renderer’s camera intrinsic parameters, where $Q_{S^{\prime}}$ is the number
of foreground pixels in $\textbf{M}_{O,S}$. Then, leveraging the one-to-one
correspondences between the pixels of $\textbf{I}_{S^{\prime}}^{RGB}$ and
$\textbf{I}_{S^{\prime}}^{D}$, we seamlessly transfer the pixel-level features
$\tilde{\textbf{F}}_{S^{\prime}}^{V}$ to the points of
$\mathcal{P}_{S^{\prime}}$ forming the set
$\mathcal{F}_{S^{\prime}}^{V}=\\{\textbf{f}_{S^{\prime},j}^{V}\\}_{j=1}^{Q_{S^{\prime}}}$.
Here $\textbf{f}_{S^{\prime},j}^{V}\in\mathbb{R}^{F_{V}}$ is the feature
corresponding to the $j$th point of $\mathcal{P}_{S^{\prime}}$. Despite
$\Phi_{\Theta}$ processes both foreground and background pixels of
$\textbf{I}_{S^{\prime}}^{RGB}$, we only transfer foreground features to
$\mathcal{P}_{S^{\prime}}$.

To compute the geometric features $\mathcal{F}_{S^{\prime}}^{G}$, we process
$\mathcal{P}_{S^{\prime}}$ with $\Psi_{\Omega}$ to obtain point-level features
$\textbf{f}_{S^{\prime},j}^{G}\in\mathbb{R}^{F_{G}}$ of dimension $F_{G}$,
each associated to one of the $Q_{S^{\prime}}$ points of
$\mathcal{P}_{S^{\prime}}$, i.e.
$\mathcal{F}_{S^{\prime}}^{G}=\Psi_{\Omega}(\mathcal{P}_{S^{\prime}})=\\{\textbf{f}_{S^{\prime},j}^{G}\\}_{j=1}^{Q_{S^{\prime}}}$.

###  3.5 Visual and geometric feature fusion

We fuse visual and geometric features through concatenation for both the query
and target objects. To account for potential differences in dimensions or
norms between $\mathcal{F}_{O}^{V}$
$\left(\mathcal{F}_{S^{\prime}}^{V}\right)$ and $\mathcal{F}_{O}^{G}$
$\left(\mathcal{F}_{S^{\prime}}^{G}\right)$, we apply $L_{2}$ normalization to
both of them independently in order to balance their contribution in the final
feature. Specifically, the features we produce have the following form

$\displaystyle\mathcal{F}_{O}=\\{\textbf{f}_{O,j}\\}_{j=1}^{Q_{O}}=\left\\{\left[\frac{\textbf{f}_{O,j}^{V}}{\lVert\textbf{f}_{O,j}^{V}\rVert_{2}}\bigg{|}\frac{\textbf{f}_{O,j}^{G}}{\lVert\textbf{f}_{O,j}^{G}\rVert_{2}}\right]\right\\}_{j=1}^{Q_{O}}$
$\displaystyle\mathcal{F}_{S^{\prime}}=\\{\textbf{f}_{S^{\prime},j}\\}_{j=1}^{Q_{S^{\prime}}}=\left\\{\left[\frac{\textbf{f}_{S^{\prime},j}^{V}}{\lVert\textbf{f}_{S^{\prime},j}^{V}\rVert_{2}}\bigg{|}\frac{\textbf{f}_{S^{\prime},j}^{G}}{\lVert\textbf{f}_{S^{\prime},j}^{G}\rVert_{2}}\right]\right\\}_{j=1}^{Q_{S^{\prime}}},$

where $\textbf{f}_{O,j},\textbf{f}_{S^{\prime},j}\in\mathbb{R}^{F_{V}+F_{G}}$,
and $\lVert\cdot\rVert_{2}$ is the $L_{2}$ norm operator. We found this simple
fusion strategy working well in practice. However, alternative fusion
strategies can be employed here, and we leave this for future developments.

###  3.6 Registration

We process the pair $(\mathcal{P}_{O},\mathcal{P}_{S^{\prime}})$ together with
their relative features $(\mathcal{F}_{O},\mathcal{F}_{S^{\prime}})$ to output
the six-degrees-of-freedom transformation
$\textbf{T}_{O,S^{\prime}}=(\textbf{R},\textbf{t})$ that aligns the query
object to the target object, where $\textbf{R}\in\mathbb{R}^{3\times 3}$ is
the rotation component and $\textbf{t}\in\mathbb{R}^{3}$ is the translation
component of $\textbf{T}_{O,S^{\prime}}$. To robustly estimate
$\textbf{T}_{O,S^{\prime}}$, we use an off-the-shelf registration algorithm
based on feature matching, like RANSAC. We repeat this process for each region
$S^{\prime}\subseteq S$ extracted with a different mask in
$\mathcal{M}_{O,S}$, and retain only the pose associated with the highest
number of registration inliers.

###  3.7 Symmetry-aware refinement

Although our features incorporate both visual and geometric information, query
object symmetries can still lead to local-minimum registration solutions. To
mitigate this, we introduce a symmetry-aware refinement step that generates a
set of candidate pose hypotheses based on $O$’s symmetries, and select the one
with highest visual feature similarity.

Symmetries estimation aims to identify symmetries of $O$’s 3D shape,
regardless of its texture. First, we build a set of rotation matrices
$\\{\textbf{S}_{i}\in SO(3)\\}_{i=1}^{N}$ by uniformly sampling the rotation
space. Then, we define the set of symmetries
$\mathcal{S}_{O}=\\{\textbf{S}_{i}\\}_{i=1}^{\tilde{N}}$ as the rotation
matrices that minimize the Chamfer distance [52] between
$\mathcal{P}_{O^{\prime},i}=\textbf{S}_{i}\mathcal{P}_{O}$ and
$\mathcal{P}_{O}$.

Candidate selection. The textured 3D model of $O$ is first transformed
according to $\textbf{S}_{i}\in\mathcal{S}_{O}$, and then according to the
coarse pose $\textbf{T}_{O,S^{\prime}}$. Given $\textbf{S}_{i}$, we render the
RGB image $\textbf{I}_{O^{\prime},i}^{RGB}$, and process it with
$\Phi_{\Theta}$ to produce visual features $\textbf{F}_{O^{\prime},i}^{V}$. We
repeat this for all elements in $\mathcal{S}_{O}$ to produce
$\\{\textbf{F}_{O^{\prime},i}^{V}\\}_{i=1}^{\tilde{N}}$. Concurrently, we
process $\textbf{I}_{S^{\prime}}^{RGB}$ with $\Phi_{\Theta}$ to obtain
$\textbf{F}_{S^{\prime}}^{V}$. The best candidate is selected by computing the
cosine similarity between
$\\{\textbf{F}_{O^{\prime},i}^{V}\\}_{i=1}^{\tilde{N}}$ and
$\textbf{F}_{S^{\prime}}^{V}$, and identifying the index $j$ corresponding to
the highest similarity. We define the refined pose as
$\tilde{\textbf{T}}_{O,S^{\prime}}=\textbf{T}_{O,S^{\prime}}\circ\textbf{S}_{j}$.
If $\Phi_{\Theta}$ is a Transformer, we empirical observed that patch-level
features are more informative than the class token. We found the class token
to be ineffective in encoding pose information, a conclusion also supported in
[27]. As last step, we use the Iterative Closest Point (ICP) algorithm to
further refine $\tilde{\textbf{T}}_{O,S^{\prime}}$ at point level. Although
ICP is known to be sensitive to local minima, we find it effective to refine
several of our initial poses that are close to the correct solution.

\begin{overpic}[width=41.19443pt]{main/images/symmetries/initial_pose.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Coarse pose}} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/view_000000.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Sym. 0 (Id)}} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/view_000003.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Sym. 1 }} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/view_000001.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Sym. 2}} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/view_000002.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Sym. 3}} \end{overpic} |
---|---|---|---|---|---
\begin{overpic}[width=41.19443pt]{main/images/symmetries/crop_input.png} \put(3.0,5.0){\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\footnotesize{Input crop}} \put(30.0,-15.0){\footnotesize Score} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/sim_000000.pdf} \put(35.0,-15.0){\footnotesize 0.36} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/sim_000003.pdf} \put(35.0,-15.0){\footnotesize 0.36} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/sim_000001.pdf} \put(35.0,-15.0){\footnotesize 0.35} \end{overpic} | \begin{overpic}[width=41.19443pt]{main/images/symmetries/sim_000002.pdf} \put(30.0,-15.0){\footnotesize 0.47 \color[rgb]{0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color<EMAIL_ADDRESS>\char 51} \end{overpic} | \begin{overpic}[width=2.243pt]{main/images/symmetries/cbar.pdf} \put(7.0,-1.0){\tiny$-0.1$} \put(7.0,93.0){\tiny$+1$} \end{overpic}
Figure 3:  Example of symmetry-aware refinement. We adjust the inaccurate
coarse pose (top left corner) by comparing the input crop (bottom left corner)
with the rendered images of the four object symmetries (first row). By
computing the patch-level cosine similarities for each candidate (second row,
red-white-blue colors), and averaging them to obtain a global score (third
row), we can select the best candidate as the one with the highest score ( ✓).

Fig. 3 shows an example of symmetry-aware refinement. Starting from an
inaccurately estimated coarse pose, we rotate and render the object according
to its four symmetries. We compute the patch-level cosine similarity between
the visual feature of the input crop and the rendered images. Symmetry no. 3
exhibits a higher similarity score ($+30\%$ improvement), enabling us to
correct the coarse pose.

##  4 Experiments

###  4.1 Experimental setup

We randomly sample 10K points from the point clouds of both the query and the
target objects. Geometric and visual features are computed from these points.
We use the masks provided by BOP Benchmark for Task 42, which are estimated
with CNOS [27]. To compute robust visual features of the query object, we use
$R=200$ rendered images.

We use DINOv2 [29] as vision encoder $\Phi_{\Theta}$, and GeDi [31] as
geometric encoder $\Psi_{\Omega}$. DINOv2 is a vision foundation model trained
through self-supervision on large-scale web data. We use its ViT-giant
version, with 1.14B parameters. The model processes images of size
224$\times$224 and outputs features of dimension $F_{V}=1536$. GeDi is a
geometric feature extractor for point clouds with strong generalization
capability across different application domains. We use a GeDi model with
$F_{G}=32$ and set the voxel size to 2mm. We use multiple scales, setting LRF
radii with values $(0.2D,0.3D,0.4D)$, where $D$ is the object’s diameter.
Multi-scale features are concatenated to form our final geometric features
with a final dimension of $F_{G}=96$.

###  4.2 Datasets

We evaluate PoMZ on the BOP Benchmark Task 4: Model-based 6D localization of
unseen objects333bop.felk.cvut.cz/challenges, accessed Nov. 2023., which
consists of seven datasets: LM-O [3], T-LESS [16], TUD-L [17], IC-BIN [11],
ITODD [12], HB [20], and YCB-V [46]. Each dataset provides both 3D object
models and RGBD images annotated with ground-truth 6D object poses, except for
ITODD and HB where their annotations are not publicly available. The
evaluation is done by submitting results to BOP Benchmark’s server. Object
types. T-LESS and ITODD contain industrial objects, such as electrical or
mechanical items, with planar sides, sharp edges, and hollow parts. The other
datasets contain common objects, such as food packages, toy models, and
stationery items, with finer geometric details and smooth boundaries.
Industrial objects are provided as CAD models and their point clouds are
sparse, with only a few points at corners or borders. Common objects are
reconstructed from images acquired with RGBD sensors and feature denser point
clouds. Number of instances. LM-O, TUD-L, HB, and YCB-V contain at most one
instance per object, while the other datasets contain multiple instances.
ITODD contains 4 instances per object on average, except a few images that
contain 84 instances per object. T-LESS and IC-BIN contains 1.3 and 9
instances per object on average, respectively. Photometric information. T-LESS
and ITODD objects are texture-less. LM-O, TUD-L, and HB objects have mostly
uniform colors. IC-BIN and YCB-V objects have rich textures. Noise levels.
TUD-L contains a single object for each scene with minimal self-occlusions,
but with difficult light conditions. LM-O scenes are highly cluttered (other
objects are present) and feature occlusions. IC-BIN scenes contain several
instances of the same object, thus featuring different levels of occlusions
and a high localization ambiguity. The other datasets contain mildly occluded
objects.

###  4.3 Metrics

We evaluate the accuracy of pose estimation using the default metrics of the
BOP Benchmark [39], which are VSD, MSSD, and MSPD. These metrics measure
different errors between two 3D models obtained by transforming the object 3D
model with the predicted and ground-truth poses. VSD measures the discrepancy
between the depth maps obtained by rendering them. MSSD and MSPD measure the
maximum Euclidean distance and the maximum reprojection error between their
points, respectively. Both MSSD and MSPD take into consideration symmetries,
by assuming the minimum value reached across all symmetrically equivalent
ground-truth poses. For each experiment, we report the average of the three
metrics, namely Average Recall (AR).

###  4.4 Quantitative results

Table 1:  Results of zero-shot object 6D pose estimation methods evaluated as
Average Recall (AR). Rows 1-8: best-performing methods from BOP Benchmark’s
leaderboard of Task 4: Localization of unseen objects2. Although not being
peer-reviewed methods, we include them for completeness. Rows 9-12 & 14-17:
published methods. Bold font indicates best AR. Our main result is
highlighted. Keys. Prior: type of instance localization prior; Refin.: pose
refinement; Avg: average AR; ‘-’: not available.

|  | Method | Prior | Refin. | LM-O | T-LESS | TUD-L | IC-BIN | ITODD | HB | YCB-V | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---
BOP leaderboard | 1 | MegaPose | CNOS |  | 22.9 | 17.7 | 25.8 | 15.2 | 10.8 | 25.1 | 28.1 | 20.8
2 | GenFlow | CNOS |  | 23.5 | 25.0 | 21.5 | 30.0 | 16.8 | 15.4 | 28.3 | 27.7
3 | ZeroPose | ZeroPose |  | 34.8 | 27.2 | 15.6 | 53.6 | 30.7 | 36.2 | 46.2 | 34.1
4 | ZeroPose | ZeroPose | ✓ | 49.3 | 34.2 | 79.0 | 39.6 | 46.5 | 62.9 | 62.3 | 53.4
5 | ZeroPose | CNOS | ✓ | 53.8 | 40.0 | 83.5 | 39.2 | 52.1 | 65.3 | 65.3 | 57.0
6 | SAM6D | CNOS | ✓ | 64.8 | 48.3 | 79.4 | 50.4 | 35.1 | 72.7 | 80.4 | 61.6
7 | MegaPose | CNOS | ✓ | 62.6 | 48.7 | 85.1 | 46.7 | 46.8 | 73.0 | 76.4 | 62.8
8 | GenFlow | CNOS | ✓ | 63.5 | 52.1 | 86.2 | 53.4 | 55.4 | 77.9 | 83.3 | 67.4
Published | 9 | OSOP [35] | OSOP |  | 27.4 | 40.3 | - | - | - | - | 29.6 | -
10 | ZS6D [2] | CNOS |  | 29.8 | 21.0 | - | - | - | - | 32.4 | -
11 | MegaPose [22] | Supervised |  | 18.7 | 19.7 | 20.5 | 15.3 | 8.00 | 18.6 | 13.9 | 16.2
12 | ZeroPose [6] | ZeroPose |  | 26.0 | 17.8 | 41.2 | 17.7 | 38.0 | 43.9 | 25.7 | 25.7
| 13 | PoMZ (ours) | CNOS |  | 65.4 | 47.9 | 89.2 | 39.1 | 49.6 | 73.2 | 70.1 | 62.1
Published | 14 | OSOP [35] | OSOP | ✓ | 48.2 | - | - | - | - | 60.5 | 57.2 | -
15 | ZeroPose [6] | ZeroPose | ✓ | 49.1 | 34.0 | 74.5 | 39.0 | 42.9 | 61.0 | 57.7 | 51.2
16 | MegaPose [22] | Supervised | ✓ | 58.3 | 54.3 | 71.2 | 37.1 | 40.4 | 75.7 | 63.3 | 57.2
17 | ZeroPose [6] | Supervised | ✓ | 56.2 | 53.3 | 87.2 | 41.8 | 43.6 | 68.2 | 58.4 | 58.4
| 18 | PoMZ (ours) | CNOS | ✓ | 68.4 | 52.1 | 94.5 | 50.3 | 55.9 | 79.1 | 84.0 | 69.2

Tab. 1 reports the results and comparative analyses with other methods on the
BOP Benchmark. The table is divided into two sections: the top part lists
concurrent and top-performing, yet unpublished, methods present in the BOP
Benchmark leaderboard of Task 4, while the bottom part contains published
methods. PoMZ outperforms all the published methods by a considerable margin.
Specifically, it outperforms the second-best published method ZeroPose [6]
(row 17) by +10.8 in Avg AR. This comparison is noteworthy as it is against
the version of ZeroPose that employs supervision for object segmentation
within the image. We can observe a significant improvement in YCB-V, where
PoMZ achieves a +25.6 in AR with respect to ZeroPose (row 17). YCB-V is a
dataset rich in symmetric objects, and here PoMZ is highly effective thanks to
its geometric and visual features, and the symmetry-aware refinement step.
This refinement step contributes to an increase of +13.9 in AR. A significant
improvement is noted in IC-BIN, where without refinement, PoMZ outperforms
ZeroPose (row 17) by +36.4 in Avg AR. The only dataset where we do not perform
best is T-LESS, however note that this is with respect to methods supervised
on instance localization. Compared to the second-best method with our setting,
i.e. ZeroPose (row 15), our improvement is +18.0 in Avg AR.

Lastly, PoMZ outperforms all methods present in the BOP Benchmark Task 4
leaderboard. For example, PoMZ achieves a +28.0 in Avg AR without refinement
(row 13 vs. row 3) and a +1.8 in Avg AR with refinement (row 18 vs. row 8).
These results are publicly available (and appropriately anonymized for double
blind review).

###  4.5 Qualitative results

\begin{overpic}[width=60.61348pt]{main/images/qualitatives/lmo/001005_000801_input_image-crop.png} \put(-13.0,13.0){\rotatebox{90.0}{\footnotesize Input images}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/lmo/001411_001168_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/tless/000313_000423_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/hb/001410_000700_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/icbin/000004_000000_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/ycbv/000110_001124_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/tudl/000234_001287_input_image-crop.png} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/ycbv/001169_000669_input_image-crop.png} \end{overpic}
---|---|---|---|---|---|---|---
\begin{overpic}[width=60.61348pt]{main/images/qualitatives/lmo/001005_000801_superimposed_grey-crop.png} \put(-13.0,20.0){\rotatebox{90.0}{\footnotesize Predictions}} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(a)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/lmo/001411_001168_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(b)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/tless/000313_000423_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(c)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/hb/001410_000700_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(d)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/icbin/000004_000000_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(e)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/ycbv/000110_001124_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(f)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/tudl/000234_001287_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(g)}} \end{overpic} | \begin{overpic}[width=60.61348pt]{main/images/qualitatives/ycbv/001169_000669_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(h)}} \end{overpic}
Figure 4:  Qualitative results of challenging cases. The top row shows input
images, while the bottom row shows PoMZ’s predictions by overlaying the object
3D model transformed according to the predicted pose. We use a grayscale
version of the input image for a better contrast. Left to right: (a,b) highly-
occluded texture-less toy models, (c,d) highly-occluded texture-less
industrial items, (e,f) common objects with cylindrical shapes and informative
textures, (g,h) difficult light conditions (strong shadows and a very bright
capture). Despite these challenges, PoMZ can always find the correct pose.

Fig. 4 shows examples of qualitative results, which include several challenges
such as occlusions, clutter, variations in background and illumination
conditions, poor object textures, severe object symmetries, and similarity
with other objects or different instances of the same object. Fig. 4(a,b) show
highly-occluded texture-less objects (the toy models of a red ape and a yellow
duck); Fig. 4(c,d) show highly-occluded texture-less symmetric objects (a
white electrical outlet and a bronze gear). To estimate the poses of
(a,b,c,d), PoMZ relies mostly on geometric information, since these objects
have non-informative uniform textures. Fig. 4(e,f) show objects with
cylindrical shapes (a coffee cup and a food can), where we can observe the
presence of multiple instances of the same coffee cup, each with different
levels of occlusions. To estimate the poses of (e,f), PoMZ relies mostly on
visual information, since cylindrical shapes present an infinite number of
symmetries, and the correct pose can be established only by relying on texture
information. Lastly, Fig. 4(g,h) show images captured under challenging light
conditions. The uniformly-colored white toy frog in (g) is affected by
shadows, while (h) shows a bright capture of a textured sugar box.

Fig. 5 shows example of failure cases. In Fig. 5(a), the red foam block is
aligned only to the closest corner, with a pose nearly perpendicular to the
correct one. This can be due to the part of the object that is occluded by the
marker. In Fig. 5(b), the yellow tea box pose is predicted laying on its side
rather than on its bottom. Because of the object’s simple geometry and poor
texture, we believe that aligning most of the corners of the box regardless of
its texture can be acceptable for PoMZ. In Fig. 5(c), the main body of the red
mug is fairly aligned, except the handle. This may be due to the limited
surface area of the handle that causes PoMZ to prioritize aligning the rest of
the object. In Fig. 5(d), the pose of the green drill handle is incorrectly
estimated. The handle is fully occluded in the input image, making it
challenging to predict the correct pose without additional prior information.
The visible portion (main body) is correctly aligned.

\begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/003000_000001_input_image-crop.png} \put(-13.0,12.0){\rotatebox{90.0}{\footnotesize Input images}} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000182_000450_input_image-zoom.png} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000012_000047_input_image-crop.png} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000078_000061_input_image-zoom.png} \end{overpic}
---|---|---|---
\begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/003000_000001_superimposed_grey-crop.png} \put(-13.0,16.0){\rotatebox{90.0}{\footnotesize Predictions}} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(a)}} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000182_000450_superimposed_grey-zoom.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(b)}} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000012_000047_superimposed_grey-crop.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(c)}} \end{overpic} | \begin{overpic}[width=53.11739pt]{main/images/qualitatives/failures/000078_000061_superimposed_grey-zoom.png} \put(-1.0,3.0){\hbox{\pagecolor{white}\footnotesize(d)}} \end{overpic}
Figure 5:  Failure cases. The structure is the same as Fig. 4: input images on
top, PoMZ’s predictions on bottom. Left to right: (a) the foam block is
aligned only to the closest corner, in a pose that is almost perpendicular to
the correct one, (b) the tea box pose is predicted as if it was laying on its
side rather than on its bottom, (c) the main body of the mug is nearly
aligned, but the handle is not, (d) the pose of the drill handle (occluded) is
incorrectly estimated.

###  4.6 Ablation study

In Tab. 2, we conduct an ablation study on LM-O [3] to assess each component
of PoMZ. Unless otherwise specified, we do not apply any pose refinement. The
ablation study on the refinement step is reported in Tab. 3. In both cases,
results are reported in terms of AR.

Feature types. We assess the fusion of geometric and visual features. In rows
1–3 of Tab. 2, we evaluate different geometric feature extraction methods:
hand-crafted approach FPFH [34], and learning-based techniques FCGF [9] and
GeDi [31]. In rows 10–12 of Tab. 2, we evaluate different vision encoders:
simple RGB values, and the foundation models CLIP [33] and DINOv2 [29]. We
observed that: (i) GeDi is highly effective across scenes (+34.5 in AR w.r.t.
FCGF); (ii) DINOv2 outperforms CLIP by about +6 in AR despite they are both
based on ViT-L; (iii) geometric features outperforms visual ones by a
significant margin.

Multi-scale geometric feature fusion. GeDi [31] uses a single radius to define
the extent of local descriptors. Instead, we use multiple radii to encode
information at multiple scales. In rows 4–9 of Tab. 2, we compare the original
single-scale version of GeDi with the proposed multi-scale fusion of GeDi
features for different radii. Radii are expressed as a percentage of the
object diameter, i.e. 0.2 corresponds to 20% of the object diameter. The best
AR is achieved when using multiple scales. Another crucial advantage of using
multiple scales lies in its generalizability. We experimentally observed that
the best-performing single-scale radius is dataset dependent, whereas using
multiple scales allow us to use the same configuration consistently across all
the seven core datasets.

Table 2:  Ablation study on the LM-O [3] dataset.

|  | geometric encoder | vision encoder | local. | refin. | AR
---|---|---|---|---|---|---
|  | method | radius | method | backbone | #params | prior
geometric abl. | 1 | FPFH | 0.3 | - | - | - | mask |  | 20.7
2 | FCGF | - | - | - | - | mask |  | 20.8
3 | GeDi | 0.3 | - | - | - | mask |  | 55.3
4 | GeDi | 0.2 | - | - | - | mask |  | 52.6
5 | GeDi | 0.3 | - | - | - | mask |  | 55.3
6 | GeDi | 0.4 | - | - | - | mask |  | 55.0
7 | GeDi | (0.2, 0.3) | - | - | - | mask |  | 56.2
8 | GeDi | (0.3, 0.4) | - | - | - | mask |  | 56.5
9 | GeDi | (0.2, 0.3, 0.4) | - | - | - | mask |  | 57.7
visual abl. | 10 | - | - | RGB | - | - | mask |  | 15.1
11 | - | - | CLIP | ViT-L | 428M | mask |  | 30.5
12 | - | - | DINOv2 | ViT-L | 304M | mask |  | 36.8
13 | - | - | DINOv2 | ViT-S | 22.1M | mask |  | 31.9
14 | - | - | DINOv2 | ViT-B | 86.6M | mask |  | 33.7
15 | - | - | DINOv2 | ViT-L | 304M | mask |  | 36.8
16 | - | - | DINOv2 | ViT-G | 1.14B | mask |  | 39.6
prior abl. | 17 | GeDi | (0.2, 0.3, 0.4) | DINOv2 | ViT-G | 1.14B | bbox |  | 65.4
18 | GeDi | (0.2, 0.3, 0.4) | DINOv2 | ViT-G | 1.14B | bbox | ICP | 67.9
19 | GeDi | (0.2, 0.3, 0.4) | DINOv2 | ViT-G | 1.14B | mask |  | 65.4
20 | GeDi | (0.2, 0.3, 0.4) | DINOv2 | ViT-G | 1.14B | mask | ICP | 68.3
| 21 | GeDi | (0.2, 0.3, 0.4) | DINOv2 | ViT-G | 1.14B | mask | ICP+SAR | 68.4

Visual features backbone. In rows 13–16 of Tab. 2, we assess different DINOv2
ViT backbones: small (S), base (B), large (L), and giant (G) models. The
latter achieves the highest AR (+2.8 in AR w.r.t. ViT-L, the CLIP backbone).

Localization prior type. In rows 17–20 of Tab. 2, we test a bounding box
localization prior rather than a segmentation mask prior. Without using ICP,
bounding boxes and segmentation masks produce the same results. When we use
ICP for the refinement of the pose, PoMZ performs better with the segmentation
mask prior. This suggests that our coarse pose estimation is robust to
background variations, while the refinement step is sensitive to background
outliers.

Pose refinement. In Tab. 3, we evaluate the effectiveness of the pose
refinement by comparing the coarse estimation (row 1) with the pose obtained
using an ICP-based refinement (row 2), and the one obtained adding also our
Symmetry-Aware Refinement (SAR for short, row 3). SAR gains the most
improvement over the baseline on datasets containing symmetric objects, such
as IC-BIN (+11.2 in AR) and YCB-V (+13.9 in AR). Instead, for datasets like
TUD-L and ITODD, SAR does not provide a substantial advantage compared to
solely relying on ICP.

Table 3:  Ablation study on the pose refinement module.

| Refin. | LM-O | T-LESS | TUD-L | IC-BIN | ITODD | HB | YCB-V | Avg
---|---|---|---|---|---|---|---|---|---
1 | None | 65.4 | 47.9 | 89.2 | 39.1 | 49.6 | 73.2 | 70.1 | 62.1
2 | +ICP | 68.3 | 51.6 | 94.5 | 41.7 | 55.9 | 75.8 | 79.8 | 66.8
3 | +SAR | 68.4 | 52.1 | 94.5 | 50.3 | 55.9 | 79.1 | 84.0 | 69.2

##  5 Conclusions

We presented PoMZ, a novel zero-shot object 6D pose estimation method. PoMZ
combines geometric descriptors learned from point cloud data with visual
features learned from large-scale web images to produce distinctive 3D point-
level descriptors. PoMZ estimates the 6D pose of the object within the scene
by using a RANSAC-based registration procedure followed by a novel symmetry-
aware pose refinement. We evaluated our approach on the BOP Benchmark, where
it ranks first for 6D localization of unseen objects, demonstrating state-of-
the-art performance across diverse objects and scenarios. Beside eliminating
the necessity for task-specific fine-tuning, PoMZ can be adapted to
incorporate new feature extraction networks that might emerge in the future,
while retaining its core approach.

Limitations. Although we leverage visual features, their fusion with geometric
features may not be sufficient to handle all symmetries. We rely on a CAD
model that in some applications may not be accessible. We need both RGB and
depth; depth may not be available in some cases.

## Acknowledgements

This work was supported by the European Union’s Horizon Europe research and
innovation programme under grant agreement No 101058589 (AI-PRISM), and by the
PNRR project FAIR – Future AI Research (PE00000013), under the NRRP MUR
program funded by the NextGenerationEU.

## References

  * Ao et al. [2021] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo.  Spinnet: Learning a general surface descriptor for 3d point cloud registration.  In _CVPR_ , 2021.
  * Ausserlechner et al. [2023] Philipp Ausserlechner, David Haberger, Stefan Thalhammer, Jean-Baptiste Weibel, and Markus Vincze.  Zs6d: Zero-shot 6d object pose estimation using vision transformers.  _arXiv:2309.11986_ , 2023.
  * Brachmann et al. [2014] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother.  Learning 6D object pose estimation using 3D object coordinates.  In _ECCV_ , 2014.
  * Bultmann et al. [2023] Simon Bultmann, Raphael Memmesheimer, and Sven Behnke.  External camera-based mobile robot pose estimation for collaborative perception with smart edge sensors.  _arXiv:2303.03797_ , 2023.
  * Castro and Kim [2023] Pedro Castro and Tae-Kyun Kim.  Posematcher: One-shot 6d object pose estimation by deep feature matching.  _arXiv:2304.01382_ , 2023.
  * Chen et al. [2023a] Jianqiu Chen, Mingshan Sun, Tianpeng Bao, Rui Zhao, Liwei Wu, and Zhenyu He.  ZeroPose: CAD-model-based zero-shot pose estimation.  _arXiv:2305.17934_ , 2023a.
  * Chen et al. [2023b] Kai Chen, Stephen James, Congying Sui, Yun-Hui Liu, Pieter Abbeel, and Qi Dou.  Stereopose: Category-level 6d transparent object pose estimation from stereo images via back-view nocs.  In _ICRA_ , 2023b.
  * Chen et al. [2020] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, and Ales Leonardis.  G2l-net: Global to local network for real-time 6d pose estimation with embedding vector features.  In _CVPR_ , 2020.
  * Choy et al. [2019] Christopher Choy, Jaesik Park, and Vladlen Koltun.  Fully convolutional geometric features.  In _ICCV_ , 2019.
  * Corsetti et al. [2023] Jaime Corsetti, Davide Boscaini, and Fabio Poiesi.  Revisiting Fully Convolutional Geometric Features for object 6D pose estimation.  In _ICCV-W_ , 2023.
  * Doumanoglou et al. [2016] Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim.  Recovering 6D object pose and predicting next-best-view in the crowd.  In _CVPR_ , 2016.
  * Drost et al. [2017] Bertram Drost, Markus Ulrich, Paul Bergmann, Philipp Hartinger, and Carsten Steger.  Introducing MVTec ITODD – A dataset for 3D object recognition in industry.  In _ICCV-W_ , 2017.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In _CVPR_ , 2023.
  * Goodwin et al. [2023] Walter Goodwin, Ioannis Havoutis, and Ingmar Posner.  You only look at one: Category-level object representations for pose estimation from a single example.  In _CoRL_ , 2023.
  * He et al. [2021] Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun.  Ffb6d: A full flow bidirectional fusion network for 6d pose estimation.  In _CVPR_ , 2021.
  * Hodan et al. [2017] Tomas Hodan, Pavel Haluza, Stepan Obdrzalek, Jiri Matas, Manolis Lourakis, and Xenophon Zabulis.  T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.  In _WACV_ , 2017.
  * Hodan et al. [2018] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders G. Buch, Dirk Kraft, Bertram Drost, Joel Vial, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiri Matas, and Carsten Rother.  BOP: Benchmark for 6D object pose estimation.  In _ECCV_ , 2018.
  * Hoque et al. [2023] Sabera Hoque, Shuxiang Xu, Ananda Maiti, Yuchen Wei, and Md Yasir Arafat.  Deep learning for 6d pose estimation of objects—a case study for autonomous driving.  _Expert Systems with Applications_ , 2023.
  * Huang et al. [2021] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, and Konrad Schindler Andreas Wieser.  Predator: Registration of 3d point clouds with low overlap.  In _CVPR_ , 2021.
  * Kaskman et al. [2019] Roman Kaskman, Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  HomebrewedDB: RGB-D dataset for 6D pose estimation of 3D objects.  In _ICCV-W_ , 2019.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Labbé et al. [2022] Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic.  Megapose: 6d pose estimation of novel objects via render & compare.  _arXiv:2212.06870_ , 2022.
  * Li et al. [2020] Lei Li, Siyu Zhu, Hongbo Fu, Ping Tan, and Chiew-Lan Tai.  End-to-end learning local multi-view descriptors for 3D point clouds.  In _CVPR_ , 2020.
  * Li et al. [2022] Lei Li, Hongbo Fu, and Maks Ovsjanikov.  Wsdesc: Weakly supervised 3d local descriptor learning for point cloud registration.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022.
  * Li et al. [2019] Zhigang Li, Gu Wang, and Xiangyang Ji.  Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In _ICCV_ , 2019.
  * Liu et al. [2023] Jian Liu, Wei Sun, Chongpei Liu, Xing Zhang, and Qiang Fu.  Robotic continuous grasping system by shape transformer-guided multi-object category-level 6d pose estimation.  _IEEE Transactions on Industrial Informatics_ , 2023.
  * Nguyen et al. [2023] Van Nguyen Nguyen, Tomas Hodan, Georgy Ponimatkin, Thibault Groueix, and Vincent Lepetit.  CNOS: A strong baseline for CAD-based novel object segmentation.  In _ICCV-W_ , 2023.
  * Okorn et al. [2021] Brian Okorn, Qiao Gu, Martial Hebert, and David Held.  Zephyr: Zero-shot pose hypothesis rating.  In _ICRA_ , 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv:2304.07193_ , 2023.
  * Peng et al. [2023] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al.  Openscene: 3d scene understanding with open vocabularies.  In _CVPR_ , 2023.
  * Poiesi and Boscaini [2023] Fabio Poiesi and Davide Boscaini.  Learning general and distinctive 3D local deep descriptors for point cloud registration.  _IEEE TPAMI_ , 2023.
  * Qi et al. [2017] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _NeurIPS_ , 2017.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 2021.
  * Rusu et al. [2009] Radu Bogdan Rusu, Nico Blodow, and Michael Beetz.  Fast point feature histograms (FPFH) for 3D registration.  In _ICRA_ , 2009.
  * Shugurov et al. [2022] Ivan Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilic.  OSOP: A Multi-Stage One Shot Object Pose Estimation Framework.  In _CVPR_ , 2022.
  * Su et al. [2019] Yongzhi Su, Jason Rambach, Nareg Minaskan, Paul Lesur, Alain Pagani, and Didier Stricker.  Deep multi-state object pose estimation for augmented reality assembly.  In _ISMAR-Adjunct_ , 2019.
  * Su et al. [2022] Yongzhi Su, Mahdi Saleh, Torben Fetzer, Jason Rambach, Nassir Navab, Benjamin Busam, Didier Stricker, and Federico Tombari.  Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation.  In _CVPR_ , 2022.
  * Sun et al. [2022] Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou.  Onepose: One-shot object pose estimation without cad models.  In _CVPR_ , 2022.
  * Sundermeyer et al. [2023] Martin Sundermeyer, Tomáš Hodaň, Yann Labbe, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiří Matas.  Bop challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.  In _CVPR_ , 2023.
  * Thalhammer et al. [2023a] Stefan Thalhammer, Dominik Bauer, Peter Hönig, Jean-Baptiste Weibel, José García-Rodríguez, and Markus Vincze.  Challenges for monocular 6d object pose estimation in robotics.  _arXiv:2307.12172_ , 2023a.
  * Thalhammer et al. [2023b] Stefan Thalhammer, Peter Hönig, Jean-Baptiste Weibel, and Markus Vincze.  Open challenges for monocular single-shot 6d object pose estimation.  _arXiv:2302.11827_ , 2023b.
  * van der Maaten and Hinton. [2008] L.J.P. van der Maaten and G.E. Hinton.  Visualizing high-dimensional data using t-sne.  _Journal of Machine Learning Research_ , 2008.
  * Wang et al. [2019] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese.  Densefusion: 6d object pose estimation by iterative dense fusion.  In _CVPR_ , 2019.
  * Wang et al. [2021] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  GDR-Net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In _CVPR_ , 2021.
  * Wang et al. [2023] Ruiqi Wang, Xinggang Wang, Te Li, Rong Yang, Minhong Wan, and Wenyu Liu.  Query6dof: Learning sparse queries as implicit shape prior for category-level 6dof pose estimation.  In _ICCV_ , 2023.
  * Xiang et al. [2018] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  PoseCNN: A Convolutional Neural Network for 6D object pose estimation in cluttered scenes.  In _Robotic Science and Systems_ , 2018.
  * Yang et al. [2017] J. Yang, Q. Zhang, Y. Xiao, and Z.-G. Cao.  TOLDI: An effective and robust approach for 3D local shape description.  _Patt. Recogn._ , 2017.
  * Yang et al. [2023] Jun Yang, Wenjie Xue, Sahar Ghavidel, and Steven L Waslander.  6d pose estimation for textureless objects on rgb frames using multi-view optimization.  In _ICRA_ , 2023.
  * Ze and Wang [2022] Yanjie Ze and Xiaolong Wang.  Category-level 6d object pose estimation in the wild: A semi-supervised learning approach and a new dataset.  _NeurIPS_ , 2022.
  * Zhao et al. [2022] Hengwang Zhao, Hanyang Zhuang, Chunxiang Wang, and Ming Yang.  G3doa: Generalizable 3d descriptor with overlap attention for point cloud registration.  _RA-L_ , 2022.
  * Zhao et al. [2023] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang.  Fast segment anything.  _arXiv:2306.12156_ , 2023.
  * Zhao et al. [2019] Y. Zhao, T. Birdal, H. Deng, and F. Tombari.  3D point capsule networks.  In _CVPR_ , 2019.
  * Zhu et al. [2023] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao.  PointCLIPv2: Prompting CLIP and GPT for Powerful 3D Open-world Learning.  In _ICCV_ , 2023.
  * Zhuang et al. [2023] Chungang Zhuang, Shaofei Li, and Han Ding.  Instance segmentation based 6d pose estimation of industrial objects using point clouds for robotic bin-picking.  _Robotics and Computer-Integrated Manufacturing_ , 2023.
  * Zou et al. [2024] Lu Zou, Zhangjin Huang, Naijie Gu, and Guoping Wang.  Learning geometric consistency and discrepancy for category-level 6d object pose estimation from point clouds.  _Pattern Recognition_ , 2024.