Overview of the proposed method. Our method operates on the tentative matches graph (with patches as nodes $P_u, P_v$ and matches as edges) without knowledge of scene and camera geometry. A neural network is used to annotate the edges of this graph with local geometric transformations $T_{u \rightarrow v}, T_{v \rightarrow u}$. Next, the graph is partitioned into tracks, each track containing at most one patch from each image. Finally, the keypoint locations $x_u, x_v$ are refined using a global optimization over all edges.

Abstract

In this work, we address the problem of refining the geometry of local image features from multiple views without known scene or camera geometry. Current approaches to local feature detection are inherently limited in their keypoint localization accuracy because they only operate on a single view. This limitation has a negative impact on downstream tasks such as Structure-from-Motion, where inaccurate keypoints lead to large errors in triangulation and camera localization. Our proposed method naturally complements the traditional feature extraction and matching paradigm. We first estimate local geometric transformations between tentative matches and then optimize the keypoint locations over multiple views jointly according to a non-linear least squares formulation. Throughout a variety of experiments, we show that our method consistently improves the triangulation and camera localization performance for both hand-crafted and learned local features.

Code

The code is available on GitHub at mihaidusmanu/local-feature-refinement.

Paper

	Mihai Dusmanu 1, Johannes L. Schönberger 2, Marc Pollefeys 1, 2 1ETH Zürich, 2Microsoft Multi-View Optimization of Local Feature Geometry In Proceedings of the 2020 European Conference on Computer Vision [Latest version on arXiv] [Long Video] [Short Video]

BibTeX

@InProceedings{Dusmanu2020Multi,
    author = "Dusmanu, Mihai and Sch\"onberger, Johannes L. and Pollefeys, Marc",
    title = "{M}ulti-{V}iew {O}ptimization of {L}ocal {F}eature {G}eometry",
    booktitle = "Proceedings of the 2020 European Conference on Computer Vision",
    year = "2020"
}