Zero-Shot Everything Sketch-Based Image Retrieval,
and in Explainable Style

Fengyin Lin^1*     Mingkang Li^1*     Da Li^2†     Timothy Hospedales^2,3     Yi-Zhe Song⁴     Yonggang Qi¹

¹Beijing University of Posts and Telecommunications     ²Samsung AI Centre, Cambridge
³University of Edinburgh     ⁴SketchX, CVSSP, University of Surrey

CVPR 2023

arXiv

     

Dataset

     

Code

Figure 1. Attentive regions of self-/cross-attention and the learned visual correspondence for tackling unseen cases. (a) The proposed retrieval token [Ret] can attend to informative regions. Different colors are attention maps from different heads. (b) Cross-attention offers explainability by explicitly constructing local visual correspondence. The local matches learned from training data are shareable knowledge, which enables ZS-SBIR to work under diverse settings (inter- / intra-category and cross datasets) with just one model. (c) An input sketch can be transformed into its image by the learned correspondence, i.e., sketch patches are replaced by the closest image patches from the retrieved image.

Introduction

This paper studies the problem of zero-short sketch-based image retrieval (ZS-SBIR), however with two significant differentiators to prior art (i) we tackle all variants (inter-category, intra-category, and cross datasets) of ZS-SBIR with just one network ("everything"), and (ii) we would really like to understand how this sketch-photo matching operates ("explainable"). Our key innovation lies with the realization that such a cross-modal matching problem could be reduced to comparisons of groups of key local patches - akin to the seasoned "bag-of-words" paradigm. Just with this change, we are able to achieve both of the aforementioned goals, with the added benefit of no longer requiring external semantic knowledge.

Technically, ours is a transformer-based cross-modal network, with three novel components (i) a self-attention module with a learnable tokenizer to produce visual tokens that correspond to the most informative local regions, (ii) a cross-attention module to compute local correspondences between the visual tokens across two modalities, and finally (iii) a kernel-based relation network to assemble local putative matches and produce an overall similarity metric for a sketch-photo pair. Experiments show ours indeed delivers superior performances across all ZS-SBIR settings. The all important explainable goal is elegantly achieved by visualizing cross-modal token correspondences, and for the first time, via sketch to photo synthesis by universal replacement of all matched photo patches.

Our Solution

Figure 2. Network overview.

As shown in Figure 2, (a) Learnable tokenization generates structure preserved tokens, preventing the generation of uninformative tokens. (b) Self-attention finds the most informative regions ready for local matching. (c) Cross-attention learns visual correspondence from visual tokens. A retrieval token [Ret] is added as a supervision signal during training. (d) Token-level relation network enables to explicitly measure the correspondences of cross-modal token pairs. Pairs of removed tokens as per token selection will not be counted.

Experiments and Results

Category-level ZS-SBIR

Bibtex