传统的基于草图的图像检索(Sketch Based Image Retrieval，SBIR)主要关注类別层次的检索，忽略了类别内的细节变化。这是不理想的，因为如果给出特定的鞋子草图(例如，高跟鞋，脚趾开口)作为查询，它可以返回任何鞋，包括具有不同语义部件的鞋(例如，平底的跑鞋)。基于草图的精细阁像检索(Fine-grained Sketch Based Image Retrieval，FG-SBIR)正在成为超越传统类别层次的基于草图的图像检索的一种方式，并充分利用草图中可传达的细节。通过提供比无处不在的文本类别搜索更具表现力的交互模式，FG-SBIR是更有可能支持任何实际商业采用的图像检索技术。 我们研究FG-SBIR，通过执行实例层次(而不是类别层次)检索，FG-SBIR体现了实时实用的应用程序。有三个因素造成了FG-SBIR的挑战性：(ⅰ)自由手绘草图本质上是抽象的和标志性的，使得与照片精细的视觉比较变得困难，(ⅱ)草图(黑色线条和白色背景组成)和照片(彩色像素点组成)来着两个不同的视觉领域，(ⅲ)当跨模态和抽象等级时，精细的区别尤其具有挑战性。为了桥接草图和图像之间巨大的跨域间隙，(a)我们提出了精细的属性，并利用部件感知的属性预测算法解除了属性之间虚假的相关性从而准确的预测属性。现有的跨模态特征校正方法都侧重于类别层次的任务，不适合实力层次的检索任务，(b)我们提出了一种协同子空间和实例层次的对齐算法，该算法同时校正草图和图像的域子空间和草图和图像对之间的特征。现有FG-SBIR模型旨在学习嵌入空间，在这个空间中可以直接比较草图和照片。虽然很成功，但它们学习到的嵌入空间是特定域的，因此这些模型不能跨类别推广，没法在新颖的物体类别上取得好的表现。这限制了FG-SBIR的实际适用性。(c)我们提出了一个FG-SBIR的泛化模型，该模型能够很好的泛化到未训练的类别上。(d)我们提出了一个通用的深度草图感知编组器，该编组器可以从图像的边缘图(Edgemap)中生成类似草图训练FG-SBIR模型，以此来泛化到无草图-图像对的类别上。 FG-SBIR最大的挑战是草图和图像之间的语义间隙，我们利用精细的属性来桥接这个间隙。但是由于虚假的相关性，很难准确的预测精细的属性。为了解决这个问题，(ⅰ)我们贡献了一个FG-SBIR数据集，并对图像和草图的部件和部件层次的属性进行标注。在这个数据集的帮助下，我们研究了(ⅱ)如何学习强监督的可变形组件模型，随后能够自动预测部件层次属性，并提供草图和图像之间的姿势对齐。最后(ⅲ)将这些多层次特征组合在一个匹配的框架中进行检索。在FG-SBIR数据集上进行的大量实验证明了所提出方法的有效性。 草图和图像来自两个本质不同的域，草图和图像的精细对齐是极具挑战性的。为了解决这个问题，我们提出了一种新的同时对齐子空间和实例层次的方法，该方法利用特征子空间和实例层次提示来更好地对齐草图和图像域。通过大量的实验证明我们的方法在FG-SBIR中的表现远远优于现有跨模态匹配算法和域自适应方法。同时我们还和FG-SBIR中常用的深度学习方法进行了对比，取得了相媲美的结果。 现有的FG-SBIR方法都无法泛化到新颖的类别上，为此我们提出一种新颖的无监督学习的解决方案，用于模拟原型视觉草图特征的通用特征描述符。然后可以使用该特征描述符来对草图/照片特征表述的学习进行参数化，使模型自动适应新颖的类别。在两个最大的FG-SBIR数据集Sketchy和QMUL-Shoe-V2的实验证明了我们的方法在实现FG-SBIR的跨类别推广方面的功效。 FG-SBIR的数据库的收集费时费力。我们通过对草图感知编组的研究来侧面解决这个问题。具体的，(ⅰ)我们提供了迄今为止最大的草图感知编组数据集，包括均匀分布在25个物体类别上的20，000个独特草图。(ⅱ)我们开发一个通用的草图感知编组器。大量实验表明，我们所提出的模型明显优于最先进的编组器和语义分割模型。(ⅲ)我们的编组器作为一个抽象模型从图像的边缘图中生成类似草图，利用生成草图和原始图像训练FG-SBIR模型，并在大型FG-SBIR数据集QMUL-Shoe-V2和Chair-V2上实验证明我们的方法的优越性能。 关键词：基于草图的精细图像检索 实例层次 子空间 跨模态泛化模型 通用草图感知编组

 Traditional sketch-based image retrieval SBIR mainly focus on category-level retrieval, where intra-category variations are neglected. This is not ideal, since if given a specific shoe sketch (e.g., high-heel, toe-open) as query, it can return any shoe, including those with different part semantics (e.g., a flat running shoe). Thus fine-grained sketch-based image retrieval (FG-SBIR) is emerging as a way to go beyond conventional category-level SBIR, and fully exploit the detail that can be conveyed in sketches. By providing a mode of interaction that is more expressive than the ubiquitous browsing of textual categories, FG-SBIR is more likely to underpin any practical commercial adoption of SBIR technology. We study the problem of fine-grained sketch-based image retrieval. By performing instance-level (rather than category-level) retrieval, it embodies a timely and practical application.Three factors contribute to the challenging nature of the problem: (ⅰ) free-hand sketches are inherently abstract and iconic, making visual comparisons with photos difficult, (ⅱ) sketches and photos are in two different visual domains, i.e. black and white lines vs. color pixels, and (ⅲ) fine-grained distinctions are especially challenging when executed across domain and abstraction-level. In order to bridge the cross domain gap between sketch and image, (a) we proposed fine-grained attributes and proposed a part-aware method to predict the attributes, which can decorrelate semantic visual attributes. Existing cross domain alignment methods focus on category-level, which are not fit for Fine-grained task, (b) we proposed a synergistic instance-level subspace alignment, which can both align subspace and instance-level cues. Existing FG-SBIR models aim to learn an embedding space in which sketch and photo can be directly compared. While successful, Since the learned embedding space is domain-specific, these models do not generalise well across categories. This limits the practical applicability of FG-SBIR. (c) we proposed a Generalising FG-SBIR method, which can improve the performance on the Unseen categories, (d) we proposed a deep universal sketch perceptual grouper, which can be used on the edgemap of photos to synthesise human-like sketches to retrain the FG-SBIR model. Such that the FG-SBIR model can generalise to novel categories without free-hand sketches. The hardest challenging in FG-SBIR is the semantic gap, we use attributes to bridge it. However, they suffer from being hard to predict due to spurious correlations. In order to address those, (ⅰ) we contribute a FG-SBIR dataset, where sketch and image are annotated with its semantic parts and associated part-level attributes. With the help of this dataset, we investigate (ⅱ) how strongly-supervised deformable part-based models can be learned that subsequently enable automatic detection of part-level attributes, and provide pose-aligned sketch-image comparisons. Finally (ⅲ) those multi level features are combined in a matching framework integrating. Extensive experiments conducted on FG-SBIR datasets demonstrate effectiveness of the proposed method. Sketch and image are inherently different domain, it is very challenging to align them in instance-level. In order to address those, we propose a novel method for instance-level domain-alignment, that exploits both subspace and instance-level cues to better align the domains. Extensive experiments conducted on FG-SBIR datasets demonstrate effectiveness of our method comparing with other cross-model matching methods and domain-adaptation method and even comparing with the deep learning methods in FG-SBIR. Existing FG-SBIR methods could not generalize to unseen categories , so (i)we propose a novel unsupervised learning approach to model a universal manifold of prototypical visual sketch traits for the first time. This manifold can then be used to paramaterise the learning of a sketch/photo representation to adapt to novel categories. Experiments on the two largest FG-SBIR datasets, Sketchy and QMUL-Shoe-V2, demonstrate the efficacy of our approach in enabling cross-category generalisation of FG-SBIR. It is time-consuming to collect an FG-SBIR database, we lateral address it with deep sketch perceptual grouping, specifically,(ⅰ) we contribute the largest sketch perceptual grouping (SPG) dataset to date, consisting of 20, 000 unique sketches evenly distributed over 25 object categories, (ⅱ) We developed a universal sketch grouper. We show that the proposed model significantly outperforms the state-of-the-art groupers, (ⅱ) our grouper is to use it as an abstraction model so that edgemaps extracted from photos can be grouped and abstracted to synthesise human-like sketches to train a state-of-the-art FG-SBIR model without using any real human sketches. We conduct experiment on the largest FG-SBIR datasets QMUL Shoe-V2 and Chair-V2 to demonstrate effectiveness of the proposed method. KEY WORDS: Fine-grained Sketch-based Image Retrieval Instance-level Subspace Cross-modal Generalising model Universal Sketch Perceptual Grouping