Sketch-based Cross-modal Face Recognition

Heterogeneous face recognition(HFR) refers to the problem of matching faces across different visual domains. As a biometric feature recognition technique, HFR can be widely used in identification, multi-media and police fields. Compared to traditional face recognition techniques, HFR can deal with data from different modalities. Therefore, it has less limitations and can be used in more practical problems.The problem of matching facial sketches to photos is commonly known as sketch-based face recognition(SBFR). It has important application in identification of the criminals and caricature-based image search. However,compared to photos, sketches have the following drawbacks: 1) lack of facial details; 2) the exaggeration of facial features; 3) human factors which will influence the similarity between photo and sketch. Meanwhile, since sketches and photos belong to different modalities, features extracted from them are not comparable. Those problems limit the performance of general face recognition techniques in SBFR.In this paper, we investigate sketch-photo face matching and go beyond the well-studied viewed sketches to tackle forensic sketches and caricature where the cross-modal gap is most extreme. Those problems will be analyzed from two angles: cross-modal face recognition system and human factors lying in SBFR. The main contributions can be summarized as follows:(1). Extracting facial structure features based on Delaunay Triangle Rules.We propose to encode faces from different modalities with facial structure features. Instead of encoding gradients, facial structure feature encodes the relative coordinates of facial features. Compared to low level feature, they are more invariant to modality gap. We modify an existing algorithm to detect fiducial points in the sketches and photos. With those fiducial points, Delaunay Triangle rule is used to encode the facial structure feature. Experiment results show that the modified facial point algorithm performs well on the sketches. Meanwhile, the extracted facial structure feature hits higher recognition accuracy than low-level feature.(2). Propose a region-based automated facial attribute detecting algorithm to extract facial attributes from facial sketches.We propose an automated facial attributes detecting algorithm to construct a high-level attribute representation of each facial modality. The idea is that this representation can be learned independently with each modality (thus completely avoiding any cross-modality challenge), but once learned, it is largely invariant to the cross-modal gap. The experiment results show that facial attributes perform better than other features (low level feature and facial structure feature) in SBFR.(3). Develop a Cross-Modal Matching by Facial Attributes (GMMFA)for SBFR.CMMFA encodes the best of both facial attributes and low-level features by learning a CCA subspace the correlates the two. The learned features are then used to tackle cross-modal face recognition problem. Experiment results show that with CMMFA, we got better recognition accuracy than using either facial attributes or low level feature alone.(4). Develop a Multi-Level Feature-Based Framework (MLFBF) for SBFR.We then propose a framework which synergistically exploiting the complementary information contained by low, mid and high-level representations in each domain. Experiment results show that with multi-level feature-based framework, the accuracy of SBFR is even higher than CMMFA.This demonstrates that the only with all three kinds of features: low, middle and high, we can get best SBFR accuracy.(5). Human factors analysis in SBFR.To further boost the matching accuracy in forensic sketch based face recognition, we propose a new facial sketch database, BUPT Face Sketch Database(BUFS), which including three kinds of facial sketches: viewed sketch, time-delayed sketch and unviewed sketch. Based on the proposed BUPT Face Sketch Database(BUFS), we build a model to complement the problems caused by human factors. Experiment results show that the models which considered human factors get better forensic sketch recognition accuracy than other methods.