Face Morphing

I. Introduction

A morph of two images is a combination of warping an image's shape to the other, and cross-dissolving the images' colours. This allows us to morph one object into another seamlessly as long as important features are preserved by defining corresponding points between the images. This can be used to morph one face into another, by outputting several in between morphs and compiling them into a gif or video and displayed as a morph sequence. It can also be used to find the average face of two faces. The mean of a population of faces can also be computed with the same idea by averaging out a whole collection of faces. These mean faces can be used to make caricatures, which is similar to computing the opposite of face morphing.

II. Morph Sequence

The basic idea behind creating a morph sequence is to create a series of "mid-way faces" that starts as the first image, and gradually becomes the second image. We will consider for now, the case where the "weight" of the images is 50. (50% the first face, and 50% the second face. The weight is the weight applied to the first image. So, the output would be p1 + (weight * (p2 - p1)).) The difficulty is warping the shapes of the faces.

Stretching an image around willy nilly will inadvertently warp features in the face that we do not want distorted. We also want these features on one image to match up with the same features on the second image. So, we would like to specify a series of points that are important, and a second series of the corresponding points of the second image. This is simply two arrays of coordinate tuples. I simply chose points to go around the eyebrow, nose, eyes, mouth, and face.

The background colour, pose, and angle of the images make a big difference in how the result looks. We want to choose images that are similar with respect to these traits. I chose a portrait of myself and then found a similar image of Emma Watson. Although the backgrounds are quite different, they are not too distractingly different. The portrait of Emma Watson is from the September 2015 issue of Vogue UK.

After defining the points, we must construct a triangulation of the points for each image. We also construct a triangulation of our goal face. The goal face in this case is the points halfway between the two faces at (p1 + p2) / 2, where p1 is the points of image one, and p2 is the points of image two. The images below show the points and triangulations for both images.

The triangles give us much smaller portions of the image to transform. We can use Affine warp to transform one triangle into a corresponding triangle. This is done by multiplying a constructed Affine warp matrix M of one triangle to another with a vector [[x], [y], [1]], where x and y are pixels in the new image, and the resulting vector [[x'], [y'], [1]] tells us which pixel of the original image to reference.

After warping both images using Affine warps on each individual triangle for each image, simply add the appropriate weight (in this case .5 and .5) of the images together.

After being able to morph a face into some mid-way face, a morph sequence can easily be created by adjusting the weight applied to the images and their points. In order to create a 46 frame warp, simply generate with weights 0, 1/46, 2/46, ..., 1. The morph with weight 0 will be image one, and the morph with weight 1 will be image two.

I also morphed the faces of President Obama and Angelina Jolie. These images are portraits taken by Martin Schoeller. They are chosen for their quality and similarity in posture.

II. "Mean" Face of a Population

Now that we are able to get the average of two face, we can consider getting the average of many different faces. Doing this is mathematically simple. For example, when averaging the faces of ten people (n = 10), let the weight of each face be 1/n, and proceed as before. The tricky part is selecting the images to be used and having them all aligned and using the same point selection system.

Thankfully, there are databases out there that have this available. I used the FEI Face Database, which is a database of Brazilian faces that come with files that have points for the eyebrows, eyes, nose, mouth, and face selected already. Unfortunately, this database has this is black and white, so the following images will not be in colour, but the same process can easily be applied to images with colour.

Selecting a meaningful subset also helps to produce interesting results. Choosing a specific population subset such as Brazilians, male Brazilians, or female Brazilians, gives us meaningful results that tells us something about that subset. I ran my program on ten males, ten males, and a combination of seven female and seven males to produce three different mean faces.

There are many interesting things that can be done with mean faces. We can take the shape of this mean shape, and morph another face's shape into the mean facial shape to make that person look like someone of that population.

Below, I attempted to make myself look like a Brazilian female. At first, I didn't add any of the mean face's colours, but it made the result look extremely unnatural and sharp in some places. So, I then used only some percentage of my face after shape warping and added some of the colouration of the mean face to it. This smooth out many odd shapes and provided a more pleasing result. I did the same thing to morph the Brazilian female into my facial shape.

I also took the composite photos of my pledge class and found the mean of our faces. Since there are eight boys and only four girls, the mean looks very masculine. Since nine of us are Asian, the result also looks very asian.

IV. Ethnicity Change

Changing the ethnicity of the face is quite simple, as long as we have some average face of that ethnicity. I was able to find some European average images on this blog. There are three ways that this can be done: change the face's colouration to the average image's colouration, change the face's shape to the average image's shape, and change both the colouration and the shape of the image.

At first I tried using entirely my own colouration for the colour change, but it did not look at me at all. After experimenting, I decided that 70% my colouration and 30% the average face colouration gave the right balance such that it is recognizable as my face while using as much of the average face colouration as possible. Shape change looked disfigured when it used wholly my couration, so I experimented and settled on using 70% my colouration and 30% the average face colouration as well.

I think that changing both shape and colouration of my face came out with the most pleasing result. Although I do look more European, I definitely do not look European. This implementation can really only make one look half of their original race and half the intended race, not fully the intended race.