Face Averages

Who is the meanest person?

Face averaging is a novel concept that smashes a few newer technologies with some ancient ones. It all started for me about 2 years ago when I got a job at a pizza shop. I quickly uncovered that the employee photo database had sequential IDs which allowed me to download every employee's photo. That's a lot of information, and I wanted to turn it into something meaningful, which is when I arrived at the idea of smashing all 30000 employees into one picture.
A soft male face with a black hat a shirt
There's a lot you can gather from this single image, for instance, most seem to be wearing the black hat and shirt as part of the uniform for it to come out so clearly black in the image. You also seem to see that the face is mostly male, showing the skew in the industry. Fascinating picture, too bad it took me weeks to understand the process first time round.
I'm sure we've all seen a face morph at some point, this is essentially face morphing on a much larger scale. Come to last month, I'm on the train to work and something interesting comes to mind. "I wonder what the average Sydneysider looks like" and a plan quickly came together. Due to the density of the city, I'm walking past a lot of faces in any given day. If I slipped my phone into my breast-pocket I could easily film all these faces without looking too out of place.
Three frames of a morph between George Bush and Arnold Schwarzenegger.
That's George Bush and Arnold Schwarzenegger
You could argue it's a privacy concern covertly filming people but it's perfectly legal here if it's for personal use and it's in public. Tick and tick. Very quickly I held hundreds of gigs of video of the people around Sydney, now to put it to use. I needed to suck out faces from the video which involved breaking the video into frames meaning that each second of video netted me 60 photos. It ads up quick. I figure, to save me time and space, I'll only get every 15th frame or four a second. With these frames, we throw them into a machine learning neural network trained purely on detecting faces in a photo. It did its job a little too well, it got faces that I couldn't even see, so I limited extraction to a minimum size.
Shows 200,000 files in a folder
Yeah, well that got messy very quickly
This sounds nice and easy until I realised that if a person is in more than one frame, they'll be saved a couple times, and that's going to skew the average. So not only do I have to detect faces, I have to detect if we've seen it before. To remedy this, when I grab a face, I calculate its fingerprint (a hash that will be the same even if the face isn't in the same position or rotation) and compare it with all other fingerprints to see if we've seen it before. Problem solved.
Shows a collage of collected faces
I guess you could call it a facial database, and if some bloke with a weekend can put this together, who's to know what else is out there
[?] I think in Australia both Perth and Hobart now have facial recognition CCTV networks
Remember when I mentioned this used some older technologys? Well that's where Delaunay triangulation comes into play. Curtly, it's mapping a bunch of points in such a way that triangles are formed with no other points inside. We use this with another nueral network specifically for mapping out 68 points on a human face. We're talking points like "left eye corner" and "tip of nose". Once we've fed all our images into this network we have a bunch of face photos and a relevant text file identifying the position of the 68 points in the image.
Shows a bunch of lines matching the facial features of two people
Here's an example with two people, and only 1/4 the points. Source
From there, it's essentially just "squish and stretch the photo around these points in such a way that all faces are exactly aligned". With all faces aligned then we can divide up the opacity so each photo has an equal portion which results in our final blended image of a single, average face. It was only at that point I realised "I didn't have to go through all this trouble, Google images exists. So I went back and edited the script in such a way where I can feed it a page of Google Image results as the faces to average. This allowed me to quickly make the facial average of "Australian male" or "New York City female" or anything.
Shows the average face of a New York City female resident
Here's your average NYC woman
If the nitty gritty script and implementation tickles your fancy, or you just want to give her a spin yourself, I implore you to give it a geese! Otherwise enjoy this photo of all US presidents.
Shows the average face of all US presidents
Despite most portraits being paintings, the face is very human, if a little sad looking