Google AI now can give YouTube videos a wacky background

Machine learning proves its worth for new video effects tech: distinguishing between faces and backgrounds at 100 frames per second.

Google's AI software can detect the subject of a
video and change the background.
Google

Google's artificial intelligence researchers have a new trick up their sleeves: giving selfie videos a new background the way you can with still photos today.

Switching out background scenery is a decades-old technology, but it's generally complicated and time-consuming -- think of making movies with superheros acting in front of green screens so computers can later replace the uniform green color with digital effects like exploding buildings. Having 3D scene data, for example extracted from newer iPhones' twin cameras, makes the process only a little easier.

Google, though, trained a neural network with lots of carefully labeled imagery that helped it learn how to distinguish facial features -- eyes, hair, glasses, mouths, and so on -- from everything else. The result is a system that can swap out backgrounds fast enough to keep up with video. Digital video frames whip by at 30 frames per second, but Google's technology works at 40 frames per second on a Google Pixel 2 phone and more than 100 frames per second on an Apple iPhone 7.

Want to try it out for yourself? Sorry. For now, only a pretty small number of high-profile YouTubers with access to the YouTube stories service, which offers an ability to share Snapchat-esque short videos.

Google detailed the research in a blog post Thursday.

"Our new segmentation technology allows creators to replace and modify the background, effortlessly increasing videos' production value without specialized equipment," Google programmers Valentin Bazarevsky and Andrei Tkachenka said in the blog post.

Google uses machine learning in its video segmentation technology that can determine background imagery that can be replaced with something funny or eye-catching.
Google

The technology shows how versatile AI and machine-learning technology is for solving complex computational problems. Neural network technology doesn't have to know endless rules, like "faces have two eyes, located above the nose, unless the face is viewed in profile, in which case there may only be one eye visible, unless the subject is wearing dark glasses." It just has to be trained on enough photos, labeled by humans who do actually know what a face is, that it eventually learns the patterns. It's proven a remarkably useful technology for everything from screening out email spam to predicting what word you're trying to type into your phone.

Google already uses AI in a related feature today on its Google Pixel 2 phones. The phones only have single cameras, but AI helps blur backgrounds for portrait-mode photos.

source:CNet News