Deep neural networks are remaking the Internet. Able to learn very human tasks by analyzing vast amounts of digital data, these artificially intelligent systems are injecting online services with a power that just wasn't viable in years past. They're identifying faces in photos and recognizing commands spoken into smartphones and translating conversations from one language to another. They're even helping Google choose its search results. All this we know. But what's less discussed is how the giants of the Internet go about building these rather remarkable engines of AI.
Part of it is that companies like Google and Facebook pay top dollar for some really smart people. Only a few hundred souls on Earth have the talent and the training needed to really push the state-of-the-art forward, and paying for these top minds is a lot like paying for an NFL quarterback. That's a bottleneck in the continued progress of artificial intelligence. And it's not the only one. Even the top researchers can't build these services without trial and error on an enormous scale. To build a deep neural network that cracks the next big AI problem, researchers must first try countless options that don't work, running each one across dozens and potentially hundreds of machines.
"It's almost like being the coach rather than the player," says Demis Hassabis, co-founder of DeepMind, the Google outfit behind the history-making AI that beat the world's best Go player. "You're coaxing these things, rather than directly telling them what to do."
That's why many of these companies are now trying to automate this trial and error---or at least part of it. If you automate some of the heavily lifting, the thinking goes, you can more rapidly push the latest machine learning into the hands of rank-and-file engineers---and you can give the top minds more time to focus on bigger ideas and tougher problems. This, in turn, will accelerate the progress of AI inside the Internet apps and services that you and I use every day.
In other words, for computers to get smarter faster, computers themselves must handle even more of the grunt work. The giants of the Internet are building computing systems that can test countless machine learning algorithms on behalf of their engineers, that can cycle through so many possibilities on their own. Better yet, these companies are building AI algorithms that can help build AI algorithms. No joke. Inside Facebook, engineers have designed what they like to call an "automated machine learning engineer," an artificially intelligent system that helps create artificially intelligent systems. It's a long way from perfection. But the goal is to create new AI models using as little human grunt work as possible.
After Facebook's $104 billion IPO in 2012, Hussein Mehanna and other engineers on the Facebook ads team felt an added pressure to improve the company's ad targeting, to more precisely match ads to the hundreds of millions of people using its social network. This meant building deep neural networks and other machine learning algorithms that could make better use of the vast amounts of data Facebook collects on the characteristics and behavior of those hundreds of millions of people.
According to Mehanna, Facebook engineers had no problem generating ideas for new AI, but testing these ideas was another matter. So he and his team built a tool called Flow. "We wanted to build a machine-learning assembly line that all engineers at Facebook could use," Mehanna says. Flow is designed to help engineers build, test, and execute machine learning algorithms on a massive scale, and this includes practically any form of machine learning---a broad technology that covers all services capable of learning tasks largely on their own.
Basically, engineers could readily test an endless stream of ideas across the company's sprawling network of computer data centers. They could run all sorts of algorithmic possibilities---involving not just deep learning but other forms of AI, including logistic regression to boosted decision trees---and the results could feed still more ideas. "The more ideas you try, the better," Mehanna says. "The more data you try, the better." It also meant that engineers could readily reuse algorithms that others had built, tweaking these algorithms and applying them to other tasks.
Soon, Mehanna and his team expanded Flow for use across the entire company. Inside other teams, it could help generate algorithms that could choose the links for your Faceboook News Feed, recognize faces in photos posted to the social network, or generate audio captions for photos so that the blind can understand what's in them. It could even help the company determine what parts of the world still need access to the Internet.
With Flow, Mehanna says, Facebook trains and tests about 300,000 machine learning models each month. Whereas it once rolled a new AI model onto its social network every 60 days or so, it can now release several new models each week.
The idea is far bigger than Facebook. It's common practice across the world of deep learning. Last year, Twitter acquired a startup, WhetLab, that specializes in this kind of thing, and recently, Microsoft described how its researchers use a system to test a sea of possible AI models. Microsoft researcher Jian Sun calls it "human-assisted search."
Mehanna and Facebook want to accelerate this. The company plans to eventually open source Flow, sharing it with the world at large, and according to Mehanna, outfits like LinkedIn, Uber, and Twitter are already interested in using it. Mehanna and team have also built a tool called AutoML that can remove even more of the burden from human engineers. Running atop Flow, AutoML can automatically "clean" the data needed to train neural networks and other machine learning algorithms---prepare it for testing without any human intervention---and Mehanna envisions a version that could even gather the data on its own. But more intriguingly, AutoML uses artificial intelligence to help build artificial intelligence.
As Mehana says, Facebook trains and tests about 300,000 machine learning models each month. AutoML can then use the results of these tests to train another machine learning model that can optimize the training of machine learning models. Yes, that can be a hard thing to wrap your head around. Mehanna compares it to Inception. But it works. The system can automatically chooses algorithms and parameters that are likely to work. "It can almost predict the result before the training," Mehanna says.
Inside the Facebook ads team, engineers even built that automated machine learning engineer, and this too has spread to the rest of the company. It's called Asimo, and according to Facebook, there are cases where it can automatically generate enhanced and improved incarnations of existing models---models that human engineers can then instantly deploy to the net. "It cannot yet invent a new AI algorithm," Mehanna says. "But who knows, down the road..."
It's an intriguing idea---indeed, one that has captivated science fiction writers for decades: an intelligent machine that builds itself. No, Asimo isn't quite as advanced---or as frightening---as Skynet. But it's a step toward a world where so many others, not just the field's sharpest minds, will build new AI. Some of those others won't even be human.
