As reported on Wired.
- BY CADE METZ
The Mac Mini. Image: Tiziano Caviglia/Flickr
If you walk into one of the massive data centers that underpin Facebook’s ever-expanding online empire, you’ll find tens of thousands of computer servers. Many of them are designed by Facebook, and most run Linux, the open source operating system that drives so much of the modern web. But those aren’t the only machines you’ll find.
You’ll also find Mac Minis. Lots and lots of Mac Minis.
The Apple Mac Mini is a small, square computer sold without a display, keyboard, or mouse. It looks kinda like a makeup case with an Apple logo on it. The thing was conceived as a home machine. You could toss it onto your desk and attach peripherals already lying around the house. But nearly a decade after its debut, the Mini has assumed a very different role at Facebook. Mark Zuckerberg and Co. operate a vast fleet of Mac Minis so they have a place to test the Facebook iOS app — the app that lets hundreds of millions of people use the world’s most popular social network from the iPhone and the iPad.
Facebook’s iOS app is so complex — designed to run on so many mobile devices across the world, and built with code written by so many Facebook engineers — that the company needs an enormous network of machines to test it. “We run hundreds of thousands of tests every couple of hours,” says Christian Legnitto, the ex-Apple man who oversees the Facebook engineering team responsible for the final release of the company’s mobile apps. “When you’re shipping [an app] to hundreds of millions of users, those one-in-a-million bugs can happen hundreds of times.”
But because Apple insists that all Apple software run on Apple machines, Facebook can’t test its iOS app on the Linux servers that drive the rest of its empire — or on the popular cloud services offered by tech giants like Amazon or Microsoft. So, says Legnitto, the company operates “racks and racks” of Mac Minis that run Apple’s Mac OS X operating system.
The setup is part of a small but burgeoning trend in the world of software development. Traditionally, coders have tested iPhone, iPad, and Mac apps on a single machine or handful of machines — a Mac laptop or a few Mac Minis. But as apps become larger and more complex, tech outfits are beginning to use Apple computer farms that can run tests on a much larger scale. Facebook has its Mac Mini farms, while specialized cloud services such as Travis CI are springing up, letting anyone test Mac and iOS apps by way of the internet.
The cloud revolution has made it far easier for coders and businesses to test and run computer software. With popular online services such as Amazon EC2, the Rackspace Cloud, and Microsoft Azure, you can instantly spin up virtual computer servers — servers that exist only as software — and using these virtual machines, you can test almost anything. But these services can’t run Apple software. It’s only recently that niche services such as Travis CI have begun to fill the gap.
“Doing testing in the cloud with Linux has been typically easy, but doing it with Mac hasn’t,” says Josh Kalderimis, one of the engineers who oversee Travis CI. “If you wanted to do large scale, you had to do a build-your-own setup.”
That’s what Facebook did. In theory, the company could have spun up virtual machines on its Linux servers, loaded those virtual machines with Apple operating systems, and tested its iOS app there. But Apple’s licensing terms won’t allow it. “We play it by the book,” Legnitto says. So Mac Minis it is. Apple now sells these tiny machines with a server version of Mac OS X for about $1000, but we’re guessing Facebook gets some sort of volume discount.
Legnitto could not say how many Minis the company uses, but he did say you’ll find Apple machines in multiple Facebook data centers. They company now operates its own data centers in Oregon, North Carolina, and Sweden, and it rents additional data center space in Virginia and the San Francisco Bay Area.
It might seem strange that Facebook would use Mac Minis to test the app, rather than beefier Apple machines. But the Mini is now the only bona fide computer server that Apple offers, and there’s ample compute power inside, including a 2.3GHz quad-core processor and 4GB memory. If you put enough of these machines together, they become rather powerful.
This is how computing works nowadays. Companies like Facebook and Google and Amazon operate their online empires using what are essentially vast supercomputers made from hundreds of relatively modest, inexpensive machines. Because it offers only the bare essentials, the Mini is sometimes a perfect fit. Mozilla, maker of the Firefox browser, runs Minis inside its data center, and a San Francisco startup called Imgix uses a large network of Minis to rapidly re-size, re-crop, and re-format digital images.
If you’re not quite the size of Facebook but still need a means of testing an iOS app or Mac app across a larger number of machines, you can tap a cloud testing service like Travis CI. CI is short for continuous integration, where you constantly build and test software as new code is created.
Using open source testing software originally developed at Facebook — an platform called xctool — the German firm tests iOS and Mac apps on a Mac hardware farm operated by another outfit called Sauce Labs. That’s the way it works in the cloud world. One service often runs on another. In any event, Travis CI is a place where you can test Apple apps across a large number of virtual machines without setting up your own hardware.
The Sauce Labs machines behind this service aren’t Mac Minis. They’re Mac Pros. But the concept is the same. You can test iOS apps on a large network of Apple machines. Sauce Labs also offers its own service for testing software, including iOS apps.
These may seem like niche services. But Travis CI is widely used in the world of open source software. Over 93,000 open source projects have used the service, and about 1,000 of those are Mac projects. As iOS and Mac apps continues to evolve, such tools will only become more important. Legnitto will tell you that Facebook is a unique case, that few other iOS operations are so large and complex. But it’s a harbinger of things to come.
A rack of Mac Minis inside Mozilla’s data center in Santa Clara, California. Photo: Ariel Zambelich