10/5/2007: Ubuntu a Mooter for global advertising platform
”Open source fosters scalable innovation” – Solutions First assists Mooter Media to develop infinitely scalable global platform using open source software.
by Rodney Gedda, ComputerWorld
10 May 2007
Massively-scalable Internet advertising applications may have been popularized by the likes of Google and Yahoo but local company Mooter Media has developed its own clustered architecture on top of Ubuntu Linux and open source software.
Based in Sydney, the ASX-listed Mooter Media began life as a Internet search start-up before realizing two years ago ad broking was the best way forward.
Mooter’s CIO, Mark Hetherington, was charged with implementing a platform that would scale from 100,000 transactions a day in 2005 to a projected 100 million a day in 2007 and beyond.
Hetherington’s design considerations were that the platform had to be massively-scalable and redundant, and had to be able to support the release of new versions of the software quickly and without interruption.
“We had to move from single server to multiple server and multiple data centres,” Hetherington said. “Our platform of choice was Linux for development reasons and our distribution of choice, Ubuntu, gave us a much more flexible way of managing software in an automated manner. The Debian packages allow you to deploy far more automatically whereas deploying to Red Hat is a pain in the bum.”
With one in-house systems administrator, Mooter contracted Sydney-based consulting firm Solutions First to help achieve its goals.
On top of Ubuntu, Mooter had been using MySQL and Apache Tomcat with Java to facilitate its extreme programming methodology, with a release every two weeks or so.
Mooter’s first challenge was database scalability, and to maintain a configuration data over multiple servers. To achieve this a three-tier architecture was implemented where a central application for updating configuration data replicates changes to all the servers. The “grand central” database is replicated to a “site central” database, which is then replicated to the servers, about 30 in total.
“Our peak was 80 million a day and we are capable of doing well over 100 million a day,” Hetherington said, adding each dual-core server is capable of doing 4 million transactions a day and all are completely redundant.
“The only local information they keep is the clicks they process. If a disk on a server [was lost] we would lose some clicks and revenue but it was deemed affordable to lose a server and a bit of data. If a server goes down there is no impact on the business at all.”
Hetherington is now confident the parallel scalability of the Mooter platform is theoretically “infinitely” scalable.
When Mooter began doing multiple machine deployments it would take two people a whole day to deploy, but by deploying its software with Debian packages it has got the process down to “an hour or two”.
“All the system software is configured in Puppet and all [hosters] have to do is put a CD in the machine and the machine will come up and know what type of machine it is and install a Puppet client and get the config from the Puppet server,” Hetherington said.
“You could do this with propriety software but it would cost you an arm and a leg and it would be hard to convince pure businesspeople it is worth doing. You can do it much more cheaply with open source software. The investment [for Mooter] was one man year as opposed to five man years.”
Interestingly, all development and monitoring of Mooter’s platform is done in Sydney but hosting is spread across four data centres in China, Korea and the US.
At each site Mooter runs two Xen hosts with several virtual machines running a number of applications like a site central, DNS, Puppet server, and load balancer.
“If a Xen machine goes down we can rebuild another in 20 minutes,” Hetherington said. “When you are looking to provide redundancy don’t over-engineer it. With load balancing you can’t be down for 30 mins so we used Ultra Monkey.”
At the other end of each cluster is a data warehouse which extracts data from each machine and “this architecture easily satisfies our business requirements”.
While the company would rather serve all its subscribers out of one site, “the great firewall of China” makes ad serving out of China too slow so it had to do its own hosting there.
For monitoring, Mooter uses Nagios to perform 600 checks every five minutes, and has extended it to monitor business metrics like how many failures there are from suppliers for ads.
Other open source tools used for the business include language processing and Pentaho for reporting and BI.
While Mooter has achieved success and innovation on top of open source software, Hetherington is adamant he’s not religious about software and “just wants it to work”.
“We still need to run a business and it is more efficient to run a business in a Windows environment than Linux, even if it’s just saving on training, that’s just a basic reality unfortunately,” he said. “If you are running a business you will want to support Windows and get a supplier that can support both because if you have Windows and Linux they will always blame each other.”
Solutions First senior developer Matt Palmer said the Mooter story is a good example of the “pure flexibility” of using open source software.
“We’ve been able to glue together various disparate pieces of software in ways nobody envisaged you would be able to do,” Palmer said. “The nice thing about open source is if it is not already there you can add it later. With proprietary software you are forced into begging to the vendor or switching horses mid-stream which is disruptive to the business.”
Palmer said a number of improvements to Puppet were made and contributed back to the open source code base with Mooter’s position.