Powerset - Launching with Ruby!
Well, we are beginning to open up the Powerset Kimono! So here it goes. We are going to use Ruby as our front end of our search engine product(s)
We announced this first to the Powerlabbers this morning and then we posted today on Kevin Clark's blog (http://glu.ttono.us). Kevin is one of Powerset's Ruby Rockstars.
The picture to the left shows just some of our Ruby folks including (from left to right) Dave, Siqi, Tom, Chris, Toby, Kevin and Max. Hint we have more too.
In Kevin's blog he goes through our thinking on the topic and its part of the extented entry of this blog if you would like to read it.
Why Ruby?
1. We've already got the brains
One thing we haven't kept secret is that we've hired some of the best Ruby developers around. Our total number of day in day out Ruby developers is somewhere around 10, and I'm constantly humbled to be working with this team. We've got the people and they have the skills, so it makes sense to apply them.
2. Ruby is already being used throughout the company
We've always spoken in general terms about how much Ruby is being used internally, but let's get specific: a substantial part of our infrastructure is being written in Ruby or being accessed through Ruby services. Our scientists use Ruby to interact with our core language technology. Our packaging infrastructure is Ruby. A big portion of our system administration work is all done with Ruby. Frankly, we as an organization use Ruby a whole heck of a lot.
Additionally, all of our product demos and prototypes are also in Ruby. We've got an interesting mix of Rails, Merb and Camping apps (depending on the scope of the project) connecting to tiny Ruby services which hook into our various back-end systems. Day to day, the majority of the product team is hacking in Ruby in some capacity.
3. We're not worried about scaling
So, inevitably, whenever we talk about Ruby or Rails scaling these days someone brings up Twitter and its scaling problems in the past. Twitter is right down the block from our offices and I know several of the devs personally, so before we made a final decision I arranged a sit down with Twitter's lead developer, Blaine Cook, to talk about the situation. Blaine was kind enough to let me bring along our Search Architect (and former search architect at Yahoo!) Chad Walters , our Head of Product Scott Prevost, and our COO Steve Newcomb, to poke and prod and get their questions answered. The simple fact is that Ruby wasn't the source of Twitter's woes. As it often happens with rapidly growing sites, they ran into architectural problems. Some design decisions don't hurt until they reach a massive scale and at that point you have to rethink your approach. In an email he writes:
For us, it's really about scaling horizontally - to that end, Rails and Ruby haven't been stumbling blocks, compared to any other language or framework. The performance boosts associated with a "faster" language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January
This is great news for Twitter, but even better for us because we don't have the bottle necks that they've struggled with - databases, instant messaging servers, and regularly recycling cache systems - which makes scaling horizontally much much smoother. At that point, our scaling issue doesn't concern Ruby. For a search engine, the front-end is largely just a templating system and the real work happens in the back when we process your query.
What does this mean for the community?
When writing this article, at some point I had to sit down and ask myself why anyone should care we're adopting Ruby for the front-end. For me, it comes down to the fact that we're good for the community as a whole.
First off, the fact that Powerset is deploying on Ruby means you've got one more high traffic site (potentially very high traffic) using Ruby in production. It's one more case study, and one more example that Ruby as a whole is ready for the big show.
Personally, I think the more interesting and useful thing to take away from this is that as we do the heavy lifting, building up infrastructure around all the aspects of Ruby development and deployment in the company, we're selecting large chunks to be open-sourced. I've got a list of things I'd love to put out into the wild (which is encouraged, and actually suggested by my manager. Man, I love this place) as soon as I can find the time. Already Tom Werner and Dave Fayram have pushed out Ruby to Erlang bindings and a sweet little (in-development) web server called Fuzed, I've gotten to hack at Merb, and a fair about of Rails patches have come directly from work in-house. Hopefully the community will be able to benefit from our code as much as we have.
Obviously we don't have a search product open to the public yet, but we'll be launching Powerlabs in September. In Powerlabs, you'll be able to play with our products and give us feedback. If you want to keep track of what Powerset is doing, sign up.
1. We've already got the brains
One thing we haven't kept secret is that we've hired some of the best Ruby developers around. Our total number of day in day out Ruby developers is somewhere around 10, and I'm constantly humbled to be working with this team. We've got the people and they have the skills, so it makes sense to apply them.
2. Ruby is already being used throughout the company
We've always spoken in general terms about how much Ruby is being used internally, but let's get specific: a substantial part of our infrastructure is being written in Ruby or being accessed through Ruby services. Our scientists use Ruby to interact with our core language technology. Our packaging infrastructure is Ruby. A big portion of our system administration work is all done with Ruby. Frankly, we as an organization use Ruby a whole heck of a lot.
Additionally, all of our product demos and prototypes are also in Ruby. We've got an interesting mix of Rails, Merb and Camping apps (depending on the scope of the project) connecting to tiny Ruby services which hook into our various back-end systems. Day to day, the majority of the product team is hacking in Ruby in some capacity.
3. We're not worried about scaling
So, inevitably, whenever we talk about Ruby or Rails scaling these days someone brings up Twitter and its scaling problems in the past. Twitter is right down the block from our offices and I know several of the devs personally, so before we made a final decision I arranged a sit down with Twitter's lead developer, Blaine Cook, to talk about the situation. Blaine was kind enough to let me bring along our Search Architect (and former search architect at Yahoo!) Chad Walters , our Head of Product Scott Prevost, and our COO Steve Newcomb, to poke and prod and get their questions answered. The simple fact is that Ruby wasn't the source of Twitter's woes. As it often happens with rapidly growing sites, they ran into architectural problems. Some design decisions don't hurt until they reach a massive scale and at that point you have to rethink your approach. In an email he writes:
For us, it's really about scaling horizontally - to that end, Rails and Ruby haven't been stumbling blocks, compared to any other language or framework. The performance boosts associated with a "faster" language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January
This is great news for Twitter, but even better for us because we don't have the bottle necks that they've struggled with - databases, instant messaging servers, and regularly recycling cache systems - which makes scaling horizontally much much smoother. At that point, our scaling issue doesn't concern Ruby. For a search engine, the front-end is largely just a templating system and the real work happens in the back when we process your query.
What does this mean for the community?
When writing this article, at some point I had to sit down and ask myself why anyone should care we're adopting Ruby for the front-end. For me, it comes down to the fact that we're good for the community as a whole.
First off, the fact that Powerset is deploying on Ruby means you've got one more high traffic site (potentially very high traffic) using Ruby in production. It's one more case study, and one more example that Ruby as a whole is ready for the big show.
Personally, I think the more interesting and useful thing to take away from this is that as we do the heavy lifting, building up infrastructure around all the aspects of Ruby development and deployment in the company, we're selecting large chunks to be open-sourced. I've got a list of things I'd love to put out into the wild (which is encouraged, and actually suggested by my manager. Man, I love this place) as soon as I can find the time. Already Tom Werner and Dave Fayram have pushed out Ruby to Erlang bindings and a sweet little (in-development) web server called Fuzed, I've gotten to hack at Merb, and a fair about of Rails patches have come directly from work in-house. Hopefully the community will be able to benefit from our code as much as we have.
Obviously we don't have a search product open to the public yet, but we'll be launching Powerlabs in September. In Powerlabs, you'll be able to play with our products and give us feedback. If you want to keep track of what Powerset is doing, sign up.