Last week saw the return of DevOpsDays to New York and like many who attended, I went into day one without a solid definition for, or real understanding of, what “DevOps” actually means. Does it mean wearing both dev and ops hats? Is it a new team outside of the existing dev and ops teams? Is it a tool?
Sounds easy, doesn’t it? Well it actually is quite simple but the error messages along the way can really trip you up! My first attempt at profiling a bit of code was to use the full-boat Eclipse stack: Eclipse Test & Performance Tools Platform Project! Well, what they don’t tell you anywhere on the project page is that it’s only supported on Windows and Linux. A Mac port was started sometime around 2004 and never completed. Yeah, it’s been that long! ...
After spending the better part of a day trying to get HDFS to mount on my Mac, I finally gave up. Luckily, I was able to find MuCommander, a cross-platform Java port of the Norton Commander of old, and as luck would have it, it supports HDFS in the latest version! Very handy for quickly browsing your HDFS clusters if you can’t mount it or don’t have the Hadoop toolset installed. ...
Hadoop has a built-in feature for easily distributing JARs to your worker nodes via HDFS but, unfortunately, it’s broken. There’s a couple of tickets open with a patch again 0.18 and 0.21 (trunk) but for some reason they still haven’t been committed yet. We’re currently running 0.20 so the patch does me no good anyway. So here’s my simple solution: I essentially copied the technique used by ToolRunner when you pass a “libjars” argument on the command line. You simply pass the function the HDFS paths to the JAR files you want included and it’ll take care of the rest. ...
Using Hadoop’s DistributedCache mechanism is fairly straightforward, but as I’m finding is common with everything-Hadoop, not very well documented. Adding files When setting up your Job configuration: // Create symlinks in the job's working directory using the link name // provided below DistributedCache.createSymlink(conf); // Add a file to the cache. It must already exist on HDFS. The text // after the hash is the link name. DistributedCache.addCacheFile( new URI("hdfs://localhost:9000/foo/bar/baz.txt#baz.txt"), conf); Accessing files Now that we’ve cached our file, let’s access it: ...
For some reason, I had a relatively hard time finding this info all in one spot, so here it is: Using MySQL with JRuby is actually pretty easy (and no annoying arch issues on OS X! :-) Install GEMs (DBI, JDBC driver, DBI adapter): $ jgem install dbi jdbc-mysql dbd-jdbc Then use it! require 'dbi' require 'jdbc/mysql' dbh = DBI.connect('dbi:jdbc:mysql://localhost:3306/test', 'root', '', { "driver" => "com.mysql.jdbc.Driver" } )
I’ve spent the last few days since joining Better Advertising working on a new feature for a Firefox extension called Ghostery. We’ll be announcing the new feature soon, but until then I thought I’d share some of what I’ve learned so far. I’ve never worked on an extension before but as it turns out, it’s really quite easy to pick up; some fairly simple XML (aka XUL) for composing the UI and JavaScript for the rest. One of the trickier bits has to do with scope. After doing some testing I figured out that the entry point into an extension is via the browser window; that is, your extension code will be executed each time you open a new window and that means that all your code is basically scoped to a single window. ...
Del.icio.us (sorry, it’s just plain old “Delicious” now) 2.0 finally launched a few days ago and the response so far has been mixed. But now that the dust has settled some, it’s time to think about just how we got here and if it was really worth all the trouble. According to the official blog post, the new and improved Delicious brings us speed, usability, and oh so good looks among other features and it was a long time in the making. The Yahoo acquisition was announced on Dec 9, 2005 and the new site finally went live a little over two and a half years later on July 31, 2008. So why did it take them so long? ...
On Thursday, Amazon announced SimpleDB, “a web service for running queries on structured data in real time.” As many others have noted this more or less completes the cloud computing stack that Amazon has been steadily building, ever since they launched the Simple Storage Service (S3), early last year. Where their earlier releases (S3, Elastic Compute Cloud [EC2], Flexible Payments, Mechanical Turk) commoditized much of the infrastructure required for building scalable applications, SimpleDB (SDB) and the earlier Simple Queue Service (SQS) are bringing cutting edge technologies and design patterns to the masses. First they made it cheap and easy to have a cluster; now they’ve made it cheap and easy to use a cluster! Amazing. ...
A large part of my job here at ICAR has been wrestling with various so-called content management systems (CMS). In an effort to build various applications I’ve been evaluating many popular opensource CMS projects and I’ve run into the same basic problem with just about all of them: I don’t want a blog, I want content management. They all claim to be flexible systems with all the latest doodads but in the end, they’re just glorified blogs. Case in point, almost every system sets itself up as a blog out of the box, and, in general, that’s the most complete part of the system. Other areas are sorely lacking.