Lookery.com

Building Voldemort read-only stores with Hadoop:

At Lookery, we have been working very hard to transform most of our data processing tasks into batch-oriented workflows in order to deal with growth. For example, we were already using Hadoop to compute our index and data files for our largest database, but the process of serving that information took place over too many network hops (load balancers, reverse proxies and Amazon S3). Therefore, as soon as I learned that Project Voldemort supported offline building of distributed stores, I decided to try it and we’re now running it in production. Just visit the guest blog post over that Project Voldemort blog for full details.

Data on every user on every web page. That’s the promise we are trying to deliver on at Lookery. The natural audience for Lookery’s services are Ad Networks, mostly those specializing in performance-based advertising. We’re seeing the next wave of interest coming from Search Marketers, Survey & Market Research Companies and Landing Page Optimizers. The [...]

Interesting conversation at lunch yesterday.  Elias and I had a different take on what it means for a dev team to be “efficient” (I’ll have to make a special post when the two of us find ourselves in total agreement on something ;-).   I’m going to caricature what we were saying a bit, because I think it captures two things developers strive for:

Efficiency Definition 1: developers spend as little time as possible not coding (meaning, as few meetings as possible, as little interaction with ticket systems as possible, etc).

Efficiency Definition 2: developers spend as little time as possible coding features which no one needs

Those two aren’t exactly in direct conflict, but they do imply different tradeoffs.  If you go for 2, you’re willing to hit a few meetings, jump through some ticket system hoops, because you live in abject fear of writing the wrong code.  That’s where I find myself, at this point in my career.  That’s part of why I don’t trust dev teams that romanticize overwork — it’s almost impossible to avoid writing the wrong code when you’re pulling 60 hour weeks, and working in constant low-grade fear.

-Dan M

Nifty post from Adam Lindberg on the Erlang-Questions mailing list, about how he structures his eunit tests:

In myapp/src/foo.erl I put a small header file inclusion:

-ifdef(TEST).
-include("foo_test.hrl").
-endif.

And myapp/test/foo_tests.hrl looks like this:

-include_lib("eunit/include/eunit.hrl").

foo_test_() ->
   [?_assertEqual(ok, public_function())].

bar_test_() ->
   [?_assertEqual(ok, private_function())].

With the right compile options, you get tests only built into the code during dev, and you get to keep your tests in a separate file (but still have full access to the internal functions, which I find absolutely necessary).

-Dan M