Food Fight

The Podcast where DevOps chefs do battle

Episode #4, Big Love for Big Data, Managing Hadoop Infrastructure With MrFlip

Guest: Flip Kromer twitter/github/irc: mrflip, http://infochimps.com,
download here


Hosts
  • Bryan Berry bryan.berry@gmail.com, Twitter/IRC/GitHub: bryanwb, blog: http://devopsanywhere.blogspot.com
  • Matt Ray matt@opscode.com Twitter/IRC/GitHub: mattray blog: http://leastresistance.net
  • Lusis, a.k.a. John E. Vincent blog: http://blog.lusis.org twitter: github: lusis
Thanks to Eric Reeves for our awesome new intro music! Eric is in charge of Environment Management at Alert Logic, and a part time producer under the moniker “Litex”. 

you can check out more of his tunes at soundcloud.com/litex.
    News & Events
    Interview: Flip Kromer, from InfoChimps

    The Hadoop Haiku courtesy of MrFlip
    Data flutters by
    Elephants make sturdy piles
    Number becomes thought
    • His infrastructure
      • 20 TB total storage
      • ~60 nodes
      • uses hosted chef
      • takes 50% of 1 engineer to maintain
      • Uses AWS for everything
      • His stack
        • Hadoop - the kernel of Big Data OS, see Hadoop Haiku
        • At least 10 different databases including hbase, cassandra, elastic search, mysql, redis
        • flume and looking to use 0mq somewhere in there
        • goliath is their main application server for Ruby apps
      • Ironfan wires clusters of nodes together
        • facets
        • aspects
        • components
        • git repository is the single source of truth
      • Ironfan-Pantry A set of Big Data cookbooks, a goldmine of AWESOMENESS that every chef users can learn from
      • Needed built-in resources
      • Uses cuken to test their cookbooks together with ironfan-ci

      Picks

      Matt

      Bryan


      Lusis

      • openvpn cookbook + cheap cloud account - uses in hostile environment to connect to his machines
      • moleskine notebooks - great if you still remember how to write by hand

      Flip

      • cuken - hooks into aruba, lets you do integration testing for your machine
      • Guard - watches for changes in your filesystem and takes actions accordingly