Erlang, Thrift and HBase

Since documentation is lacking for Thrift and HBase (and Erlang), though the examples in Thrift’s code are pretty good, I figured as I got things going I’d post snippets. I assume you have Erlang installed, but I will suggest checking out Erlware for setting up Erlang and installing releases and applications.

First, download Thrift and HBase.

The Thrift instructions are below. However, note you need to install C++ boost libs and the Ruby development package (maybe more, but thats all I needed on my machine at least, hehe), I give only the Debian/Ubuntu way:

$ sudo apt-get install libboost-dev
$ sudo apt-get install ruby1.8-dev
$ cd ~/Desktop
$ svn co http://svn.apache.org/repos/asf/incubator/thrift/trunk thrift
$ cd thrift
$ ./bootstrap.sh
$ ./configure
$ make
$ make install

For HBase:

$ sudo apt-get install sun-java6-jdk
$ update-java-alternatives -s java-6-sun
$ sudo apt-get install ant

Download HBase, uncompress and edit hbase-env.sh to point to your Java and compile:

$ cd ~/Desktop
$ wget http://apache.imghat.com/hadoop/hbase/hbase-0.19.3/hbase-0.19.3.tar.gz
$ tar -zxvf hbase-0.19.3.tar.gz
$ cd hbase-0.19.3/
$ nano conf/hbase-env.sh
...
# The java implementation to use.  Java 1.6 required.
# export JAVA_HOME=/usr/java/jdk1.6.0/
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre/
...
$ ant
Copy Thrift's Erlang libs to your Erlang lib dir:
$ sudo cp -R ~/Desktop/thrift/lib/erl /usr/lib/erlang/lib/thrift-0.1.0/
I just picked that name at "random". I wish they had picked a standard and useful name with a version number and not named erl... But whatever.
Now you are able to use Thrifts Erlang modules. Next, we must get the HBase Thrift bindings created.
$ cd ~/Desktop/hbase-0.19.3/src/java/org/apache/hadoop/hbase/thrift/
$ thrift -gen erl Hbase.thrift
This creates gen-erl:
$ ls  gen-erl/
hbase_constants.hrl  hbase_thrift.erl  hbase_thrift.hrl  hbase_types.erl  hbase_types.hrl
Open up two new separate terminals, tabs, whatever, go back to where you compiled HBase and run:
$ cd ~/Desktop/hbase-0.19.3
$ ./bin/start-hbase.sh

$ cd ~/Desktop/hbase-0.19.3
$ ./bin/hbase thrift start
09/08/31 11:11:00 INFO ThriftServer: starting HBase Thrift server on port 9090
Finally, under the gen-erl directory do the following:
$ cd gen-erl
$ erlc *.erl
$ erl 
Erlang (BEAM) emulator version 5.6.5  [smp:2] [async-threads:0] [kernel-poll:false]
Eshell V5.6.5  (abort with ^G)
1> rr(hbase_types).
[alreadyExists,batchMutation,columnDescriptor,iOError,
 illegalArgument,mutation,notFound,tCell,tRegionInfo,
 tRowResult]
2> {ok, Client} = thrift_client:start_link("127.0.0.1", 9090, hbase_thrift).
{ok,<0.33.0>}
3> thrift_client:call(Client, getTableNames, []).
{ok,[]}
4> thrift_client:call(Client, createTable, ["test", [#columnDescriptor{name="test_col:"}]]).
{ok,ok}
5> thrift_client:call(Client, mutateRow, ["test", "test_row",
    [#mutation{isDelete=false,column="test_col:new", value="wooo"}]]).
{ok,ok}
6> thrift_client:call(Client, getTableNames, []).
{ok,[<<"test">>]}
7> thrift_client:call(Client, getRow, ["test", "test_row"]).
{ok,#tRowResult{row = <<"test_row">>,
 columns = {dict,1,16,16,8,80,48,
 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
 {{[],[],[],[],[],[],[],[],[],
 [[<<"test_col"...>>|#tCell{...}]],
 [],[],[],[],...}}}}}
Here we are compiling the Erlang module generated by Thrift, loading up an Erlang shell with the current working directory added to the path to look for beam files. After we are in the shell we add the records contained in hbase_types.hrl to our running environment. Next, we connect to the running Thrift server and tell it its called hbase_thrift on port 9090 on our localhost. To understand the function calls I make after that dig into the Erlang header files and source files HBase generates. You'll also need some understanding of HBase, which I don't have time to go into here. But basically we create a table with one column family called test_col. Then, we insert a value into the column new of the column family test_col. Lastly, we get the row test_row we just added from the table.
You can go into the HBase shell and see all this now:
$ ./bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.19.3, r, Mon Aug 31 11:55:38 CDT 2009
hbase(main):001:0> list
test
1 row(s) in 0.2660 seconds
hbase(main):002:0>scan 'test'
ROW                          COLUMN+CELL
test_row                    column=test_col:new, timestamp=1251739784969, value=wooo
1 row(s) in 0.0503 seconds


About these ads