Archive for the ‘Uncategorized’ Category

Erlang, Thrift and HBase

Since documentation is lacking for Thrift and HBase (and Erlang), though the examples in Thrift’s code are pretty good, I figured as I got things going I’d post snippets. I assume you have Erlang installed, but I will suggest checking out Erlware for setting up Erlang and installing releases and applications.

First, download Thrift and HBase.

The Thrift instructions are below. However, note you need to install C++ boost libs and the Ruby development package (maybe more, but thats all I needed on my machine at least, hehe), I give only the Debian/Ubuntu way:

$ sudo apt-get install libboost-dev
$ sudo apt-get install ruby1.8-dev
$ cd ~/Desktop
$ svn co http://svn.apache.org/repos/asf/incubator/thrift/trunk thrift
$ cd thrift
$ ./bootstrap.sh
$ ./configure
$ make
$ make install

For HBase:

$ sudo apt-get install sun-java6-jdk
$ update-java-alternatives -s java-6-sun
$ sudo apt-get install ant

Download HBase, uncompress and edit hbase-env.sh to point to your Java and compile:

$ cd ~/Desktop
$ wget http://apache.imghat.com/hadoop/hbase/hbase-0.19.3/hbase-0.19.3.tar.gz
$ tar -zxvf hbase-0.19.3.tar.gz
$ cd hbase-0.19.3/
$ nano conf/hbase-env.sh
# The java implementation to use.  Java 1.6 required.
# export JAVA_HOME=/usr/java/jdk1.6.0/
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre/
$ ant

Copy Thrift's Erlang libs to your Erlang lib dir:

$ sudo cp -R ~/Desktop/thrift/lib/erl /usr/lib/erlang/lib/thrift-0.1.0/

I just picked that name at "random". I wish they had picked a standard and useful name with a version number and not named erl... But whatever.

Now you are able to use Thrifts Erlang modules. Next, we must get the HBase Thrift bindings created.

$ cd ~/Desktop/hbase-0.19.3/src/java/org/apache/hadoop/hbase/thrift/
$ thrift -gen erl Hbase.thrift

This creates gen-erl:

$ ls  gen-erl/
hbase_constants.hrl  hbase_thrift.erl  hbase_thrift.hrl  hbase_types.erl  hbase_types.hrl

Open up two new separate terminals, tabs, whatever, go back to where you compiled HBase and run:

$ cd ~/Desktop/hbase-0.19.3
$ ./bin/start-hbase.sh

$ cd ~/Desktop/hbase-0.19.3
$ ./bin/hbase thrift start
09/08/31 11:11:00 INFO ThriftServer: starting HBase Thrift server on port 9090

Finally, under the gen-erl directory do the following:

$ cd gen-erl
$ erlc *.erl
$ erl 
Erlang (BEAM) emulator version 5.6.5  [smp:2] [async-threads:0] [kernel-poll:false]
Eshell V5.6.5  (abort with ^G)
1> rr(hbase_types).
2> {ok, Client} = thrift_client:start_link("", 9090, hbase_thrift).
3> thrift_client:call(Client, getTableNames, []).
4> thrift_client:call(Client, createTable, ["test", [#columnDescriptor{name="test_col:"}]]).
5> thrift_client:call(Client, mutateRow, ["test", "test_row",
    [#mutation{isDelete=false,column="test_col:new", value="wooo"}]]).
6> thrift_client:call(Client, getTableNames, []).
7> thrift_client:call(Client, getRow, ["test", "test_row"]).
{ok,#tRowResult{row = <<"test_row">>,
 columns = {dict,1,16,16,8,80,48,

Here we are compiling the Erlang module generated by Thrift, loading up an Erlang shell with the current working directory added to the path to look for beam files. After we are in the shell we add the records contained in hbase_types.hrl to our running environment. Next, we connect to the running Thrift server and tell it its called hbase_thrift on port 9090 on our localhost. To understand the function calls I make after that dig into the Erlang header files and source files HBase generates. You'll also need some understanding of HBase, which I don't have time to go into here. But basically we create a table with one column family called test_col. Then, we insert a value into the column new of the column family test_col. Lastly, we get the row test_row we just added from the table.

You can go into the HBase shell and see all this now:

$ ./bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.19.3, r, Mon Aug 31 11:55:38 CDT 2009
hbase(main):001:0> list
1 row(s) in 0.2660 seconds
hbase(main):002:0>scan 'test'
ROW                          COLUMN+CELL
test_row                    column=test_col:new, timestamp=1251739784969, value=wooo
1 row(s) in 0.0503 seconds

Read Full Post »

Checkout my post on the Erlware blog about using a simple one for one to handle DB requests from webmachine for a better tutorial.


I found the posts online about Erlang’s simple_one_for_one supervisors pretty lacking so I thought I’d throw something together about how I use them.

Using Nitrogen I have a chat site that communicates with an ejabberd backend. Each user gets a user_server process (a gen_server) that handles all the communication with ejabberd through the exmpp library from Process One. These are dynamic servers so they are unnamed and use simple_one_for_one supervisors for their supervision, which sit under the main applications supervisor.

The user_simple_one_for_one.erl looks basically like this:

start_link() ->
  supervisor:start_link(?MODULE, []).

init([]) ->
  UserSpec = {user_server, {user_server, start_link, []},
                            temporary, 2000, worker, [user_server]},
  StartSpecs = {{simple_one_for_one, 0, 1}, [UserSpec]},
  {ok, StartSpecs}.

Unlike a normal supervisor no child process is started after calling user_simple_one_for_one:start_link(). 0 and 1 define the number of restarts in 1 second and we use temporary for the worker since users come and go. Thus the server should shutdown on normal exit and not restart.  It order to start this child we call user_server:start(Pid). So in user_server.erl we need both start_link/2 and start/3.

start_link(UserName, Password) ->
  gen_server:start_link(?MODULE, [UserName, Password], []).

start(Super, UserName, Password) ->
  supervisor:start_child (Super, [UserName, Password]).

We then need to start the supervisor somewhere and store its Pid to be passed into a function that passes the supervisors Pid to the user_server for it to use to call start_child:

{ok, Pid} = user_simple_one_for_one:start_link().
jabber_login (Sup, UserName, Password) ->
  user_server:start(Sup, UserName, Password).

Now we have dynamic supervision of a dynamic gen_server.

Read Full Post »

The documentation and examples for Process One’s just released EXMPP library is very lacking. Not to fault them, it was just released a couple days ago. So here I’ll describe how I used EXMPP for working with a multi-user chat system. Obviously you need to have your Jabber server setup for muc. Uncomment the mod_muc stuff in your ejabberd config if you are using that. I have mine set to use conference.localhost.

First, you can look in echo_client.erl that is provided with the source to see how to setup a connection to your Jabber server. The key pieces that are a pain to figure out are the records for connecting to a room and for creating a message.

Below is a function for creating the record to send that will move your user to the specified room:

-define (MUC, <<"http://jabber.org/protocol/muc">>).
-define (JABBER_HOST, "localhost").
-define (JABBER_MUC_HOST, "conference.localhost").
-define (JABBER_PORT, 5222).

create_move_room (Room, Username) ->
  #xmlel {name=presence, attrs=[#xmlattr{name=to,
          children=[#xmlel{name=x, attrs=[#xmlattr{name=xmlns, value=?MUC}]}]}.

exmpp_session:send_packet(Session, create_move_room (Room, Username)),

Now that we are in a room we can start sending messages to the people there:

create_message (Room, Message) ->
  #xmlel {name=message, attrs=[#xmlattr{name=to,
          #xmlattr{name=type, value=list_to_binary("groupchat")}],

exmpp_session:send_packet(Session, create_message(To, Msg)),

Not much too it. I was just annoyed trying to figure out their records for XML but they turned out be very easy as well. Its great to have a complete and well done Erlang Jabber library at last.

Read Full Post »

I’m going to be writing some articles on Free software and media to hopefully get published. I was wondering if people in the Chicago GNU/Linux Users Group, or other Free software lovers who find this, could give me their ideas regarding Free software and socialism (Marxist not Euro-style). Or opposted views, comparing Free software and media to liberatiarianism or individual anarchism.

Read Full Post »

Democrats: When Republicans point out that the economy IS still on a downward spiral and that the Obama administration IS attacking our freedoms, it is not a viable argument to point out Bush did the same and Republicans weren’t attacking it then. You are up in arms while the economy collapses, wars are waged and freedoms are suspended during a Republican administration but where are you now?

Read Full Post »

Dear U.S. Navy

For every smile of a child in your new commercial, how many will never smile again because of your imperialist aggression?

Read Full Post »

Having read “Statism and Anarchy” by Bakunin and works by Chomsky I realized what I think is a correlation between anarchists’ reasoning for the destruction of the State and its difference from the socialists’ reasoning. We both agree wage slavery is “intolerable” and that true freedom will only come with the removal of the State, and that this is a feasible goal. But my feeling is the anarchist, or libertarian-socialist, focuses on the illegitimacy of almost all authority while the socialists focus more on the contradictions found in the capitalist system and evils of wage slavery.

Obviously I understand both sides talk about both, but I get the feeling each puts emphasis on one over the other.

I’m not going to say one is better or worse than the either, I think both need to grow together to focus on these issues equally. This is simply an observation and I wonder if others agree.

States are violent institutions. The government of any country, including ours, represents some sort of domestic power structure, and it’s usually violent. States are violent to the extent that they’re powerful, that’s roughly accurate. – Noam Chomsky

Read Full Post »

Similar to my last post, regarding the capitalists attempts to save their system by giving small victories to the working class, this one is about the health insurance industry doing the same thing. Major insurance companies are telling the Obama administration they are ready to play ball this time. This isn’t because they want to help provide all citizens with adequate health care for minimum cost, but because they know through Obama’s plan they can gain more control and rake in larger profit for their private companies.

Of course, if private insurers can provide everyone with the health care they need there would be no reason to complain. Just because its private is not necessarily a reason to attack something, but if its private and horribly inefficient and down right evil at times, it is.

In Paul Krugman’s op-ed today he is much less cynical about this and seems to genuinely feel it is very good news. However, I think all of his fears will come true. We can not allow the option of buying into a public insurance plan be removed, and we can not allow this to be a way for the private insurers to dig in their claws to prevent the eventual move to fully public heath insurance.

These are the same people who have been preventing reform and fighting to limit Medicare for decades. It is obvious they can not be trusted. They want Obama’s plan because it will mean more people having private insurance and they hope by being at the table they can mold it to be even more beneficial to them and less to us.

Single payer is the only way to go. We all know the Democrats are employees of the large insurance companies and not us, so I have little faith we will see true health care reform. But we must do our best to make our voices heard and pressure the Democrats to work for us not profit.

Read Full Post »

The New Economy

Its been pointed out by many “talking heads” that when we come out of this recession we will have a “new economy”. This point is made to say that we won’t know when we are out of it exactly, since things will look different. And some make the point that we can’t go back to the economy of speculation and debt. So, what will this new economy look like? Can it exist?

With stagnant wages for workers the economy has become based on credit card debt and mortgages. I’ve heard lots of talk about what the banks have been doing to ruin the economy, but it is rare to hear someone put out any sort of idea on how to fix a fundamental problem such as providing the working class with living wages, health care and housing. The banks are showing they are very much opposed to changing how they do business and no one has a plan to help the working class, so how can anything change?

The current system does not work and will not be changed willingly by those in power. However, they will do their best to modify it in order to keep the masses from revolting. The depression gave us the New Deal, but only to ensure the continuation of the capitalist system and the increasing profits for corporations. But will Obama be able to pull of something similar? With the Republicans so out of it I see this being a very tough thing to accomplish.

The only true solution is the removal of this system and its replacement with a society run democratically by the workers, a socialist state. I will be very interested in seeing what the Democrats are able to come up with to patch the global capitalist system, as it stands I see a good chance for real change — not Obama “change”.

Read Full Post »

Torture Investigation

Following an investigation and trial of those involved in sanctioning torture to be used we’ll be rid of a large portion of the top Republicans and Democrats. Happy days, to see Cheney and Pelosi behind bars.

I may be wrong, but I believe all criminal trials are about something that happened in the past. So why does this, “We don’t want to look back”, nonsense have any acceptance!?

Read Full Post »

Older Posts »


Get every new post delivered to your Inbox.