Hadoop: Find Inner Product of Two Sparse Vectors

Its been a while I wrote a hadoop program and thought to come back in action. I wrote the program for the following problem

Inner product of two sparse vectors—A vector is a list of values. Given two vectors, X=[x1,x2,…]andY=[y1,y2,…],theirinnerproductisZ=x1*y1+x2*y2+… . When X and Y are long but have many elements with zero value, they’re usually given in a sparse representation:1,0.469,0.2117,0.92…

Hadoop in Action Chapter 4 Problem 3

Given a problem statement this is pretty straightforward thing to do.

The program is simple to write

Lets run this ..

Think of these as (x1, x2) pairs

This runs pretty quickly

and the output

Lets confirm this if this is right output

So we have implemented this correctly. Now you can run this on any number of machines as long as input format is same as mentioned above.

Packaging Module : Build Jar With Dependencies Included

I have been working with a project lately which will migrate all the documents from datacenter to Amazon Web Services S3.

As part of this project, I had to build a small upload app, which will put these documents in Amazon.

The requirement I had was to build my app as jar and include all dependencies in the jar.

I googled around and realized that the following code in your `pom.xml` will do the trick

Find the Hourly Traffic From Server Logs Using Hadoop

This was relatively eaiser than I thought, or I understood it completely wrong ;)

The question goes like

Web traffic measurement—Take a web server log file and write a Streaming program with the Aggregate package to find the hourly traffic to that site.

Hadoop in Action Chapter 4 Problem 2

This program makes use of hadoop-streaming and uses Aggregate package to implement this program

The idea is simple

``````- get the timestamp from logs
- make key such that each everything between HOUR:00 to HOUR:59 is HOUR:00
- call the aggregate package to sum up the values for you
``````

Then write a mapper as

and you need to run this using streaming.jar

get the output

and your data after aggregation will look like

This is my first attempt to write a Map Reduce program myself :)

The input data - apat63_99.zip

The idea I had is following:

``````- read each record
- get the relevant column
- group the elements by common key, so that one reducer will get all the data
- get the unique elements in the list received by reducer and sort it
- return only the number of values requested as K parameter on command line
``````

The entire source code is

For running the program, do the following

Please feel free to point out the better ways of doing it, would be happy to learn

Passing Parameters to Mappers and Reducers in New Hadoop API

This came out as my part of learning where I had to pass variable on runtime to reducer to show up only the number of records I pass as parameter

I looked up and found that there are two ways - using old API and using new API.

Assuming that you are using the latest hadoop distribution, you are using new API, here is how you would do it

and this is all you need!

Basic Template for Your Most Map Reduce Programs

This is true that you would never have to write a Map Reduce program from scratch. This is what I learned while reading Hadoop in Action

I thought I would be a nice thing to write a basic Map Reduce skeleton that I and almost anyone want to write Map Reduce program can leverage.

Here is it

Let me know if you see any issues with that or you would like to share anything that would be useful.

Extracting Specific Fields From Your Mongo Collection

This morning I had a requirement where in I had to get the list of all names in my collections in a file.

I looked over the MongoDB documentation and found mongoexport utility that makes your life easier

To use it, it was pretty simple. Consider my schema as

Suppose you need to take out all the ‘name’ and ‘value’ from collection in a file. You will do the following

Thats it!!

This is my first post in attempt to learn Hadoop using Java. I am using “Hadoop in Action” book to learn. The first program is about building a Citation Histogram (mentioned completely in the book) It has two parts:

Part - 1 : Building the map reduce program to count the number of citations “cited”

Part - 2 : Building the map reduce program to count the counts and plot them

I have used R for plotting the graph.

For part - 1, you need to uncomment the part

because input dataset has “,” as input separator.

The output of part - 1 becomes the input for part - 2 and the input separator is tab(\t). Our InputFormat class is

and the default input separator is tab(\t) for this class.

So we see that the same program is reused twice.

Now the way you run this is as following:

`````` - Build jar for the class CitationHistogram (you can do this by using your IDE, I am using IntelliJ IDEA)

- copy the input to hadoop file system
``````

Once your program completes, you can run the following to get the data out from HDFS

and the you can read this in R(language) to plot the graph

and you can see the CitationHistogram following Power Law as following

First Post

This is my first post