Word Count Map Reduce program

 Aim: Run a basic Word Count Map Reduce program to understand Map Reduce Paradigm

 

Program:

Source Code

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;// provides access to configuration parameters

import org.apache.hadoop.fs.Path;// Path class names a file or directory in a HDFS

import org.apache.hadoop.io.IntWritable;// primtive Writable Wrapper class for integers.

import org.apache.hadoop.io.Text;// This class stores text and provides methods to serialize, deserialize, and compare texts at byte level

import org.apache.hadoop.mapreduce.Job;//Job class allows the user to configure the job, submit it, control its execution, and query the state

//The Hadoop Map-Reduce framework spawns one map task for each InputSplit generated by the InputFormat for the job

import org.apache.hadoop.mapreduce.Mapper;//Maps input key/value pairs to a set of intermediate key/value pairs.

import org.apache.hadoop.mapreduce.Reducer;//Reduces a set of intermediate values which share a key to a smaller set of values.

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;//A base class for file-based InputFormats.

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; //A base class for file-based OutputFormats.

 

public class WordCount {

 

  public static class TokenizerMapper

       extends Mapper<Object, Text, Text, IntWritable>{

 

    private final static IntWritable one = new IntWritable(1);

    private Text word = new Text();

 

    public void map(Object key, Text value, Context context

                    ) throws IOException, InterruptedException {

      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {

        word.set(itr.nextToken());

        context.write(word, one);

      }

    }

  }

 

  public static class IntSumReducer

       extends Reducer<Text,IntWritable,Text,IntWritable> {

    private IntWritable result = new IntWritable();

 

    public void reduce(Text key, Iterable<IntWritable> values,

                       Context context

                       ) throws IOException, InterruptedException {

      int sum = 0;

      for (IntWritable val : values) {

        sum += val.get();

      }

      result.set(sum);

      context.write(key, result);

    }

  }

 public static void main(String[] args) throws Exception {

    Configuration conf = new Configuration();

    Job job = Job.getInstance(conf, "word count");

    job.setJarByClass(WordCount.class);

    job.setMapperClass(TokenizerMapper.class);

    job.setCombinerClass(IntSumReducer.class);

    job.setReducerClass(IntSumReducer.class);

    job.setOutputKeyClass(Text.class);

    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));

    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);

  }

}

Usage

$export CLASSPATH="$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:$HADOOP_HOME/share/hadoop/coomon/hadoop-common-2.8.1.jar"

Compile WordCount.java and create a jar:

$ javac WordCount*.java

$ jar -cvf wc.jar WordCount*.class

Assuming that:

• input - input directory in HDFS
• output - output directory in HDFS

Sample text-files as input:

$ hadoop fs -put file1.txt input

 

$ bin/hadoop fs -cat input/file1.txt

Hai welcome Hai cse

Run the application:

$ hadoop jar wc.jar WordCount input/file1.txt output

Output:

$ bin/hadoop fs -cat output/part-r-00000`

cse 1

Hai 2

welcome 1