Skip to main content

Word Count Map Reduce program

 Aim: Run a basic Word Count Map Reduce program to understand Map Reduce Paradigm

 

Program:

Source Code

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;// provides access to configuration parameters

import org.apache.hadoop.fs.Path;// Path class names a file or directory in a HDFS

import org.apache.hadoop.io.IntWritable;// primtive Writable Wrapper class for integers.

import org.apache.hadoop.io.Text;// This class stores text and provides methods to serialize, deserialize, and compare texts at byte level

import org.apache.hadoop.mapreduce.Job;//Job class allows the user to configure the job, submit it, control its execution, and query the state

//The Hadoop Map-Reduce framework spawns one map task for each InputSplit generated by the InputFormat for the job

import org.apache.hadoop.mapreduce.Mapper;//Maps input key/value pairs to a set of intermediate key/value pairs.

import org.apache.hadoop.mapreduce.Reducer;//Reduces a set of intermediate values which share a key to a smaller set of values.

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;//A base class for file-based InputFormats.

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; //A base class for file-based OutputFormats.

 

public class WordCount {

 

  public static class TokenizerMapper

       extends Mapper<Object, Text, Text, IntWritable>{

 

    private final static IntWritable one = new IntWritable(1);

    private Text word = new Text();

 

    public void map(Object key, Text value, Context context

                    ) throws IOException, InterruptedException {

      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {

        word.set(itr.nextToken());

        context.write(word, one);

      }

    }

  }

 

  public static class IntSumReducer

       extends Reducer<Text,IntWritable,Text,IntWritable> {

    private IntWritable result = new IntWritable();

 

    public void reduce(Text key, Iterable<IntWritable> values,

                       Context context

                       ) throws IOException, InterruptedException {

      int sum = 0;

      for (IntWritable val : values) {

        sum += val.get();

      }

      result.set(sum);

      context.write(key, result);

    }

  }

 public static void main(String[] args) throws Exception {

    Configuration conf = new Configuration();

    Job job = Job.getInstance(conf, "word count");

    job.setJarByClass(WordCount.class);

    job.setMapperClass(TokenizerMapper.class);

    job.setCombinerClass(IntSumReducer.class);

    job.setReducerClass(IntSumReducer.class);

    job.setOutputKeyClass(Text.class);

    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));

    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);

  }

}

Usage

$export CLASSPATH="$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:$HADOOP_HOME/share/hadoop/coomon/hadoop-common-2.8.1.jar"

Compile WordCount.java and create a jar:

$ javac WordCount*.java

$ jar -cvf wc.jar WordCount*.class

Assuming that:

  • input - input directory in HDFS
  • output - output directory in HDFS

Sample text-files as input:

$ hadoop fs -put file1.txt input

 

$ bin/hadoop fs -cat input/file1.txt

Hai welcome Hai cse

Run the application:

$ hadoop jar wc.jar WordCount input/file1.txt output

Output:

$ bin/hadoop fs -cat output/part-r-00000`

cse 1

Hai 2

welcome 1

Comments

Post a Comment

Popular posts from this blog

Array of Objects

An array can be of any data type including struct. Similarly, we can also have arrays of variables of the type class. Such variables are called arrays of objects. Class Definition: class employee {           char name[30];          float age;     public:          void getdata(void);           void putdata(void); }; The identifier employee is a user-defined data type and can be used to create objects related to different employee categories. employee manage[3];          //aray of managers employee foreman[15];       //array of foreman employee worker[75];        // array of worker the array manager contains three objects(managers), namely, manager[0],  manager[1], and manager[2], of type employee class similarly, the foreman array contains 15 objects. and the worker array contains 75 objectives.(work...
Post a Comment
Read more

Binning Method by Data smoothing in python

 Binning Method Binning is a technique for smoothing data or dealing with noisy data. The data is sorted first, and then the sorted values are dispersed into a number of buckets or bins in this approach. Binning methods provide local smoothing since they consult the vicinity of values.  Smoothing can be accomplished in three ways: Bin smoothing entails:  Each value in a bin is replaced by the bin's mean value when smoothing by bin means is used.  Smoothing by bin median:  Each bin value is replaced by its bin median value in this method.  Smoothing by bin borders:  In smoothing by bin boundaries, the bin boundaries are determined as the minimum and maximum values in a given bin. The nearest boundary value is then used to replace each bin value. Example: Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34 Smoothing by bin means:       - Bin 1: 9, 9, 9, 9       - Bin 2: 23, 23, 23, 23   ...
Post a Comment
Read more

Hadoop file Management Tasks

  Implement the following file management tasks in Hadoop: a) Adding files and directories b) Retrieving files c) Deleting files Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into HDFS using one of the above command line utilities. Program:  The most common file management tasks in Hadoop includes: Adding files and directories to HDFS Retrieving files from HDFS to local filesystem Deleting files from HDFS Hadoop file commands take the following form:     hadoop fs - cmd Where cmd is the specific file command and <args> is a variable number of arguments. The command cmd is usually named after the corresponding Unix equivalent. For example, the command for listing files is ls as in Unix. a) Adding Files and Directories to HDFS Creating Directory in HDFS    $ hadoop fs - mkdir foldername (syntax)  $ ha...
Post a Comment
Read more