Skip to main content

Hadoop installation in Windows

 

Installing Hadoop 3.2.1 on Windows

1.Prerequisites

First, we need to make sure that the following prerequisites are installed:

2. Download Hadoop Barriers

The first step is to download Hadoop binaries from the official website. The binary package size is about 342 MB.
or 
Better to download Hadoop Binaries from the nonofficial. you can directly utilize these file. 

After finishing the file download, we should unpack the package using 7zip int two steps. First, we should extract the hadoop-3.2.1.tar.gz library, and then, we should unpack the extracted tar file

3. Setting up environmental Variables

After installing Hadoop and its prerequisites, we should configure the environment variables to define Hadoop and Java default paths.
To edit environment variables, 
                go to Control Panel > 
                                                    System and Security > 
                                                                                         System (or right-click > properties on My Computer icon) and 
                                  click on the “Advanced system settings” link.

When the “Advanced system settings” dialog appears, 
go to the “Advanced” tab and click on the “Environment variables” button located on the bottom of the dialog.

In the “Environment Variables” dialog, press the “New” button to add a new variable.

There are two variables to define:
  1. JAVA_HOME: JDK installation folder path
  2. HADOOP_HOME: Hadoop installation folder path
Adding JAVA_HOME Variable

Adding JAVA_HOME PATH

Adding HADOOP_HOME PATH

Editing Path Variables

Adding New Path to the PATH Variables


3.1 JAVA Incorrectly set error

Now, let’s open PowerShell and try to run the following command:

hadoop -version

In this example, since the JAVA_HOME path contains spaces, I received the following error:

JAVA_HOME is incorrectly set

JAVA_HOME Error

To solve this issue, we should use the windows 8.3 path instead. As an example:

Use “Progra~1” instead of “Program Files”
Use “Progra~2” instead of “Program Files(x86)”

After replacing “Program Files” with “Progra~1”, we closed and reopened PowerShell and tried the same command. As shown in the screenshot below, it runs without errors.
Hadoop version command executed successfully


4. Configuring Hadoop Cluster

There are four files we should alter to configure Hadoop cluster:

%HADOOP_HOME%\etc\hadoop\hdfs-site.xml
%HADOOP_HOME%\etc\hadoop\core-site.xml
%HADOOP_HOME%\etc\hadoop\mapred-site.xml
%HADOOP_HOME%\etc\hadoop\yarn-site.xml

4.1. HDFS site configuration

As we know, Hadoop is built using a master-slave paradigm. Before altering the HDFS configuration file, we should create a directory to store all master node (name node) data and another one to store data (data node). In this example, we created the following directories:

E:\hadoop-env\hadoop-3.2.1\data\dfs\namenode
E:\hadoop-env\hadoop-3.2.1\data\dfs\datanode

Now, let’s open “hdfs-site.xml” file located in “%HADOOP_HOME%\etc\hadoop” directory, and we should add the following properties within the <configuration></configuration> element:

Note: Give You Drive Name Properly in the place of file path.
<property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.name.dir</name><value>file:///Your Drive Name( C/D/E/etc):/hadoop-env/hadoop-3.2.1/data/dfs/namenode</value></property><property><name>dfs.datanode.data.dir</name><value>file:///Your Drive Name( C/D/E/etc):/hadoop-env/hadoop-3.2.1/data/dfs/datanode</value></property>

Note that we have set the replication factor to 1 since we are creating a single node cluster.

4.2. Core site configuration

Now, we should configure the name node URL adding the following XML code into the <configuration></configuration> element within “core-site.xml”:
<property><name>fs.default.name</name><value>hdfs://localhost:9820</value></property>

4.3. Map Reduce site configuration

Now, we should add the following XML code into the <configuration></configuration> element within “mapred-site.xml”:
<property><name>mapreduce.framework.name</name><value>yarn</value><description>MapReduce framework name</description></property>

4.4. Yarn site configuration

Now, we should add the following XML code into the <configuration></configuration> element within “yarn-site.xml”:
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Yarn Node Manager Aux Service</description></property>


5. Formatting Name node

After finishing the configuration, let’s try to format the name node using the following command:
hdfs namenode -format


6. Start Hadoop Services

Now, we will open PowerShell, and navigate to “%HADOOP_HOME%\sbin” directory. Then we will run the following command to start the Hadoop nodes:

.\start-dfs.cmd









Comments

Post a Comment

Popular posts from this blog

Big Data Analytics Programs

  List of Programs for Big Data Analytics   CLICK ON ME 1.  Implement the following Data structures in Java       a)  Linked Lists            b)   Stacks       c)  Queues     d)   Set            e)   Map 2.  Perform setting up and Installing Hadoop in its three operating modes:      Standalone,     Pseudo distributed,     Fully distributed. 3.  Implement the following file management tasks in Hadoop:    a) Adding files and directories    b) Retrieving files    c) Deleting files 4. Run a basic Word Count Map Reduce program to understand Map Reduce Paradigm. 5. Write a Map Reduce program that mines weather data.     Weather sensors collecting data every hour at many locations across the globe gather a large volume of log data, which is a ...
Post a Comment
Read more

How to Install Parrot Operating System in Virtual Box using OVA

Step by Step Process of Parrot OS Installation What is Parrot OS Parrot is a free and open-source Linux system based on Debian that is popular among security researchers, security experts, developers, and privacy-conscious users. It comes with cyber security and digital forensics arsenal that is totally portable. It also includes everything you'll need to make your own apps and protect your online privacy. Parrot is offered in Home and Security Editions, as well as a virtual machine and a Docker image, featuring the KDE and Mate desktop environments. Features of Parrot OS The following are some of the features of Parrot OS that set it apart from other Debian distributions: Tor, Tor chat, I2P, Anonsurf, and Zulu Crypt, which are popular among developers, security researchers, and privacy-conscious individuals, are included as pre-installed development, forensics, and anonymity applications. It has a separate "Forensics Mode" that does not mount any of the system's hard...
Post a Comment
Read more

Hadoop file Management Tasks

  Implement the following file management tasks in Hadoop: a) Adding files and directories b) Retrieving files c) Deleting files Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into HDFS using one of the above command line utilities. Program:  The most common file management tasks in Hadoop includes: Adding files and directories to HDFS Retrieving files from HDFS to local filesystem Deleting files from HDFS Hadoop file commands take the following form:     hadoop fs - cmd Where cmd is the specific file command and <args> is a variable number of arguments. The command cmd is usually named after the corresponding Unix equivalent. For example, the command for listing files is ls as in Unix. a) Adding Files and Directories to HDFS Creating Directory in HDFS    $ hadoop fs - mkdir foldername (syntax)  $ ha...
Post a Comment
Read more