Skip to main content

Big Data Analytics

 Big Data Analytics


CO1: Understanding BigData Defining Data, Types of Data, Structured Data, Semi-Structured Data, Unstructured Data, How data being Generated, Different sources of Data Generation, Rate at which data is being generated, Different V’s, Volume, Variety, Velocity, Veracity, Value, How a single person is contributing towards BigData, Significance for BigData, Reason for BigData, Understanding RDBMS and why it is failing to store BigData. Future of BigData, BigData use cases for major IT Industries.

PPT | Material | Sample Questions

CO2: Introduction to Hadoop What is Hadoop, Apache Community, Cluster, Node, Commodity Hardware, Rack Awareness, History of Hadoop, Need for Hadoop, How is Hadoop Important, Apache Hadoop Ecosystem, Different Hadoop offering, Hadoop 1.x Architecture, Apache Hadoop Framework, Master-Slave Architecture, Advantages of Hadoop. 

 PPT| Material | Sample Questions

CO3: Storage Unit Hadoop Distributed File System, Design of HDFS, HDFS Concept, How files are stored in HDFS, Hadoop File system, Replication factor, Name Node, Secondary Name Node, Job Tracker, Task Tracker, Data Node, FS Image, Edit-logs, Check-pointing Concept, HDFS Federation, HDFS High availability. Architectural description for Hadoop Cluster, When to use or not to use HDFS, Block Allocation in Hadoop Cluster, Read operation in HDFS, Write operation in HDFS, Hadoop Archives, Data Integrity in HDFS, Compression & Input Splits. 

Processing Unit What is MapReduce, History of MapReduce, How does MapReduce work, Input files, Input Format types Output Format Types, Text Input Format, Key-Value Input Format, Sequence File Input Format, Input split, Record Reader, MapReduce overview, Mapper Phase, Reducer Phase, Sort and Shuffle Phase, Importance of MapReduce, Data Flow, Counters, Combiner Function, Partition Function, Joins, Map Side Join, Reduce Side Join, MapReduce Web UI, Job Scheduling, Task Scheduling, Fault Tolerance, Writing MapReduce Application, Driver Class, Mapper Class, Reducer Class, Serialization, File-Based Data Structure, Writing a simple MapReduce program to Count Number of words, MapReduce WorkFlows.

PPT| Material | Sample Questions

CO4: YARN &Hadoop Cluster YARN, YARN Architecture, YARN Components, Resource Manager, Node Manager, Application Master, Concept of Container, Difference between Hadoop 1.x and 2.x Architecture, Execution of Job in Yarn Cluster, Comparing and Contrasting Hadoop with Relational Databases. Cluster Specification, Cluster Setup and Installation Creating Hadoop user Installing Hadoop, SSH Configuration, Hadoop Configuration, Hadoop daemon properties, Different modes of Hadoop, Standalone Mode, Pseudo Distributed Mode, Fully Distributed Modes.

PPT | Material | Sample Questions


Important Question cum Long Answer Questions

Comments

Post a Comment

Popular posts from this blog

Array of Objects

An array can be of any data type including struct. Similarly, we can also have arrays of variables of the type class. Such variables are called arrays of objects. Class Definition: class employee {           char name[30];          float age;     public:          void getdata(void);           void putdata(void); }; The identifier employee is a user-defined data type and can be used to create objects related to different employee categories. employee manage[3];          //aray of managers employee foreman[15];       //array of foreman employee worker[75];        // array of worker the array manager contains three objects(managers), namely, manager[0],  manager[1], and manager[2], of type employee class similarly, the foreman array contains 15 objects. and the worker array contains 75 objectives.(work...
Post a Comment
Read more

Binning Method by Data smoothing in python

 Binning Method Binning is a technique for smoothing data or dealing with noisy data. The data is sorted first, and then the sorted values are dispersed into a number of buckets or bins in this approach. Binning methods provide local smoothing since they consult the vicinity of values.  Smoothing can be accomplished in three ways: Bin smoothing entails:  Each value in a bin is replaced by the bin's mean value when smoothing by bin means is used.  Smoothing by bin median:  Each bin value is replaced by its bin median value in this method.  Smoothing by bin borders:  In smoothing by bin boundaries, the bin boundaries are determined as the minimum and maximum values in a given bin. The nearest boundary value is then used to replace each bin value. Example: Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34 Smoothing by bin means:       - Bin 1: 9, 9, 9, 9       - Bin 2: 23, 23, 23, 23   ...
Post a Comment
Read more

Hadoop file Management Tasks

  Implement the following file management tasks in Hadoop: a) Adding files and directories b) Retrieving files c) Deleting files Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into HDFS using one of the above command line utilities. Program:  The most common file management tasks in Hadoop includes: Adding files and directories to HDFS Retrieving files from HDFS to local filesystem Deleting files from HDFS Hadoop file commands take the following form:     hadoop fs - cmd Where cmd is the specific file command and <args> is a variable number of arguments. The command cmd is usually named after the corresponding Unix equivalent. For example, the command for listing files is ls as in Unix. a) Adding Files and Directories to HDFS Creating Directory in HDFS    $ hadoop fs - mkdir foldername (syntax)  $ ha...
Post a Comment
Read more