Home / Tag Archives: mapreduce

Tag Archives: mapreduce

301.2.7-Shuffle and reduce

Shuffle and Reduce

    Shuffle & Reduce Stage Reduce stage is the combination of the Shuffle stage and the Reduce stage. The Reducer’s job is to process the data that comes from the mapper. After processing, it produces a new set of output, which will be stored in the HDFS Scaling-up is …

Read More »

301.2.6-How Map Reduce Works

How MapReduce Works

  How MapReduce Works We will understand MapReduce with the help of an example Word count example Imagine a huge data set that contains user comments and discussions about a laptop We want to count the frequency of words to get an idea on the popular features Map Reduce Word …

Read More »

301.2.5-Map Reduce Alternatives

MapReduce Alternatives

  MapReduce programming is not easy MapReduce programs are written in Java Not everyone has Java background We need to be an expert in Java to write MapReduce code for intermediate and Complex problems Hadoop alone can’t make our life easy While HDFS and Map Reduce programming solves our issue …

Read More »

301.2.4-Map Reduce Code for Finding Average

MapReduce Code for Finding Average

  LAB: Map Reduce Function-Average Dataset: Stock_Price_Data/stock_price.txt The dataset contains stock price data collected for every minute. Find the average stock price. Solution Is the Hadoop started? jps Start Hadoop if not started already start-all.sh you can also use start-dfs.sh Is the Hadoop started now? jps check your files on …

Read More »

301.2.2-Map Reduce for word count

MapReduce for Word Count

  LAB: Map Reduce for word count Dataset User_Reviews/User_Reviews.txt Move the data to hdfs Write a word count program to count the frequency of each word Take the final output inside a text file Solution Is the Hadoop started? jps Start Hadoop if not started already start-all.sh You can also …

Read More »

301.2.1-MapReduce – Basics

MapReduce - Basics

Contents What is MapReduce Map Reduce Algorithm Map Reduce Program Word Count map reduce example MapReduce Programming Map Reduce Model Distributed Computing for distributed data Achieve the final objective by two stage Map and Reduce method of computation Map Reduce Programming style is inspired from functional programming languages (eg. Lisp) …

Read More »