Overview of the Types of Hadoop-based application:

Databases/Stores

? ? Avro: It lets us use date structures within context of Hadoop MapReduce jobs, so process data very efficiently.

????HBase: distributed non-relational database

? ? Cassandra: distributed data management system

Querying

? ? Pig: analyzing large data sets in HDFS, it has its own high-level language (Pig Latin) for you.

? ??Hive: Query and manage large datasets in HDFS or in HBase, with a SQL-like interface.

? ? Impala: High-performance and low-tatency query with SQL-like interface, providing from Cloudera VM (Hue).

? ? Spark: General processing engine for streaming, SQL, machine learning and graph processing

Machine Learning/Graph Processing

? ? Giraph: Iterative graph processing using Hadoop framework

? ? Mahout: Framework for machine learning applications using Hadoop, Spark.

? ? Spark: General processing engine for streaming, SQL, machine learning and graph processing

Apache Pig

Two componets:

? ? Own script language - PigLatin.?PigLatin can be embedded in host language like Java

????Infrastructure Layer - it takes what we wrote in PigLatin, and transforms into back-end jobs of Tez or MapReduce, etc.

Usage:

? ? extract / transform / load / handling "raw" data. more

Extendibility:

????It has built-in operators and functions, as well as supporting us to write constant functions if we have complex processing to do.

Use cases:

Step1 . Put a passwd into HDFS

? ? [cloudera@quickstart ~]$ hadoop fs -put /etc/passwd /user/cloudera/

? ????? * Command "hadoop fs ..." == "hdfs dfs ..."

Step 2. With ?MapReduce as execution type and launch Pig inteactive shell "grunt"

? ??[cloudera@quickstart ~]$ pig -x mapreduce ? ? ? ? ? --> it use MapReduce to track data and print it out.

? ? grunt>

Step 3. wrote PigLatin - PigLatin need ';' to end one command. Like below:

? ? ?grunt> A = load '/user/cloudera/passwd' using PigStorage(':'); ? ? ? ?--> telling the deparator is colon

? ? grunt> B = foreach A generate $0, $4, $5 ; ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? --> doing the sub-setting part

? ? grunt> dump B;

Step 4. Store B output into HDFS

? ? grunt> store B into 'userinfo.out';

? ? grunt> quit;

Step 5. Check the result

? ???[cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/

Apache Hive

Two componets:

????SQL Language - HiveQL.?

????Interactive Client - beeline / Hive own CLI / Hcatalog / WebHcat. It takes what we wrote in HiveQL, and transforms into back-end jobs of Tez or Spark, MapReduce, Yarn, etc.

Usage:

? ? As Data warehouse software, handling data in HDFS, HBase. It can do:

? ? ? ? Date mining, analytics

? ? ? ? Machine Learning

? ? ? ? Ad hoc analysis

Extendibility:

????It has built-in operators and functions.

Use cases:

? ??Step 1. Put a passwd into HDFS

? ????? [cloudera@quickstart ~]$ hadoop fs -put /etc/passwd /tmp/

????Step 2. lauch beeline with DB URL

? ??????[cloudera@quickstart ~]$ beeline -u jdbc:hive2://

? ? step 3. Create table 'userinfo'

? ? Step 4. Overwrite table with the data from HDFS /tmp/passwd ?and then do querying

Apache HBase

Two componets:

????SQL Language -?like Hive, Spart, Impala.

????Interactive Shell - hbase shell (Other options: HBase MapReduce / HBase API/ HBase External API).?It takes what we wrote, and runs it on top of HDFS.

Usage:

????Scalable data store as Non-relational distributed database

Feature:

? ??Compression - lower the network traffic and the size of data on the disk

????In-memory operations - MemStore, BlockCache

? ? Consistency - data transation between ?? without intermediate changes

? ? High Availability - spreads out "keys" across nodes/various regions, and it has its owne replication as well as HDFS replication mechanism.

? ? Automatic Shareing - table is distributed in regions that could benifit performance

? ? Security - authorization process for both client side and server side

Use cases:

? ??Step 1. Launch hbase shell

????????[cloudera@quickstart ~]$ hbase shell