Setting up Hive 1.2.1 - Hadoop @ Desk (Single Node Cluster)

 

Hope you've setup your Hadoop Single Node Cluster @ Your Desk.

In this tutorial, we will setup and test Hive 1.2.1. (Before you start, snapshot your VM, if not already done)

Note: You need to change paths as per your environment (i.e in my case I'm using '/media/SYSTEM', you've to replace it with yours)

Steps below: (I've compiled the steps from here and here)

1. Start your VM (Or Host, if you've installed Hadoop directly on Host)

2. Get Hive 1.2.1 and move to our dedicated partition (as that of Hadoop) for better management

$ su hduser
$ cd
$ wget http://archive.apache.org/dist/hive/stable/apache-hive-1.2.1-bin.tar.gz
$ tar -xzvf apache-hive-1.2.1-bin.tar.gz
$ mkdir -p /media/SYSTEM/hadoop/hive/apache-hive-1.2.1/
$ mv apache-hive-1.2.1-bin/ /media/SYSTEM/hadoop/hive/apache-hive-1.2.1/

3. Update .bashrc file, to have 'hive' specific configuration

$ vi ~/.bashrc

#HIVE VARIABLES START
HIVE_HOME=/media/SYSTEM/hadoop/hive/apache-hive-1.2.1
export HIVE_HOME
export PATH=$PATH:$HIVE_HOME/bin
#HIVE VARIABLES END

4. Update hive-config.sh, to have Hadoop Home directory

Append the export line, to the file

$ vi /media/SYSTEM/hadoop/hive/apache-hive-1.2.1/bin/hive-config.sh 

export HADOOP_HOME=/media/SYSTEM/hadoop/hadoop-2.7.0

5. Start hadoop, if not already done
 

$ start-all.sh

6. Create Hive specific directories

$ hadoop fs -mkdir /tmp && hadoop fs -mkdir -p /user/hive/warehouse && hadoop fs -chmod g+w /tmp && hadoop fs -chmod g+w /user/hive/warehouse
 

$ cp /media/SYSTEM/hadoop/hive/apache-hive-1.2.1/lib/jline-2.12.jar /media/SYSTEM/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/

7. Stop Hadoop and Reboot

$ stop-all.sh
$ sudo reboot

8. Start Hadoop and then Hive

$ su hduser
$ cd
$ start-all.sh
$ hive

9. Test Hive (On the hive prompt Create a Hive Table, Do some insert and Select)

hive> !clear;
hive> create table employee(name string, id int);
hive> insert into employee values('george',1);
hive> insert into employee values('mathew',2);
hive> select name from employee where id = 2;
hive>quit;

Note: you can actually see Map Reduce jobs are being created on the fly while executing these commands




11. Stop Hadoop, Shutdown and Snapshot your VM

$ stop-all.sh
$ sudo shutdown now