Installation and Configuration guide

Installation and Configuration guide for Hadoop Eco-system and other Big-Data Technologies

👉How to Install Java 8 on CentOS
👉How to Install MySql on CentOS
👉How to Install Hadoop 2.6 with Single Node on CentOS
👉How to Install Hive 0.14 with Single Node on CentOS
👉How to Install Pig 0.14 with Single Node on CentOS
👉How to Install Sqoop 1.4.4 with Single Node on CentOS
👉How to Install Flume 1.4.0 with Single Node on CentOS
👉How to Install HBase 0.98.4 with Single Node on CentOS
👉How to Install Drill 1.10.0 with Single Node on CentOS
👉How to Install Spark 1.6.1 with Single Node on CentOS
👉How to Install Zookeeper 3.4.6 with Single Node on CentOS
👉How to Install NiFi 1.3.0 with Single Node on CentOS

Read More

MongoDB Installation & Configuration on Ubuntu 64 Bit

MongoDB is Document-oriented database system. It is part of the NoSQL family of database systems. Instead of storing data in tables as is done in a "classical" relational database, MongoDB stores structured data as JSON-like documents.

  

Prerequisites

1. Java

2. Linux 64 Bit OS (Ubuntu/Debian)

Installing MongoDB

Import the public key used by the package management system.

$sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 –recv 0C49F3730359A14518585931BC711F9BA15703C6

Create the /etc/apt/sources.list.d/mongodb-org-3.4.list list file using the command appropriate for your version of Ubuntu.

Ubuntu 12.04: $sudo echo "deb [ arch=amd64 ] http://repo.mongodb.org/apt/ubuntu precise/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list Ubuntu 14.04: $sudo echo "deb [ arch=amd64 ] http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list Ubuntu 16.04: $sudo echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list.

Reload local package database

$sudo apt-get update

Reload Install the MongoDB packages. (Install the latest stable version of MongoDB).

$sudo apt-get install -y mongodb-org

Start MongoDB

$sudo service mongod start

Verify that MongoDB has started successfully

$sudo service mongod status

Check below file successfully created or not.

$cd /var/log/mongodb/ $ls –l mongod.log $cd /etc $ls -l mongod.conf

Stop Mongodb service and modify mongod.conf file in case of need to add Ip address. (Default bindIp: 127.0.0.1)

$sudo service mongod stop $sudo nano /etc/mongod.conf

storage:   dbPath: /var/lib/mongodb   journal:   enabled: true systemLog:   destination: file   logAppend: true   path: /var/log/mongodb/mongod.log net:   port: 27017   bindIp: 127.0.0.1,localhost

Change the read & write permission of data directory

$sudo chmod 776 /var/lib/mongodb/*

Check Mongodb start with both IP addresses.

$sudo service mongod start $netstat -nlp | grep 27017 tcp        0      0 localhost:27017      0.0.0.0:* LISTEN      - tcp        0      0 127.0.0.1:27017         0.0.0.0:* LISTEN      -

Enter into Mongodb shell with IP address.

$sudo mongo localhost:27017 MongoDB shell version v3.4.3 connecting to: localhost:27017 MongoDB server version: 3.4.3 >

Read More

Hive 0.14 Installation & Configuration

Apache Hive is It is an Open Source data warehousing system. It is exclusively used to query and analyze huge data sets stored in theHadoop storage.Hive gives a SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop.

Prerequisites

Java
Hadoop
Mysql or Any relational database.

Download and Installing Hive

Download Hive 0.14 using below command on /usr/local directory.

$ cd /usr/local $sudo wget http://archive.apache.org/dist/hive/hive-0.14.0/apache-hive-0.14.0-bin.tar.gz

Unpack apache-hive-0.14.0-bin.tar.gz file

$ sudo tar -xvf apache-hive-0.14.0-bin.tar.gz

Rename apache-hive-0.14.0-bin to hive 

$ sudo mv apache-hive-0.14.0-bin hive

Setting up environment for Hive

Edit ~/.bashrc file for set up the Hive environment by appending the following lines.

$ sudo nano ~/.bashrc

export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH

Reload the configuration file ~/.bashrc with the following command.

$ source ~/.bashrc

Copy hive-env.sh.template as hive-env.sh.

$ cd /usr/local/hive/conf $ sudo cp hive-env.sh.template hive-env.sh

Edit the hive-env.sh file by appending the following line.

$ sudo nano hive-env.sh

Add the below line to hive-env.sh file.

export HADOOP_HOME=/usr/local/hadoop

Configure Mysql Database

Check mysql service running or not

$ sudo service mysqld status

If mysql service is not running then start the service

$ sudo service mysqld start

Note:You also need a MySQL separate user account for Hive to use to access the metastore. It is very important to prevent this user account from creating or altering tables in the metastore database schema.
Login with root user (administrator credential)

$ mysql -u root -p $ Enter Password

Enter into Mysql database

Mysql>use mysql;

Creating New user

mysql> CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'hive@123';

Grant privilege to New user

mysql> GRANT all on *.* to 'hiveuser'@localhost identified by 'hive@123';

flush privileges to New user

mysql> flush privileges;

Configuring Hive metastore with Mysql.

Configuring Metastore means specifying to Hive where the database is stored. You can do this by editing the hive-site.xml file, which is in the /usr/local/hive/conf directory.
Copy the template file as hive-site.xml using the following command

$ cd /usr/local/hive/conf $ sudo cp hive-default.xml.template hive-site.xml

Edit hive-site.xml file using this command

$ sudo nano /usr/local/hive/conf/hive-site.xml

Before adding below properties, comment all existing properties within <Configuration></Configuration> tag.

<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost/hivemetastore?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Mysql JDBC Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value> <description>Username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive@123</value> <description>password to use against metastore database</description> </property>

Download mysql-connector-java-5.1.28.jar to /usr/local/hive/lib/ folder

$ cd /usr/local/hive/lib $sudo wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.28/mysql-connector-java-5.1.28.jar

Change the ownership and permissions of the directory /usr/local/hive

$ sudo chown -R hdfs:hdfs /usr/local/hive $ sudo chmod -R 755 /usr/local/hive

Check Hive Properly Installed

Check whether hive properly install or not (type hive and press Enter)

$ hive

It showing hive prompt

hive>

Also check in mysql whether hive database automatically created or not

$ mysql –u root –p Enter Password admin@123

Showing created database

Mysql>show databases; +--------------------+ | Database           | +--------------------+ | information_schema | | hivemetastore      | | mysql              | +--------------------+

.

Read More