NiFi 1.3.0 Installation & Configuration

Apache NiFi is an open source project which enables the enterprise integration and dataflow automation tool that allows a user to send, receive, route, transform, and sort data, as needed, in an automated and configurable way.

Prerequisites

Java 8

Linux Operating Systems

Supported Web Browsers

Downlod and Installing NiFi

Download NiFi 1.3.0 using below command on /usr/local directory.

$cd /usr/local $sudo wget http://www-us.apache.org/dist/nifi/1.3.0/nifi-1.3.0-bin.tar.gz

Unpack the compressed tar file by using this command

$ sudo tar -xvf nifi-1.3.0-bin.tar.gz

Rename nifi-1.3.0 directory to nifi in /usr/local directory by using give command.

$ sudo mv nifi-1.3.0 nifi

Setting up NiFi Environment Variables

First we need to set environment variable for NiFi. Edit ~/.bashrc file.

# sudo nano ~/.bashrc

Append following values at end of file and save the file.

export NIFI_HOME=/usr/local/nifi export PATH=$NIFI_HOME/bin:$PATH

Change the ownership and permissions of the directory /usr/local/nifi

$ sudo chown -R hdfs:hdfs /usr/local/nifi $ sudo chmod -R 755 /usr/local/nifi

Reload the configuration file ~/.bashrc with the following command.

$ source ~/.bashrc

Edit Configuration Files

Edit nifi.properties file.

$ cd /usr/local/nifi/conf $ sudo nano nifi.properties

Change the value for below properties as hostname/server IP and save It.

nifi.web.http.host=localhost nifi.web.http.port=8089

Start and verify NiFi Service

Start & Stop NiFi services

$ cd /usr/local/nifi $ ./bin/nifi.sh start $ ./bin/nifi.sh stop OR $ nifi.sh start $ nifi.sh stop

Get status of NiFi services

$ ./bin/nifi.sh status OR $ nifi.sh status

Showing service is running

2018-01-11 19:53:02,974 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 42449, PID=1295

NiFi Web Interfaces

Access your NiFi Web Interfaces on port 8089 in your favourite web browser.

http://localhost:8089/nifi

Showing NiFi Web Interface.

Read More

Spark Installation & Configuration

   

Apache Spark is an open source parallel processing framework for running large-scale data analytics applications across clustered computers. It can handle both batch and real-time analytics and data processing workloads. Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.

Prerequisites:

Java 8
Hadoop 2.6
Scala (Spark comes prebuilt with Hadoop and scala)

Downloading Apache Spark

Download spark-1.6.1 using below command on /usr/local directory.

$ cd /usr/local $ sudo wget https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz

Extract the Spark tar file

$ sudo tar -xvf spark-1.6.1-bin-hadoop2.6.tgz $ sudo mv spark-1.6.1-bin-hadoop2.6 spark

Set Environment veriable

First we need to set environment variable for java. Edit ~/.bashrc file.

# nano ~/.bashrc

Append following values at end of file and save the file.

export SPARK_HOME=/usr/local/spark export PATH=$ SPARK_HOME/bin:$PATH

Change the ownership and permissions of the directory /usr/local/spark

$ sudo chown -R hdfs:hdfs /usr/local/spark $ sudo chmod -R 755 /usr/local/spark

For spark-sql, copy hive-site.xml file to /usr/local/spark/conf folder.

$ sudo cp /usr/local/hive/conf/hive-site.xml /usr/local/spark/conf/

Edit hive-site.xml and add the following code in the file

$ sudo nano /usr/local/spark/conf/hive-site.xml

<property>   <name>hive.metastore.uris</name>   <value>thrift://localhost:9083</value> </property>

Start the Spark Services

Start the spark service using following command.

$ cd /usr/local/spark/sbin $ ./start-all.sh

Get spark-shell prompt using following command.

$ cd /usr/local/spark/bin $ ./spark-shell

Get spark-sql prompt using following command.

$ cd /usr/local/spark/bin $ ./spark-sql

Read More

Drill 1.10.0 Installation & Configuration

Apache Drill is an open-source software framework that supports data-intensive distributed applications for interactive analysis of large-scale datasets. Apache Drill is first schema-free SQL engine. Unlike Hive, it does not use MR job internally and compared to most distributed query engines, it does not depend on Hadoop.

Prerequisites

Java 8
ZooKeeper quorum

Download and Installing Drill

Download Drill 1.10.0 using below command on /usr/local directory.

$ cd /usr/local $ sudo wget http://www-eu.apache.org/dist/drill/drill-1.10.0/apache-drill-1.10.0.tar.gz

Unpack the compressed tar file by using this command

$ sudo tar -xvf apache-drill-1.10.0.tar.gz

Rename apache-drill-1.10.0 directory to drill in /usr/local directory by using give command

$ sudo mv apache-drill-1.10.0 drill

Setting up Drill Environment Variables

First we need to set environment variable for Drill. Edit ~/.bashrc file.

$ sudo nano ~/.bashrc

Append following values at end of file and save the file.

export DRILL_HOME=/usr/local/drill export PATH=$DRILL_HOME/bin:$PATH

Change the ownership and permissions of the directory /usr/local/drill

$ sudo chown -R hdfs:hdfs /usr/local/drill $ sudo chmod -R 755 /usr/local/drill

Reload the configuration file ~/.bashrc with the following command.

$ source ~/.bashrc

Start Drill services

To stat Drill service used following command

$ drill-embedded

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Jul 20, 2017 7:19:59 PM org.glassfish.jersey.server.ApplicationHandler initialize INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26... apache drill 1.10.0 "say hello to my little drill"

Drill Web Interfaces

Access your Drill Web Interfaces on port 8047 in your favorite web browser.

http://localhost:8047

Showing below web Interface

Read More

HBase 0.98.4 Installation & Configuration

HBase is a column-oriented database management system that runs on top of Hadoop Distributed File System (HDFS). HBase can host very large tables, billions of rows, millions of columns and can provide real-time, random read/write access to Hadoop data.HBase scales linearly across very large datasets and easily combines data sources with different structures and schemas.

Prerequisites

Java
Hadoop 

Download HBase File

Download HBase 0.98.4 using below command on /usr/local directory.

$sudo wget http://archive.apache.org/dist/hbase/hbase-0.98.4/hbase-0.98.4-hadoop2-bin.tar.gz /usr/local/

Unpack  hbase-0.98.4-hadoop2-bin.tar.gz file.

$ cd /usr/local $ sudo tar –xvf hbase-0.98.4-hadoop2-bin.tar.gz

Rename hbase-0.98.4-hadoop2-bin to hbase

$ sudo mv hbase-0.98.4-hadoop2 hbase

Setting up environment for HBase

Edit ~/.bashrc file for set up the HBase environment by appending the following lines.

$ sudo nano ~/.bashrc

export HBASE_HOME=/usr/local/hbase export PATH=$HBASE_HOME/bin:$PATH

Reload the configuration file ~/.bashrc with the following command.

# source ~/.bashrc

Edit hbase-env.sh file.

$ cd /usr/local/hbase/conf $ sudo nano hbase-env.sh

Set value for following environment variable.

export JAVA_HOME=/usr/local/jdk export HBASE_REGIONSERVERS=/usr/local/hbase/conf/regionservers export HBASE_MANAGES_ZK=true

Change the ownership and permissions of the directory /usr/local/hbase

$ sudo chown -R hdfs:hdfs /usr/local/hbase $ sudo chmod -R 755 /usr/local/hbase

Edit Configuration Files

Edit hbase-site.xml and place the following properties inside the file.

$ cd /usr/local/hbase/conf $ sudo nano hbase-site.xml

<configuration>  <property>   <name>hbase.rootdir</name>   <value>hdfs://localhost:9000/hbase</value>  </property>  <property>   <name>hbase.cluster.distributed</name>   <value>true</value>  </property>  <property>   <name>hbase.zookeeper.quorum</name>   <value>localhost</value>  </property>  <property>   <name>dfs.replication</name>   <value>1</value>  </property>  <property>   <name>hbase.zookeeper.property.clientPort</name>   <value>2181</value>  </property>  <property>   <name>hbase.zookeeper.property.dataDir</name>   <value>/home/hduser/hbase/zookeeper</value>  </property> </configuration>

Check Version

Check the version of HBase

$ hbase version 2017-04-27 14:25:17,790 INFO  [main] util.VersionInfo: HBase 0.98.4-hadoop2 2017-04-27 14:25:17,791 INFO  [main] util.VersionInfo: Subversion git://acer/usr/src/hbase -r 890e852ce1c51b71ad180f626b71a2a1009246da 2017-04-27 14:25:17,791 INFO  [main] util.VersionInfo: Compiled by apurtell on Mon Jul 14 19:45:06 PDT 2014

Start and verify the Services

Start HBase Server

$ start-hbase.sh

To verify the services will be start or not using following command.

$ jps

It should be provide following list of services.

15230 HMaster 15441 HRegionServer

Check whether HBaseproperly install or not

$ hbase shell

It showing HBase prompt

hbase(main):001:0>

To Stop HBase Server

$ stopt-hbase.sh

HBase Web Interfaces

Access your HBase Master server on port 60010 in your favourite web browser.

http://localhost:60010

Access your HBase Master Region server on port 60030 in your favourite web browser

http://localhost:60030

Read More