Wednesday, 13 March 2019

Cloud Computing


Cloud is a term referring to accessing computer, information technology (IT), and software applications through a network connection. The term Cloud refers to a Network or Internet. In other words, we can say that Cloud is something, which is present at remote location



What is Cloud Computing?

Cloud Computing is the on-demand delivery of computer Power,Database store,Application and other IT-resources through a cloud services platform via the internet with pay-as-you-go pricing.


Benefit of Cloud Computing

Trade Capital expense for variable Expense
Instead of having to invest heavily in data center and server before you know how your going to use them, you can only pay when you consume computing resources and only pay for how much you consume.

Stop Guessing Capacity
Eliminate guessing on your infrastructure capacity needs.When you make a capacity decision prior to deploying an application either sitting with expensive resources or dealing with limited capacity. Using cloud you can access as much or as little as you need.

Increase speed & agility
In a cloud Computing environment ,reduce the time it takes to make IT resources available to your developer from weeks to just minutes.

Stop Spending Money on running and maintaining data center.
Focus on project that differentiate your business, not the infrastructure.

Go global in minutes
Easily deploy your application in multiple region around the world.This means you can provide a lower latency and better experience for your costumers.


Essential Cloud Computing Characteristics

On-demand self-service
Cloud services are expected to be available on-demand. A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service provider.

Broad network access
Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).

Resource pooling
The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state or data center). Examples of resources include storage, processing, memory and network bandwidth.

Rapid elasticity
Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time.

Measured service
Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth and active user accounts). Resource usage can be monitored, controlled and reported, providing transparency for the provider and consumer.

Cloud Deployment Models


Private Cloud: The Private Cloud allows systems and services to be accessible within an organization. It may be owned, managed and operated by organization or third party and it may exist on or off  premises.

Benefits
  • Private cloud operations are not available to general public so its provide high security and privacy.
  • It provide more control on its resources because it is accessed only within an organization.
  • Private cloud resources are not as cost effective as public clouds but they offer more efficiency.
Disadvantage
  • Private cloud is only accessible locally and is very difficult to deploy globally
  • Private cloud can be scaled only within capacity of internal hosted resources.
  • To maintain the cloud, Organization requires more skilled and expertise.

Community Cloud: The Community Cloud allows systems and services to be accessible by group of organizations.It may be owned, manage and operated by one or more organizations. It may exist on or off premises.

Benefits
  • Community cloud provides an infrastructure to share cloud resources and capabilities among several organizations.
  • Community cloud is comparatively more secure than the public cloud.
Disadvantages
  • Data might be accessible by others.
  • Challenging to allocate responsibilities of governance, security and cost.

Public Cloud: The Public Cloud allows systems and services to be easily accessible to the general public. It may be owned, managed and operated by business or government organization. It exist on the premises of cloud provider.

Benefits
  • Public cloud share same resources with large number of consumer, it has low cost.
  • Easy to integrate public cloud with private cloud, hence gives consumers a flexible approach.
  • Public cloud services are delivered through Internet, therefore ensures location independence.
  • Public cloud is also based on pay-per-use model.
  • Resources can be scaled up or down according the requirement.
Disadvantages
  • Public cloud model, Resources are shared publicly, therefore low level of security.
  • It is comparatively less customizable than private cloud.

Hybrid Cloud: The Hybrid Cloud is composition of public and private cloud. However, the critical activities are performed using private cloud while the non-critical activities are performed using public cloud.

Benefits
  • It offers both features of public cloud scalability and private cloud scalability.
  • Private cloud in hybrid cloud ensures higher degree of security.
  • Public cloud are more cost effective than private, therefore hybrid cloud can have this saving.
Disadvantages
  • Networking becomes complex due to presence of private and public cloud.
  • It is necessary to ensure that cloud services are compliant with organization's security policies.
  • Hybrid cloud model is dependent on internal IT infrastructure.


Cloud Service Models


Infrastructure as a Service (IaaS): Infrastructure as a Service (IaaS) IaaS is the delivery of technology infrastructure as an on demand scalable service. IaaS provides access to fundamental resources such as physical machines, virtual machines, virtual storage, etc

Advantages
  • Consumer gets access to the Infrastructure to deploy their stuff.
  • Manage OS, Storage, Network Component and deployed Application.
  • Does not manage or control core Infrastructure.
Examle of Iaas: EC2

Platform as a Service (PaaS): Platform as a Service (PaaS) PaaS provides the runtime environment for applications, development & deployment tools, etc. PaaS provides all of the facilities required to support the complete life cycle of building and delivering web applications and services entirely from the Internet. Typically applications must be developed with a particular platform in mind.

Advantages
  • Consumer control over deployed application on a cloud
  • No need  to bother about the administration because it's the responsibility of cloud provider.
  • No need to manage or control OS, Storage, Network Component .
  • Very easy to scale up or down automatically based on application resource demands.
Example of Paas: Elastic Beanstalk

Software as a Service (SaaS): Software as a Service (SaaS) SaaS model allows to use software applications as a service to end users. SaaS is a software delivery methodology that provides licensed multi-tenant access to software and its functions remotely as a Web-based service.

Advantages
  • No need to manage or control cloud infrastructure, OS, Storage, Network Component and individual application. 
  • They can be scaled up or down on demand.
  • They are automatically upgraded and updated.
  • Application make it available over a network.
Example of Saas: Microsoft Office 365, DropBox.
Share:

Wednesday, 9 January 2019

NiFi 1.3.0 Installation & Configuration

Apache NiFi is an open source project which enables the enterprise integration and dataflow automation tool that allows a user to send, receive, route, transform, and sort data, as needed, in an automated and configurable way.




Prerequisites

Java 8
Linux Operating Systems
Supported Web Browsers

Downlod and Installing NiFi

Download NiFi 1.3.0 using below command on /usr/local directory.

$cd /usr/local
$sudo wget http://www-us.apache.org/dist/nifi/1.3.0/nifi-1.3.0-bin.tar.gz
Unpack the compressed tar file by using this command
$ sudo tar -xvf nifi-1.3.0-bin.tar.gz
Rename nifi-1.3.0 directory to nifi in /usr/local directory by using give command.
$ sudo mv nifi-1.3.0 nifi

Setting up NiFi Environment Variables

First we need to set environment variable for NiFi. Edit ~/.bashrc file.
# sudo nano ~/.bashrc
Append following values at end of file and save the file.
export NIFI_HOME=/usr/local/nifi
export PATH=$NIFI_HOME/bin:$PATH
Change the ownership and permissions of the directory /usr/local/nifi
$ sudo chown -R hdfs:hdfs /usr/local/nifi
$ sudo chmod -R 755 /usr/local/nifi
Reload the configuration file ~/.bashrc with the following command.
$ source ~/.bashrc

Edit Configuration Files

Edit nifi.properties file.
$ cd /usr/local/nifi/conf
$ sudo nano nifi.properties
Change the value for below properties as hostname/server IP and save It.
nifi.web.http.host=localhost
nifi.web.http.port=8089

Start and verify NiFi Service

Start & Stop NiFi services
$ cd /usr/local/nifi
$ ./bin/nifi.sh start
$ ./bin/nifi.sh stop
OR
$ nifi.sh start
$ nifi.sh stop
Get status of NiFi services
$ ./bin/nifi.sh status
OR
$ nifi.sh status
Showing service is running
2018-01-11 19:53:02,974 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 42449, PID=1295

NiFi Web Interfaces

Access your NiFi Web Interfaces on port 8089 in your favourite web browser.
Showing NiFi Web Interface.



Share:

Thursday, 15 November 2018

Spark Installation & Configuration

   
Apache Spark is an open source parallel processing framework for running large-scale data analytics applications across clustered computers. It can handle both batch and real-time analytics and data processing workloads. Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.


Prerequisites:

Java 8
Hadoop 2.6
Scala (Spark comes prebuilt with Hadoop and scala)

Downloading Apache Spark

Download spark-1.6.1 using below command on /usr/local directory.
$ cd /usr/local
$ sudo wget https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz
Extract the Spark tar file
$ sudo tar -xvf spark-1.6.1-bin-hadoop2.6.tgz
$ sudo mv spark-1.6.1-bin-hadoop2.6 spark

Set Environment veriable

First we need to set environment variable for java. Edit ~/.bashrc file.
# nano ~/.bashrc
Append following values at end of file and save the file.
export SPARK_HOME=/usr/local/spark
export PATH=$ SPARK_HOME/bin:$PATH
Change the ownership and permissions of the directory /usr/local/spark
$ sudo chown -R hdfs:hdfs /usr/local/spark
$ sudo chmod -R 755 /usr/local/spark
For spark-sql, copy hive-site.xml file to /usr/local/spark/conf folder.
$ sudo cp /usr/local/hive/conf/hive-site.xml /usr/local/spark/conf/

Edit hive-site.xml and add the following code in the file
$ sudo nano /usr/local/spark/conf/hive-site.xml
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9083</value>
</property>

Start the Spark Services

Start the spark service using following command.
$ cd /usr/local/spark/sbin
$ ./start-all.sh
Get spark-shell prompt using following command.
$ cd /usr/local/spark/bin
$ ./spark-shell
Get spark-sql prompt using following command.
cd /usr/local/spark/bin
$ ./spark-sql
Share:

Monday, 8 October 2018

Drill 1.10.0 Installation & Configuration

Apache Drill is an open-source software framework that supports data-intensive distributed applications for interactive analysis of large-scale datasets. Apache Drill is first schema-free SQL engine. Unlike Hive, it does not use MR job internally and compared to most distributed query engines, it does not depend on Hadoop.





Prerequisites

Java 8
ZooKeeper quorum

Download and Installing Drill

Download Drill 1.10.0 using below command on /usr/local directory.
$ cd /usr/local
$ sudo wget http://www-eu.apache.org/dist/drill/drill-1.10.0/apache-drill-1.10.0.tar.gz
Unpack the compressed tar file by using this command
$ sudo tar -xvf apache-drill-1.10.0.tar.gz
Rename apache-drill-1.10.0 directory to drill in /usr/local directory by using give command
$ sudo mv apache-drill-1.10.0 drill

Setting up Drill Environment Variables

First we need to set environment variable for Drill. Edit ~/.bashrc file.
$ sudo nano ~/.bashrc
Append following values at end of file and save the file.
export DRILL_HOME=/usr/local/drill
export PATH=$DRILL_HOME/bin:$PATH
Change the ownership and permissions of the directory /usr/local/drill
$ sudo chown -R hdfs:hdfs /usr/local/drill
$ sudo chmod -R 755 /usr/local/drill
Reload the configuration file ~/.bashrc with the following command.
$ source ~/.bashrc

Start Drill services

To stat Drill service used following command
$ drill-embedded
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Jul 20, 2017 7:19:59 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"say hello to my little drill"

Drill Web Interfaces

Access your Drill Web Interfaces on port 8047 in your favorite web browser.

Showing below web Interface


Share:

Total Pageviews

Lables

Powered by Blogger.

Followers