Please see my other blog for Oracle EBusiness Suite Posts - EBMentors

Search This Blog

Note: All the posts are based on practical approach avoiding lengthy theory. All have been tested on some development servers. Please don’t test any post on production servers until you are sure.

Monday, July 31, 2017

Installing/Configuring Hortonworks Data Platform [HDP]

Ambari is  completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop. As part of the Hortonworks Data Platform, allows enterprises to plan, install and securely configure HDP making it easier to provide ongoing cluster maintenance and management, no matter the size of the cluster.


I've total 6 nodes, I'll use one node for Ambari Server and remaining nodes for Hadoop Cluster.
EN01 ==> Ambari Server
NN01 ==> Name Node
DN01 ==> Data Node 1
DN02 ==> Data Node 2
DN03 ==> Data Node 3
DN04 ==> Hadoop Server components + Clients

Apache Ambari Installation

Download Ambari from below location

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html

[hdpsysuser@en01 ~]$ su -
Password:
Last login: Mon Jul 17 16:01:30 AST 2017 on pts/0

[root@nn01 ~]# pwd
/root

[root@en01 ~]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari.repo -O /etc/yum.repos.d/ambari.repo
2017-07-25 11:11:00 URL:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari.repo [304/304] -> "/etc/yum.repos.d/ambari.repo" [1]
[root@nn01 ~]# yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
ambari-2.5.0.3                                                                                                            | 2.9 kB  00:00:00
ambari-2.5.0.3/primary_db                                                                                                 | 8.5 kB  00:00:00
repo id                                                      repo name                                                                     status
ambari-2.5.0.3                                               ambari Version - ambari-2.5.0.3                                               12
repolist: 12



[root@en01 ~]#  yum install ambari-server
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package ambari-server.x86_64 0:2.5.0.3-7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================
 Package                              Arch                          Version                          Repository                             Size
=================================================================================================================================================
Installing:
 ambari-server                        x86_64                        2.5.0.3-7                        ambari-2.5.0.3                        663 M

Transaction Summary
=================================================================================================================================================
Install  1 Package

Total download size: 663 M
Installed size: 722 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7Server/ambari-2.5.0.3/packages/ambari-server-2.5.0.3-7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 07513cad: NOKEY
Public key for ambari-server-2.5.0.3-7.x86_64.rpm is not installed
ambari-server-2.5.0.3-7.x86_64.rpm                                                                                        | 663 MB  00:17:14
Retrieving key from http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
Importing GPG key 0x07513CAD:
 Userid     : "Jenkins (HDP Builds) <jenkin@hortonworks.com>"
 Fingerprint: df52 ed4f 7a3a 5882 c099 4c66 b973 3a7a 0751 3cad
 From       : http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : ambari-server-2.5.0.3-7.x86_64                                                                                                1/1
  Verifying  : ambari-server-2.5.0.3-7.x86_64                                                                                                1/1

Installed:
  ambari-server.x86_64 0:2.5.0.3-7

Complete!
[root@nn01 ~]#


Set Up the Ambari Server


[root@en01 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/default
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK


About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.5.0.3.7.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


Installing, Configuring, and Deploying a Cluster

1- Login to Ambari server
By default Ambari listens on 8080, so if this port is already used stop the related service first and then runt the  Ambari server.

[root@en01 ~]# cd /usr/hadoopsw/prestoadmin/presto-admin-install/bin/
[root@en01 bin]# ./presto-admin server stop
[localhost] out: Stopping presto
[localhost] out: Stopped 8255
[localhost] out:

[root@en01 bin]# ambari-server status
Using python  /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid

[root@en01 bin]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...............
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

[root@en01 bin]# ambari-server status
Using python  /usr/bin/python
Ambari-server status
Ambari Server running
Found Ambari Server PID: 4126 at: /var/run/ambari-server/ambari-server.pid

2- Now log in to Ambari Web using a web browser, default user name/password: admin/admin. For a new cluster, the Cluster Install wizard displays a Welcome page. choose Launch Install Wizard.
http://<<yourServer>>:8080/#/login
http://en01:8080/#/login



Follow the instructions


For the next steps you need to run the below commands' output to be provided in the Ambari wizard.

[root@en01 ~]# hostname --fqdn
nn01

[root@en01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
70:30:60:31:6f:20:14:21:26:d5:3b:a3:e7:fd:c3:f0 root@nn01
The key's randomart image is:
+--[ RSA 2048]----+
|o+*+*oo          |
|o. o.+ o         |
|     .+ .        |
|    +. o         |
|   . o  S        |
|  . . .          |
|   o . +         |
|    . . E        |
|       ...       |
+-----------------+

[root@en01 ~]# ssh-copy-id root@dn01
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@dn01's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@dn01'"
and check to make sure that only the key(s) you wanted were added.

[root@en01 ~]# ssh root@dn01
Last login: Mon Jul 24 16:40:10 2017

[root@en01 ~]#  ssh-copy-id root@nn01
[root@en01 ~]# ssh-copy-id root@dn02
[root@en01 ~]# ssh-copy-id root@dn03


[root@en01 ~]# ssh root@dn02
[root@en01 ~]# ssh root@dn03
[root@en01 ~]# ssh root@dn04

[root@en01 ~]# cat /root/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAyNIU01WyjpZDMZaQ4ZA/oR1NVL7RsgJWU8N2colXOsDkJtG3
812Yh7XNU9kQxH1XFDVQ1J1a+rD2yxXyOjyiVFHFTq6op+BN6iZcQIEZnmf0/Sz1
...
...
-----END RSA PRIVATE KEY-----

ntpd should be running on all hosts, time should be synchronized with NTP server
[root@en01 ~]# ntpdate -u 192.168.0.90
[root@dn01 ~]# service ntpd start
[root@nn01 ~]# ntpdate -u 192.168.0.90
[root@dn01 ~]# ntpdate -u 192.168.0.90
[root@dn02 ~]# ntpdate -u 192.168.0.90
[root@dn03 ~]# ntpdate -u 192.168.0.90
[root@dn04 ~]# ntpdate -u 192.168.0.90

If you face any host check issues resolve them
[root@dn03 ~]# ps -eaf | grep java




[hdpsysuser@dn01 ~]$ w
 16:38:03 up 84 days,  1:10,  3 users,  load average: 0.17, 0.06, 0.06
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    dn04  16:37   41.00s  8.54s  3.23s /usr/bin/python /usr/libexec/urlgrabber-ext-down
hdpsysus pts/1    ws9374.elm.com.s 16:37    3.00s  0.16s  0.01s w













For Hive below MySQL user used.
mysql> create database hive;
mysql> CREATE USER 'hive'@'en01' IDENTIFIED BY 'P@ssw0rd';
mysql> GRANT all on *.* to 'hive'@'%' identified by 'P@ssw0rd' with grant option;
mysql>  flush privileges;
[root@en01 ~]# mysql -u hive -p

Setup ambari to use MySQL JDBC Driver
[root@en01 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/hadoopsw/apache-hive-2.1.1-bin/lib/mysql-connector-java-5.1.42-bin.jar

Using python  /usr/bin/python
Setup ambari-server
Copying /usr/hadoopsw/apache-hive-2.1.1-bin/lib/mysql-connector-java-5.1.42-bin.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.42-bin.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.


Install Required RH Packages (if required)

I had to install the below packages 

subscription-manager register
Username: xxxxxxxx
Password:xxxxxxx

subscription-manager repos
subscription-manager list --available --all
subscription-manager  attach --pool=8a85f98156fa00f40156fa51506e4415
subscription-manager repos
yum install redhat-lsb-core
rpm -qa | grep redhat-lsb
subscription-manager unregister

[root@nn01 ~]# yum repolist
[root@nn01 ~]# yum-config-manager --enable rhel-7-server-optional-rpms
[root@nn01 ~]# yum install libtirpc-devel
[root@dn01 ~]# yum install python-devel


Uninstall Ambari
If your installation goes with something wrong, you may need to uninsall Ambari and reinstall.


[root@dn04 ~]# ps aux | grep ambari-server
root      4126  0.4  2.9 10520524 713096 ?     Sl   Jul24   5:38 /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer
root      8841  0.0  0.0 112648   960 pts/0    S+   11:16   0:00 grep --color=auto ambari-server

[root@dn04 ~]# rpm -qa | grep ambari
ambari-server-2.5.0.3-7.x86_64

Following steps: On the Server node
[root@dn04 ~]# ambari-server stop
Using python  /usr/bin/python
Stopping ambari-server
Waiting for server stop...
Ambari Server stopped

[root@dn04 ~]# ambari-server reset
Using python  /usr/bin/python
Resetting ambari-server
**** WARNING **** You are about to reset and clear the Ambari Server database. This will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard.
Are you SURE you want to perform the reset [yes/no] (no)? yes
Confirm server reset [yes/no](no)? yes
Resetting the Server database...
Creating schema and user...
done.
Creating tables...
done.
Ambari Server 'reset' completed successfully.

[root@dn04 ~]# ambari-agent stop

[root@dn04 ~]# yum erase ambari-server
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package ambari-server.x86_64 0:2.5.0.3-7 will be erased
--> Finished Dependency Resolution
ambari-2.5.0.3                                                                                                            | 2.9 kB  00:00:00
https://www.apache.org/dist/cassandra/redhat/311x/repodata/repomd.xml: [Errno 12] Timeout on https://www.apache.org/dist/cassandra/redhat/311x/repodata/repomd.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received')
Trying other mirror.
https://www.apache.org/dist/cassandra/redhat/311x/repodata/repomd.xml: [Errno 12] Timeout on https://www.apache.org/dist/cassandra/redhat/311x/repodata/repomd.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received')
Trying other mirror.
cassandra/signature                                                                                                       |  819 B  00:00:00
cassandra/signature                                                                                                       | 2.9 kB  00:00:00 !!!
mysql-connectors-community/x86_64                                                                                         | 2.5 kB  00:00:00
mysql-connectors-community/x86_64/primary_db                                                                              |  16 kB  00:00:00
mysql-tools-community/x86_64                                                                                              | 2.5 kB  00:00:00
mysql57-community/x86_64                                                                                                  | 2.5 kB  00:00:00
Not using downloaded repomd.xml because it is older than what we have:
  Current   : Sun Jul 16 12:16:47 2017
  Downloaded: Mon Apr 10 14:10:29 2017
treasuredata/7Server/x86_64                                                                                               | 2.9 kB  00:00:00

Dependencies Resolved

=================================================================================================================================================
 Package                             Arch                         Version                            Repository                             Size
=================================================================================================================================================
Removing:
 ambari-server                       x86_64                       2.5.0.3-7                          @ambari-2.5.0.3                       722 M

Transaction Summary
=================================================================================================================================================
Remove  1 Package

Installed size: 722 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

ambari-server   0:off   1:off   2:on    3:on    4:on    5:on    6:off
  Erasing    : ambari-server-2.5.0.3-7.x86_64                                                                                                1/1
warning: file /usr/lib/python2.6/site-packages/ambari_server/utils.pyo: remove failed: No such file or directory
...
...

  Verifying  : ambari-server-2.5.0.3-7.x86_64                                                                                                1/1

Removed:
  ambari-server.x86_64 0:2.5.0.3-7

Complete!


[root@dn04 ~]# rm -rf /var/lib/ambari-server
[root@dn04 ~]# rm -rf /var/run/ambari-server
[root@dn04 ~]# rm -rf /usr/lib/ambari-server
[root@dn04 ~]# rm -rf /etc/ambari-server
[root@dn04 ~]# rm -rf /var/log/ambari-server
[root@dn04 ~]# rm -rf /usr/lib/python2.6/site-packages/ambari*



On each worker node:

[root@dn04 ~]# ambari-agent stop
[root@dn04 ~]# yum erase ambari-agent
[root@dn04 ~]# rm -rf /var/lib/ambari-agent
[root@dn04 ~]# rm -rf /var/run/ambari-agent
[root@dn04 ~]# rm -rf /usr/lib/amrbari-agent
[root@dn04 ~]# rm -rf /etc/ambari-agent
[root@dn04 ~]# rm -rf /var/log/ambari-agent

rm -rf /var/lib/ambari-agent
rm -rf /var/run/ambari-agent
rm -rf /usr/lib/amrbari-agent
rm -rf /etc/ambari-agent
rm -rf /var/log/ambari-agent

Reset Ambari Agent
[root@dn03 ~]# ambari-agent stop
[root@dn03 ~]# ambari-agent reset nn01
[root@dn03 ~]# ambari-agent start



Binaries/Config Location - HDP installation

Binaries Information
[root@dn04 ~]# ls -l /usr/hdp/current
[root@dn04 ~]# ls -l /usr/hdp

Configs for each component
[root@dn04 ~]# ls -l /etc

No comments: