Install using yum:
# yum install hadoop-httpfs
Installed:
hadoop-httpfs.noarch 0:2.7.3.2.6.1.0-129
Dependency Installed:
hadoop_2_6_1_0_129-httpfs.x86_64 0:2.7.3.2.6.1.0-129 hadoop_2_6_1_0_129-httpfs-server.x86_64 0:2.7.3.2.6.1.0-129
Complete!
Add java path to httpfs config
# echo $JAVA_HOME
/mnt/vol1/jdk1.8.0_144
# vi /etc/hadoop-httpfs/conf/httpfs-env.sh
export JAVA_HOME=/mnt/vol1/jdk1.8.0_144
Add the restart script
# ln -s /usr/hdp/current/hadoop-httpfs/etc/rc.d/init.d/hadoop-httpfs /etc/init.d/hadoop-httpfs
Adding Hue properties to httpfs
# vi /etc/hadoop-httpfs/conf/httpfs-site.xml
<configuration>
<!-- HUE proxy user setting -->
<property>
<name>httpfs.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>httpfs.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>httpfs.hadoop.config.dir</name>
<value>/etc/hadoop/conf</value>
</property>
</configuration>
Starting service
# service hadoop-httpfs start
Starting Hadoop httpfs: [ OK ]
WARNING: current setting of HTTPFS_HOME ignored
Setting HTTPFS_HOME: /usr/hdp/2.6.1.0-129/hadoop-httpfs
Using HTTPFS_CONFIG: /etc/hadoop-httpfs/conf
Sourcing: /etc/hadoop-httpfs/conf/httpfs-env.sh
setting JAVA_HOME=/mnt/vol1/jdk1.8.0_144
Using HTTPFS_LOG: /var/log/hadoop-httpfs/
Using HTTPFS_TEMP: /var/run/hadoop-httpfs
Setting HTTPFS_HTTP_PORT: 14000
Setting HTTPFS_ADMIN_PORT: 14001
Setting HTTPFS_HTTP_HOSTNAME: dh-hw-m1d1.dbp.dailyhunt.in
Setting HTTPFS_SSL_ENABLED: false
Setting HTTPFS_SSL_KEYSTORE_FILE: /var/lib/hadoop-httpfs/.keystore
Setting HTTPFS_SSL_KEYSTORE_PASS: password
Setting HTTPFS_MAX_HTTP_HEADER_SIZE: 65536
Using CATALINA_BASE: /etc/hadoop-httpfs/tomcat-deployment
Setting HTTPFS_CATALINA_HOME: /etc/hadoop-httpfs/tomcat-deployment
Setting CATALINA_OUT: /var/log/hadoop-httpfs//httpfs-catalina.out
Using CATALINA_PID: /var/run/hadoop-httpfs/hadoop-httpfs-httpfs.pid
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dhttpfs.home.dir=/usr/hdp/2.6.1.0-129/hadoop-httpfs -Dhttpfs.config.dir=/etc/hadoop-httpfs/conf -Dhttpfs.log.dir=/var/log/hadoop-httpfs/ -Dhttpfs.temp.dir=/var/run/hadoop-httpfs -Dhttpfs.admin.port=14001 -Dhttpfs.http.port=14000 -Dhttpfs.http.hostname=dh-hw-m1d1.dbp.dailyhunt.in -Dhttpfs.ssl.enabled=false -Dhttpfs.ssl.keystore.file=/var/lib/hadoop-httpfs/.keystore -Dhttpfs.ssl.keystore.pass=password
Using CATALINA_BASE: /etc/hadoop-httpfs/tomcat-deployment
Using CATALINA_HOME: /etc/hadoop-httpfs/tomcat-deployment
Using CATALINA_TMPDIR: /var/run/hadoop-httpfs
Using JRE_HOME: /mnt/vol1/jdk1.8.0_144
Using CLASSPATH: /etc/hadoop-httpfs/tomcat-deployment/bin/bootstrap.jar
Using CATALINA_PID: /var/run/hadoop-httpfs/hadoop-httpfs-httpfs.pid
Check if up
# netstat -plant | grep 14000
tcp 0 0 0.0.0.0:14000 0.0.0.0:* LISTEN 352471/java
Adding it to auto start
# chkconfig hadoop-httpfs on
Example:
$ curl "http://192.168.3.164:14000/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs"
{"FileStatus":{"pathSuffix":"","type":"DIRECTORY","length":0,"owner":"hdfs","group":"hadoop","permission":"777","accessTime":0,"modificationTime":1557479480454,"blockSize":0,"replication":0}}
Core site modifications
Add the following to core-site.xml:
hadoop.proxyuser.httpfs.groups=*
hadoop.proxyuser.httpfs.hosts=*
0 Comments