UBUNTU HADOOP 2.2.0 설치
우분투 13.10 에 하둡 2.2.0 설치 single-node setup
0. 하둡 다운로드 및 유저 설치 디렉토리로 이동 시키기
$ wget http://mirror.apache-kr.org/hadoop/common/stable/hadoop-2.2.0.tar.gz
다운로드 후
$ sudo tar xvzf hadoop-2.2.0.tar.gz && sudo cp -r hadoop-2.2.0 /usr/local/hadoop
1. JDK 7.0 설치
- hadoop 을 돌리기위하여 설치
$ sudo apt-get install openjdk-7-jdk
2. 계정생성
- 하둡 관련 그룹및 계정을 생성하여 관리의 용의성 등을 활용 하기 위함
- 그룹 : hadoop, 계정 : hadoop
$ sudo addgroup hadoop && sudo adduser --ingroup hadoop hadoop
hadoop 계정과 그룹을 등록하였으니 이제 압축해제 하여 옮겨둔 하둡 디렉토리의 권한 설정
$sudo chown -R hadoop:hadoop /usr/local/hadoop
3. ssh 설치
- hadoop ssh 접속관리를 위해 설치 후 재실행
$ sudo apt-get install openssh-server && sudo /etc/init.d/ssh restart
혹신 방화벽 문제가.. 엉뚱하게도 생기시는 분은
$ ufw allow 22/tcp
4. export 설정
- 접속시 마다 export해서 사용하기 불편함 해소
$ su - hadoop
$ vi ~/.profile
아래 내용 등록
# JDK 7.0
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
# hadoop home
export HADOOP_HOME=/usr/local/hadoop
# hadoop bin
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
저장후 아래 명령어를 이용하여 로드
$ source ~/.profile
확인
hadoop@sidcode-worlds:~$ echo $JAVA_HOME - $HADOOP_HOME - $PATH
/usr/lib/jvm/java-7-openjdk-amd64/jre - /usr/local/hadoop - /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/hadoop/bin
5. ssh 자동 로그인을 위하여 rsa 키 만들기
hadoop@sidcode-worlds:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): (엔터)
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
d6:d3:a1:2a:ae:d1:94:ad:00:f8:6e:a1:5e:f9:31:bb hadoop@sidcode-worlds
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . |
|. . . |
| . . o . o . |
| o . o S + . |
| o ..+ o . . |
|. oo.o+ . |
|... .o+. |
| . .Eo |
+-----------------+
생성후 인증키 복사 ~/.ssh/authorized_keys 가 존재 할지도 모르니 cat >> 이용
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 755 ~/.ssh/authorized_keys
접속 테스트
hadoop@sidcode-worlds:~$ ssh hadoop@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 키값(삭제함).
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
hadoop@sidcode-worlds:~$
6. hadoop 설정
$ vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh
아래 내용 추가 및 변경
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
# The jsvc implementation to use. Jsvc is required to run secure datanodes.
#export JSVC_HOME=${JSVC_HOME}
#export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
7. 하둡 tmp 만들기
- chmod 는 생각하시는 보안정책에 따라 변경 하세요. (core-site.xml, HDFS 기본 템프럴리 디렉토리) 위치는 원하시는곳에
mkdir -p $HADOOP_HOME/tmp && chmod 750 $HADOOP_HOME/tmp
8. *-site.xml 수정
core-site.xml 수정
$ vi /usr/local/hadoop/etc/hadoop/core-site.xml
아래내용 추가, 하둡코어 환경설정(HDFS, MapReDuce을 위한)포트는 원하는 범위를 잡으세요.
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:37050</value>
</property>
</configuration>
hdfs-site.xml수정
$ vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml
아래내용 추가, HDFS 데몬환경설정( 데이터 노드랑 네임노드 설정)
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.name.edits.dir</name>
<value>${dfs.name.dir}</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop/dfs/data</value>
</property>
mapred-site.xml 수정 (템플릿만 존재하므로 복사)
$ cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.tem* /usr/local/hadoop/etc/hadoop/mapred-site.xml
$ vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml
아래 추가, MapReduce데몬 환경설정)job tracker
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:37051</value>
</property>
9. namenode 포멧
* 꼭, 네임노드 포멧 실행하실경우 신중을 기하세요.
* 처음 설치시 실행해주세요. 왜냐면 네임노드 사용중인데 또 사용하시면 말그대로 포멧됨
$ /usr/local/hadoop/bin/hadoop namenode -format
10.실행, 확인 및 종료
* 실행
$ /usr/local/hadoop/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
13/11/15 00:15:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/11/15 00:15:21 WARN fs.FileSystem: "localhost:37050" is a deprecated filesystem name. Use "hdfs://localhost:37050/" instead.
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-sidcode-worlds.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-sidcode-worlds.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 키값(삭제함)
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-sidcode-worlds.out
13/11/15 00:15:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-sidcode-worlds.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-sidcode-worlds.out
hadoop@sidcode-worlds:/usr/local/hadoop/sbin$
*확인
hadoop@sidcode-worlds:/usr/local/hadoop/etc/hadoop$ jps
21433 ResourceManager
21566 NodeManager
21280 SecondaryNameNode
21882 Jps
20946 NameNode
21096 DataNod
웹 환경 에서 데몬 확인
- 네임노드
http://localhost:50070/
* 종료
$ /usr/local/hadoop/sbin/stop-all.sh