Taiwan Hadoop Forum

台灣 Hadoop 技術討論區
現在的時間是 2022-08-12, 19:37

所有顯示的時間為 UTC + 8 小時




發表新文章 回覆主題  [ 4 篇文章 ] 
發表人 內容
 文章主題 : Virtualbox架設叢集的問題
文章發表於 : 2013-11-06, 20:23 
離線

註冊時間: 2013-06-03, 12:40
文章: 15
想請問Jazz大,及有用 Virtualbox 架設過叢集的大哥們
我在用Virtualbox架設三台叢集時,我的Node一直起不來。設定應該確定沒錯了,但是一開始,假如是馬上 ./stop-all.sh至少還可以看得出來tasktracker都有起來,但是datanode都沒起來。再隔一段時間測試時,發現 ./start-all.sh後等待一段時間再 ./stop-all.sh,不但tasktracker都沒起來,是連datanode都根本沒有起來!! 就連我在node1上看50030也是,Nodes數都只有1
同時也發現,假如一開始在node1 ./start-all.sh跑完後,馬上觀察node2jps(指令)會有TaskTracker,但是node3都不會出現TaskTracker進程!!
(如下有截取片段測試結果)及(各個該設定的設定檔設定)
所以,想請教幾個問題:

1../start-all.sh 的執行結果後的角度上,應該是設定沒錯,那為什麼節點會起不來?
2. 如前言所提及,不但tasktracker都沒起來,是連datanode都根本沒有起來!! 想請問為什麼datanode在不論重複多少次測試都一樣起不來?
3. 是不是用Virtualbox架設叢集,難免或是一定會碰到Node起不來的問題?
4. 有沒有什麼好建議假如要練習自行搭建叢集,但node不怕起不來的方式? 或是有什麼可練習自行搭建叢集的軟體? 我爬過文章後,多數都是Virtualbox測試居多的說!?
5. 請問在Virtualbox架設叢集的操作設定檔設定,跟在一般真正實體電腦的設定檔設定,是一樣的嗎?

# vim /opt/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_25
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=/opt/hadoop/conf

# vim /opt/hadoop/conf/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop/hadoop-${user.name}</value>
</property>
</configuration>

# vim /opt/hadoop/conf/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

# vi m /opt/hadoop/conf/mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>node1:9001</value>
</property>
</configuration>

# vim /opt/hadoop/conf/masters
node1

# vim /opt/hadoop/conf/slaves
node1
node2
node3


[root@node1 bin]# ./start-all.sh
starting namenode, logging to /opt/hadoop/logs/hadoop-root-namenode-node1.out
node3: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node3.out
node2: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node2.out
node1: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node1.out
node1: starting secondarynamenode, logging to /opt/hadoop/logs/hadoop-root-secondarynamenode-node1.out
starting jobtracker, logging to /opt/hadoop/logs/hadoop-root-jobtracker-node1.out
node3: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node3.out
node2: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node2.out
node1: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node1.out
[root@node1 bin]#
[root@node1 bin]# ./stop-all.sh
stopping jobtracker
node3: stopping tasktracker
node2: stopping tasktracker
node1: stopping tasktracker
stopping namenode
node3: no datanode to stop
node2: no datanode to stop
node1: stopping datanode
node1: stopping secondarynamenode

第二次測試(隔一段時間後)
[root@node1 bin]# ./start-all.sh
starting namenode, logging to /opt/hadoop/logs/hadoop-root-namenode-node1.out
node2: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node2.out
node3: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node3.out
node1: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-node1.out
node1: starting secondarynamenode, logging to /opt/hadoop/logs/hadoop-root-secondarynamenode-node1.out
starting jobtracker, logging to /opt/hadoop/logs/hadoop-root-jobtracker-node1.out
node3: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node3.out
node2: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node2.out
node1: starting tasktracker, logging to /opt/hadoop/logs/hadoop-root-tasktracker-node1.out
[root@node1 bin]#
[root@node1 bin]# ./stop-all.sh
stopping jobtracker
node3: no tasktracker to stop
node2: no tasktracker to stop
node1: stopping tasktracker
stopping namenode
node3: no datanode to stop
node2: no datanode to stop
node1: stopping datanode
node1: stopping secondarynamenode

感恩


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: Virtualbox架設叢集的問題
文章發表於 : 2013-11-07, 02:03 
離線

註冊時間: 2013-06-03, 12:40
文章: 15
hadoop-root-datanode-node2.loghadoop-root-datanode-node3.logError 一樣
2013-11-07 01:01:37,716 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = node3/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branche
s/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2013-11-07 01:01:41,757 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 0 time(s).
2013-11-07 01:01:42,760 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 1 time(s).
2013-11-07 01:01:43,763 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 2 time(s).
2013-11-07 01:01:44,765 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 3 time(s).
2013-11-07 01:01:45,767 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 4 time(s).
2013-11-07 01:01:49,769 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 5 time(s).
2013-11-07 01:01:50,771 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 6 time(s).
2013-11-07 01:01:51,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 7 time(s).
2013-11-07 01:01:52,777 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 8 time(s).
2013-11-07 01:01:53,781 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node1/192.168.10.1:9000. Already tried 9 time(s).
2013-11-07 01:01:53,786 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Call to node1/192.168.10.1:9000 failed on local exception: java.net.NoRouteToHostException: No route to host

at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
... 13 more
2013-11-07 01:01:53,789 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at node3/127.0.0.1
************************************************************/


hadoop-root-tasktracker-node2.loghadoop-root-tasktracker-node3.logError 也一樣
2013-11-07 01:01:37,643 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = node3/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop ... /branches/
branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2013-11-07 01:01:39,697 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.lang.RuntimeException: Not a host:port pair: local

at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:136)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:123)
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:1807)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:905)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
2013-11-07 01:01:39,699 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at node3/127.0.0.1
************************************************************/

不知道能不能告訴小的No route to hostNot a host:port pair: local...各是在講什麼錯誤的說?


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: Virtualbox架設叢集的問題
文章發表於 : 2013-11-08, 11:03 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
> 1. 從 ./start-all.sh 的執行結果後的角度上,
> 應該是設定沒錯,那為什麼節點會起不來?

從討論看起來,SSH 會通,但節點起不來。而 Log 顯示是網路問題(No route...)。首先得確定您的網路配置方式是 Bridge, NAT 還是 Host-Only。您是否有自行做 22 port 的 port forward ?

> 2. 如前言所提及,不但tasktracker都沒起來,
> 是連datanode都根本沒有起來!!
> 想請問為什麼datanode在不論重複多少次測試都一樣起不來?

先把網路問題確認後再說吧。

> 3. 是不是用Virtualbox架設叢集,『難免』或是『一定會』碰到『Node起不來』的問題?

問題不能這樣推論,並不是工具不好,而是「易用性」確實還沒有那麼好。
不管用 VirtualBox 或 VMWare, Hyper-V 我想首先開發者必須有基本的網路觀念,因為 Bridge, NAT, Host-Only 三種模式常讓初學者搞不清楚狀況。簡單來說 Bridge 就是您的 VM 會跟實體主機接在同一個實體網路孔上,若三台都是 Bridge,那就得要有路由(router)有 Gateway 幫忙轉才行。若是 NAT,就視為三台 VM 是接在家用 IP 分享器。若是 Host-Only 就像接在一台有 DHCP 的 Switch 上,出不去。

> 4. 有沒有什麼好建議假如要練習自行搭建叢集,
> 但node不怕起不來的方式? 或是有什麼可練習自行
> 搭建叢集的軟體? 我爬過文章後,
> 多數都是Virtualbox測試居多的說!?

還是回歸到叢集環境本來所需的背景知識就比較多,這也是我們在推動社群時,一直希望以降低這個門檻為首務。

> 5. 請問在Virtualbox架設叢集的操作設定檔設定,
> 跟在一般真正實體電腦的設定檔設定,是一樣的嗎?

一樣的。

- Jazz


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: Virtualbox架設叢集的問題
文章發表於 : 2013-11-12, 22:20 
離線

註冊時間: 2013-11-12, 18:38
文章: 3
你虚拟机的网卡MAC地址是不是冲突,网络配置不对


回頂端
 個人資料 E-mail  
 
顯示文章 :  排序  
發表新文章 回覆主題  [ 4 篇文章 ] 

所有顯示的時間為 UTC + 8 小時


誰在線上

正在瀏覽這個版面的使用者:沒有註冊會員 和 2 位訪客


不能 在這個版面發表主題
不能 在這個版面回覆主題
不能 在這個版面編輯您的文章
不能 在這個版面刪除您的文章
不能 在這個版面上傳附加檔案

搜尋:
前往 :  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
正體中文語系由 竹貓星球 維護製作