Taiwan Hadoop Forum

台灣 Hadoop 技術討論區
現在的時間是 2018-12-13, 03:07

所有顯示的時間為 UTC + 8 小時




發表新文章 回覆主題  [ 3 篇文章 ] 
發表人 內容
 文章主題 : Issue: File could only be replicated to 0 nodes instead of 1
文章發表於 : 2013-10-26, 22:34 
離線

註冊時間: 2013-10-26, 21:21
文章: 2
Dear all,
I had set up three virtual machines which one for master and two for slaves. After configuring the hadoop located at ~/hadoop in master, i copied all the files to slaves using scp -r ~/hadoop hduser@slave.0:~/ and scp -r ~/hadoop hduser@slave.1:~/. The hadoop in master started normally using ~/hadoop/sbin/start-dfs.sh and ~/hadoop/sbin/start-yarn.sh. But i put any file into the hadoop in master such as ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/, and the following exceptions were shown. I have tried all the solutions here and the StackOverflow for three days, but they doesn't work for me. Any advice will be appreciated! Thanks in advance.

Error messages after ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/
引言回覆:
13/10/26 21:45:35 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
put: File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
13/10/26 21:45:35 ERROR hdfs.DFSClient: Failed to close file /input/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)


cat /etc/hosts in master and slaves
代碼:
127.0.0.1       localhost
192.168.0.10    master
192.168.0.20    slave.0
192.168.0.21    slave.1


jps information after start-dfs.sh and start-yarn.sh in master and slaves.
代碼:
hduser@master:~$ jps
15308 NodeManager
15343 Jps
15220 ResourceManager
14848 NameNode
14952 DataNode
15075 SecondaryNameNode

hduser@slave.0:~$ jps
5100 NodeManager
5012 DataNode
5203 Jps

hduser@slave.1:~$ jps
5231 Jps
5045 NodeManager
4957 DataNode


cat ~/hadoop/etc/hadoop/hdfs-site.xml in master and slaves.
代碼:
<configuration>
  <property><name>dfs.namenode.secondary.http-address</name><value>master:9001</value></property>
  <property><name>dfs.namenode.name.dir</name><value>file:/home/hduser/dfs/name</value></property>
  <property><name>dfs.datanode.data.dir</name><value>file:/home/hduser/dfs/data</value></property>
  <property><name>dfs.replication</name><value>1</value></property>
  <property><name>dfs.webhdfs.enabled</name><value>true</value></property>
  <property><name>dfs.http.address</name><value>master:50070</value></property>
</configuration>


cat ~/hadoop/etc/hadoop/mapred-site.xml in master and slaves.
代碼:
<configuration>
  <property><name>mapreduce.framework.name</name><value>yarn</value></property>
  <property><name>mapreduce.jobhistory.address</name><value>master:10020</value></property>
  <property><name>mapreduce.jobhistory.webapp.address</name><value>master:19888</value></property>
</configuration>


cat ~/hadoop/etc/hadoop/core-site.xml in master and slaves.
代碼:
<configuration>
  <property><name>fs.defaultFS</name><value>hdfs://master:9000</value></property>
  <property><name>hadoop.tmp.dir</name><value>/home/hduser/tmp</value></property>
</configuration>


cat ~/hadoop/etc/hadoop/yarn-site.xml in master and slaves.
代碼:
<configuration>
  <property><name>yarn.resourcemanager.address</name><value>master:8032</value></property>
  <property><name>yarn.resourcemanager.scheduler.address</name><value>master:8030</value></property>
  <property><name>yarn.resourcemanager.resource-tracker.address</name><value>master:8031</value></property>
  <property><name>yarn.resourcemanager.admin.address</name><value>master:8033</value></property>
  <property><name>yarn.resourcemanager.webapp.address</name><value>master:8088</value></property>
</configuration>


cat ~/hadoop/etc/hadoop/slaves in master and slaves.
代碼:
master
slave.0
slave.1


附加檔案:
檔案註釋: The logs in slave 1.
logs.slave.1.7z [4.6 KiB]
被下載 119 次
檔案註釋: The logs in slave 0.
logs.slave.0.7z [4.61 KiB]
被下載 114 次
檔案註釋: The logs in master.
logs.master.7z [9.24 KiB]
被下載 113 次
回頂端
 個人資料 E-mail  
 
 文章主題 : Re: Issue: File could only be replicated to 0 nodes instead
文章發表於 : 2013-10-27, 01:43 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
joyolkreg 寫:
Dear all,
I had set up three virtual machines which one for master and two for slaves. After configuring the hadoop located at ~/hadoop in master, i copied all the files to slaves using scp -r ~/hadoop hduser@slave.0:~/ and scp -r ~/hadoop hduser@slave.1:~/. The hadoop in master started normally using ~/hadoop/sbin/start-dfs.sh and ~/hadoop/sbin/start-yarn.sh. But i put any file into the hadoop in master such as ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/, and the following exceptions were shown. I have tried all the solutions here and the StackOverflow for three days, but they doesn't work for me. Any advice will be appreciated! Thanks in advance.


According to the error message
代碼:
13/10/26 21:45:35 ERROR hdfs.DFSClient: Failed to close file /input/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.


It do know there are 3 datanodes.

So.... here are my two guesses:
(1) The NameNode is in Safe Mode
(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?

In NameNode log, it shows "Not able to place enough replicas, still in need of 1 to reach 1".
代碼:
2013-10-26 21:45:36,927 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on org.apache.commons.logging.impl.Log4JLogger
2013-10-26 21:45:36,928 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hduser (auth:SIMPLE) cause:java.io.IOException: File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and no node(s) are excluded in this operation.


- Jazz


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: Issue: File could only be replicated to 0 nodes instead
文章發表於 : 2013-10-27, 07:46 
離線

註冊時間: 2013-10-26, 21:21
文章: 2
I am grateful for your help, and I did some tests accroding to your guesses.

(1) The NameNode is in Safe Mode
Here is the report of dfsadmin, but i think the NameNode is not in Safe Mode. Otherwise, the message "Safe Mode is ON" will locate at the top of the report if the NameNode is in Safe Mode.
代碼:
bin/hdfs dfsadmin -report
Configured Capacity: 5913821184 (5.51 GB)
Present Capacity: 831266816 (792.76 MB)
DFS Remaining: 831193088 (792.69 MB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Live datanodes:
Name: 192.168.0.10:50010 (master)
Hostname: master
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1623498752 (1.51 GB)
DFS Remaining: 347750400 (331.64 MB)
DFS Used%: 0.00%
DFS Remaining%: 17.64%
Last contact: Sun Oct 27 07:29:51 CST 2013


Name: 192.168.0.20:50010 (slave.0)
Hostname: slave.0
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1729728512 (1.61 GB)
DFS Remaining: 241520640 (230.33 MB)
DFS Used%: 0.00%
DFS Remaining%: 12.25%
Last contact: Sun Oct 27 07:29:51 CST 2013


Name: 192.168.0.21:50010 (slave.1)
Hostname: slave.1
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1729327104 (1.61 GB)
DFS Remaining: 241922048 (230.71 MB)
DFS Used%: 0.00%
DFS Remaining%: 12.27%
Last contact: Sun Oct 27 07:29:50 CST 2013


(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?
The OS installed is Debian 7, its default policy in a default installation in iptables is to ACCEPT all traffic as following. There are no rules on any chain. So after testing, I know that it's able to ping vm guests from another one. Each vm guest connect to host via bridge adapter with allowing all frames through. Here is the iptables chain rules. The topology of my network is appended.
代碼:
iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


Talking to "Not able to place enough replicas, still in need of 1 to reach 1", I set dfs.replication to 3 at the first time. But the same error occured. Now I set dfs.replication to 1, the error still existed. I have no idea what's wrong with it. Still need help...

Henry, Joyolkreg

jazz 寫:
It do know there are 3 datanodes.

So.... here are my two guesses:
(1) The NameNode is in Safe Mode
(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?

In NameNode log, it shows "Not able to place enough replicas, still in need of 1 to reach 1".

- Jazz


附加檔案:
檔案註釋: The topology of my network
Network Topology.png
Network Topology.png [ 22.11 KiB | 被瀏覽 4197 次 ]
回頂端
 個人資料 E-mail  
 
顯示文章 :  排序  
發表新文章 回覆主題  [ 3 篇文章 ] 

所有顯示的時間為 UTC + 8 小時


誰在線上

正在瀏覽這個版面的使用者:沒有註冊會員 和 3 位訪客


不能 在這個版面發表主題
不能 在這個版面回覆主題
不能 在這個版面編輯您的文章
不能 在這個版面刪除您的文章
不能 在這個版面上傳附加檔案

搜尋:
前往 :  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
正體中文語系由 竹貓星球 維護製作