Issue: File could only be replicated to 0 nodes instead of 1

joyolkreg · **註冊時間:** 2013-10-26, 21:21 **文章:** 2

Dear all,
I had set up three virtual machines which one for master and two for slaves. After configuring the hadoop located at ~/hadoop in master, i copied all the files to slaves using scp -r ~/hadoop hduser@slave.0:~/ and scp -r ~/hadoop hduser@slave.1:~/. The hadoop in master started normally using ~/hadoop/sbin/start-dfs.sh and ~/hadoop/sbin/start-yarn.sh. But i put any file into the hadoop in master such as ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/, and the following exceptions were shown. I have tried all the solutions here and the StackOverflow for three days, but they doesn't work for me. Any advice will be appreciated! Thanks in advance.

Error messages after ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/

引言回覆:

13/10/26 21:45:35 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
put: File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
13/10/26 21:45:35 ERROR hdfs.DFSClient: Failed to close file /input/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)

cat /etc/hosts in master and slaves

代碼:

0.0.1       localhost
168.0.10    master
168.0.20    slave.0
168.0.21    slave.1

jps information after start-dfs.sh and start-yarn.sh in master and slaves.

代碼:

hduser@master:~$ jps
NodeManager
Jps
ResourceManager
NameNode
DataNode
SecondaryNameNode

hduser@slave.0:~$ jps
NodeManager
DataNode
Jps

hduser@slave.1:~$ jps
Jps
NodeManager
DataNode

cat ~/hadoop/etc/hadoop/hdfs-site.xml in master and slaves.

代碼:

<configuration>
  <property><name>dfs.namenode.secondary.http-address</name><value>master:9001</value></property>
  <property><name>dfs.namenode.name.dir</name><value>file:/home/hduser/dfs/name</value></property>
  <property><name>dfs.datanode.data.dir</name><value>file:/home/hduser/dfs/data</value></property>
  <property><name>dfs.replication</name><value>1</value></property>
  <property><name>dfs.webhdfs.enabled</name><value>true</value></property>
  <property><name>dfs.http.address</name><value>master:50070</value></property>
</configuration>

cat ~/hadoop/etc/hadoop/mapred-site.xml in master and slaves.

代碼:

<configuration>
  <property><name>mapreduce.framework.name</name><value>yarn</value></property>
  <property><name>mapreduce.jobhistory.address</name><value>master:10020</value></property>
  <property><name>mapreduce.jobhistory.webapp.address</name><value>master:19888</value></property>
</configuration>

cat ~/hadoop/etc/hadoop/core-site.xml in master and slaves.

代碼:

<configuration>
  <property><name>fs.defaultFS</name><value>hdfs://master:9000</value></property>
  <property><name>hadoop.tmp.dir</name><value>/home/hduser/tmp</value></property>
</configuration>

cat ~/hadoop/etc/hadoop/yarn-site.xml in master and slaves.

代碼:

<configuration>
  <property><name>yarn.resourcemanager.address</name><value>master:8032</value></property>
  <property><name>yarn.resourcemanager.scheduler.address</name><value>master:8030</value></property>
  <property><name>yarn.resourcemanager.resource-tracker.address</name><value>master:8031</value></property>
  <property><name>yarn.resourcemanager.admin.address</name><value>master:8033</value></property>
  <property><name>yarn.resourcemanager.webapp.address</name><value>master:8088</value></property>
</configuration>

cat ~/hadoop/etc/hadoop/slaves in master and slaves.

代碼:

master
slave.0
slave.1

jazz · **註冊時間:** 2009-11-09, 19:52 **文章:** 2897

joyolkreg 寫:

Dear all,
I had set up three virtual machines which one for master and two for slaves. After configuring the hadoop located at ~/hadoop in master, i copied all the files to slaves using scp -r ~/hadoop hduser@slave.0:~/ and scp -r ~/hadoop hduser@slave.1:~/. The hadoop in master started normally using ~/hadoop/sbin/start-dfs.sh and ~/hadoop/sbin/start-yarn.sh. But i put any file into the hadoop in master such as ~/hadoop/bin/hadoop fs -put ~/hadoop/etc/hadoop/core-site.xml /input/, and the following exceptions were shown. I have tried all the solutions here and the StackOverflow for three days, but they doesn't work for me. Any advice will be appreciated! Thanks in advance.

According to the error message

代碼:

13/10/26 21:45:35 ERROR hdfs.DFSClient: Failed to close file /input/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.

It do know there are 3 datanodes.

So.... here are my two guesses:
(1) The NameNode is in Safe Mode
(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?

In NameNode log, it shows "Not able to place enough replicas, still in need of 1 to reach 1".

代碼:

2013-10-26 21:45:36,927 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on org.apache.commons.logging.impl.Log4JLogger
2013-10-26 21:45:36,928 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hduser (auth:SIMPLE) cause:java.io.IOException: File /input/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and no node(s) are excluded in this operation.

- Jazz

joyolkreg · **註冊時間:** 2013-10-26, 21:21 **文章:** 2

I am grateful for your help, and I did some tests accroding to your guesses.

(1) The NameNode is in Safe Mode
Here is the report of dfsadmin, but i think the NameNode is not in Safe Mode. Otherwise, the message "Safe Mode is ON" will locate at the top of the report if the NameNode is in Safe Mode.

代碼:

bin/hdfs dfsadmin -report
Configured Capacity: 5913821184 (5.51 GB)
Present Capacity: 831266816 (792.76 MB)
DFS Remaining: 831193088 (792.69 MB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Live datanodes:
Name: 192.168.0.10:50010 (master)
Hostname: master
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1623498752 (1.51 GB)
DFS Remaining: 347750400 (331.64 MB)
DFS Used%: 0.00%
DFS Remaining%: 17.64%
Last contact: Sun Oct 27 07:29:51 CST 2013


Name: 192.168.0.20:50010 (slave.0)
Hostname: slave.0
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1729728512 (1.61 GB)
DFS Remaining: 241520640 (230.33 MB)
DFS Used%: 0.00%
DFS Remaining%: 12.25%
Last contact: Sun Oct 27 07:29:51 CST 2013


Name: 192.168.0.21:50010 (slave.1)
Hostname: slave.1
Decommission Status : Normal
Configured Capacity: 1971273728 (1.84 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 1729327104 (1.61 GB)
DFS Remaining: 241922048 (230.71 MB)
DFS Used%: 0.00%
DFS Remaining%: 12.27%
Last contact: Sun Oct 27 07:29:50 CST 2013

(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?
The OS installed is Debian 7, its default policy in a default installation in iptables is to ACCEPT all traffic as following. There are no rules on any chain. So after testing, I know that it's able to ping vm guests from another one. Each vm guest connect to host via bridge adapter with allowing all frames through. Here is the iptables chain rules. The topology of my network is appended.

代碼:

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Talking to "Not able to place enough replicas, still in need of 1 to reach 1", I set dfs.replication to 3 at the first time. But the same error occured. Now I set dfs.replication to 1, the error still existed. I have no idea what's wrong with it. Still need help...

Henry, Joyolkreg

jazz 寫:

It do know there are 3 datanodes.

So.... here are my two guesses:
(1) The NameNode is in Safe Mode
(2) There are some network issue between VMs. Which mode of NIC setup in your VMs? NAT? HostOnly? Bridge?

In NameNode log, it shows "Not able to place enough replicas, still in need of 1 to reach 1".

- Jazz

Taiwan Hadoop Forum

Issue: File could only be replicated to 0 nodes instead of 1

誰在線上