Taiwan Hadoop Forum

台灣 Hadoop 技術討論區
現在的時間是 2020-08-08, 22:07

所有顯示的時間為 UTC + 8 小時




發表新文章 回覆主題  [ 2 篇文章 ] 
發表人 內容
 文章主題 : HBase 的 ImportFromFile Failed
文章發表於 : 2014-03-07, 18:25 
離線

註冊時間: 2014-03-07, 16:40
文章: 1
大家好,最近在學HBase,用HBase技術手冊的ImportFromFile.java,遇到了TableOutputFormat找不到的錯誤,連結是java原始碼,再下面是部分的紀錄。
https://github.com/larsgeorge/hbase-book/blob/master/ch07/src/main/java/mapreduce/ImportFromFile.java

我的環境是
Hadoop 2.2.0
HBase 0.96
Zookeeper 3.4.5

先確認'testtable'是否存在
代碼:
hbase(main):001:0> scan 'testtable'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
ROW                                                        COLUMN+CELL                                                                                                                                                             
r1                                                        column=colfam1:q1, timestamp=1393400146053, value=0.01                                                                                                                   
r2                                                        column=colfam1:q2, timestamp=1393400168983, value=0.02                                                                                                                   
r3                                                        column=colfam1:q3, timestamp=1393400181278, value=0.03                                                                                                                   
r3                                                        column=colfam1:q4, timestamp=1393400241351, value=0.04                                                                                                                   
3 row(s) in 0.1440 seconds


並且確認test-data.txt在HDFS
代碼:
[w514@w514001 importfromfile]$ hadoop fs -ls .
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Found 3 items
drwxrwx---   - w514 hadoop          0 2014-03-03 02:00 .Trash
drwx------   - w514 hadoop          0 2014-03-07 18:20 .staging
-rw-r--r--   3 w514 hadoop    1583565 2014-03-07 17:28 test-data.txt


再執行jar檔
代碼:
[w514@w514001 importfromfile]$ hadoop jar mapreduce.jar mapreduce.ImportFromFile -i test-data.txt -c fal:col -t testtable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/03/07 13:57:55 INFO client.RMProxy: Connecting to ResourceManager at w514002/192.168.0.202:8050
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
14/03/07 13:57:56 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/07 13:57:56 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/07 13:57:56 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-101--1, built on 01/09/2014 05:37 GMT
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:host.name=w514001
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_31
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.6.0_31/jre
14/03/07 13:57:56 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/hadoop/etc/hadoop:/usr/lib/hbase/conf:/usr/lib/zookeeper/conf:/etc/tez/conf:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:
......
/usr/lib/hbase/lib/hbase-server-0.96.1.2.0.6.1-101-hadoop2.jar    (很多JAR檔的路徑)
........
14/03/07 13:58:16 INFO mapreduce.Job: Job job_1394157070271_0005 failed with state FAILED due to: Application application_1394157070271_0005 failed 2 times due to AM Container for appattempt_1394157070271_0005_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
   at org.apache.hadoop.util.Shell.run(Shell.java:379)
   at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
   at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
   at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
   at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)


.Failing this attempt.. Failing the application.
14/03/07 13:58:16 INFO mapreduce.Job: Counters: 0


接著到Resource Manager找log看錯誤訊息
代碼:
2014-03-07 17:30:39,975 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1394182587525_0001_000001
2014-03-07 17:30:40,308 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-03-07 17:30:40,326 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-03-07 17:30:40,554 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2014-03-07 17:30:40,554 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@7cbe41ec)
2014-03-07 17:30:40,624 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts: 2 for application: 1. Attempt num: 1 is last retry: false
2014-03-07 17:30:40,650 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2014-03-07 17:30:40,878 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-03-07 17:30:40,906 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-03-07 17:30:41,494 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2014-03-07 17:30:41,617 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:467)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:368)
   at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1444)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
   at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:463)
   ... 8 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
   ... 10 more
2014-03-07 17:30:41,620 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:467)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:368)
   at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1444)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
   at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
   at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:463)
   ... 8 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
   ... 10 more
2014-03-07 17:30:41,628 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.


訊息有提到
Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
但是我找到了TableOutputFormat
代碼:
[w514@w514001 importfromfile]$ test -e /usr/lib/hbase/lib/hbase-server-0.96.1.2.0.6.1-101-hadoop2.jar && echo "exist" || echo "Not exist"
exist
[w514@w514001 importfromfile]$ cp /usr/lib/hbase/lib/hbase-server-0.96.1.2.0.6.1-101-hadoop2.jar .
[w514@w514001 importfromfile]$ jar -xf hbase-server-0.96.1.2.0.6.1-101-hadoop2.jar
[w514@w514001 importfromfile]$ ls org/apache/hadoop/hbase/mapreduce/ | grep 'TableOutputFormat'
MultiTableOutputFormat.class
MultiTableOutputFormat$MultiTableRecordWriter.class
TableOutputFormat.class
TableOutputFormat$TableRecordWriter.class
[w514@w514001 importfromfile]$


明明路徑都有參考到需要的JAR檔,但是卻出現找不到TableOutputFormat的錯誤
麻煩大家幫我看看,謝謝!


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: HBase 的 ImportFromFile Failed
文章發表於 : 2014-03-09, 23:39 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
不是很確定,還沒真正在 2.x 架構上試過 YARN + HBase ....
但或許您可以試著將 hbase 的 jar 檔複製到 Hadoop 的 lib 底下,並重新啟動相關 daemon (Ex. Resource Manager)

- Jazz


回頂端
 個人資料 E-mail  
 
顯示文章 :  排序  
發表新文章 回覆主題  [ 2 篇文章 ] 

所有顯示的時間為 UTC + 8 小時


誰在線上

正在瀏覽這個版面的使用者:沒有註冊會員 和 1 位訪客


不能 在這個版面發表主題
不能 在這個版面回覆主題
不能 在這個版面編輯您的文章
不能 在這個版面刪除您的文章
不能 在這個版面上傳附加檔案

搜尋:
前往 :  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
正體中文語系由 竹貓星球 維護製作