Taiwan Hadoop Forum

台灣 Hadoop 技術討論區
現在的時間是 2022-06-29, 02:31

所有顯示的時間為 UTC + 8 小時




發表新文章 回覆主題  [ 3 篇文章 ] 
發表人 內容
 文章主題 : 抓取Hbase中的資料
文章發表於 : 2013-09-12, 22:47 
離線

註冊時間: 2013-09-10, 13:25
文章: 6
此程式為參考書籍隨附程式
附加檔案:
maxCPU.zip [6.87 KiB]
被下載 330 次

Hadoop版本為0.20.2
Hbase版本為0.90.2
Eclipse版本為3.6.2
此程式是將本地端的檔案上傳到HDFS後
再將maper處理後資料填入Hbase中
reducer處理後再將資料存入HDFS中
此部分是利用eclipse加 hadoop-plugin去執行
job沒有執行完成有輸出一些錯誤訊息
搜尋過後沒有類似的文章
想請教各位是否有哪邊設定有錯
錯誤訊息如下
代碼:
13/09/12 22:25:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:host.name=hdp0
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/jdk1.6.0_32/jre
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadoop/workspace/.metadata/.plugins/org.apache.hadoop.eclipse/hadoop-conf-9010345737760457656:/home/hadoop/workspace/HadoopLab/bin:/home/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar:/home/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar:/home/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.14.jar:/home/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar:/home/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar:/home/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar:/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar:/home/hadoop/hadoop-0.20.2/lib/jetty-6.1.14.jar:/home/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar:/home/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar:/home/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar:/home/hadoop/hadoop-0.20.2/lib/junit-3.8.1.jar:/home/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar:/home/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar:/home/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar:/home/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar:/home/hadoop/hadoop-0.20.2/lib/commons-codec-1.3.jar:/home/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.0.jar:/home/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar:/home/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar:/home/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-0.20.2/hadoop-0.20.2-ant.jar:/home/hadoop/hadoop-0.20.2/hadoop-0.20.2-core.jar:/home/hadoop/hadoop-0.20.2/hadoop-0.20.2-tools.jar:/home/hadoop/hbase-0.90.2/hbase-0.90.2.jar:/home/hadoop/hbase-0.90.2/hbase-0.90.2-tests.jar:/home/hadoop/hbase-0.90.2/lib/zookeeper-3.3.2.jar:/home/hadoop/hbase-0.90.2/lib/activation-1.1.jar:/home/hadoop/hbase-0.90.2/lib/asm-3.1.jar:/home/hadoop/hbase-0.90.2/lib/avro-1.3.3.jar:/home/hadoop/hbase-0.90.2/lib/commons-lang-2.5.jar:/home/hadoop/hbase-0.90.2/lib/guava-r06.jar:/home/hadoop/hbase-0.90.2/lib/jackson-core-asl-1.5.5.jar:/home/hadoop/hbase-0.90.2/lib/jackson-jaxrs-1.5.5.jar:/home/hadoop/hbase-0.90.2/lib/jackson-mapper-asl-1.4.2.jar:/home/hadoop/hbase-0.90.2/lib/jackson-xc-1.5.5.jar:/home/hadoop/hbase-0.90.2/lib/jaxb-api-2.1.jar:/home/hadoop/hbase-0.90.2/lib/jaxb-impl-2.1.12.jar:/home/hadoop/hbase-0.90.2/lib/jersey-core-1.4.jar:/home/hadoop/hbase-0.90.2/lib/jersey-json-1.4.jar:/home/hadoop/hbase-0.90.2/lib/jersey-server-1.4.jar:/home/hadoop/hbase-0.90.2/lib/jettison-1.1.jar:/home/hadoop/hbase-0.90.2/lib/jruby-complete-1.6.0.jar:/home/hadoop/hbase-0.90.2/lib/jsp-2.1-6.1.14.jar:/home/hadoop/hbase-0.90.2/lib/jsp-api-2.1-6.1.14.jar:/home/hadoop/hbase-0.90.2/lib/jsr311-api-1.1.1.jar:/home/hadoop/hbase-0.90.2/lib/protobuf-java-2.3.0.jar:/home/hadoop/hbase-0.90.2/lib/stax-api-1.0.1.jar:/home/hadoop/hbase-0.90.2/lib/thrift-0.2.0.jar:/home/hadoop/hadoop-0.20.2/hadoop-0.20.2-eclipse-plugin.jar:/home/hadoop/hbase-0.90.2/lib/commons-cli-1.2.jar:/home/hadoop/hbase-0.90.2/lib/commons-codec-1.4.jar:/home/hadoop/hbase-0.90.2/lib/commons-el-1.0.jar:/home/hadoop/hbase-0.90.2/lib/commons-httpclient-3.1.jar:/home/hadoop/hbase-0.90.2/lib/commons-logging-1.1.1.jar:/home/hadoop/hbase-0.90.2/lib/commons-net-1.4.1.jar:/home/hadoop/hbase-0.90.2/lib/core-3.1.1.jar:/home/hadoop/hbase-0.90.2/lib/hadoop-0.20.2-core.jar:/home/hadoop/hbase-0.90.2/lib/hbase-0.90.2.jar:/home/hadoop/hbase-0.90.2/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hbase-0.90.2/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hbase-0.90.2/lib/jetty-6.1.26.jar:/home/hadoop/hbase-0.90.2/lib/jetty-util-6.1.26.jar:/home/hadoop/hbase-0.90.2/lib/log4j-1.2.16.jar:/home/hadoop/hbase-0.90.2/lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hbase-0.90.2/lib/slf4j-api-1.5.8.jar:/home/hadoop/hbase-0.90.2/lib/slf4j-log4j12-1.5.8.jar:/home/hadoop/hbase-0.90.2/lib/xmlenc-0.52.jar
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/jdk1.6.0_32/jre/lib/i386/client:/usr/lib/jvm/jdk1.6.0_32/jre/lib/i386:/usr/lib/jvm/jdk1.6.0_32/jre/../lib/i386:/usr/lib/jvm/jdk1.6.0_32/jre/lib/i386/client:/usr/lib/jvm/jdk1.6.0_32/jre/lib/i386::/usr/java/packages/lib/i386:/lib:/usr/lib
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-33-generic
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop/workspace/HadoopLab
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdp2:2222,hdp1:2222,hdp0:2222 sessionTimeout=180000 watcher=hconnection
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Opening socket connection to server hdp2/192.168.0.102:2222
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Socket connection established to hdp2/192.168.0.102:2222, initiating session
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Session establishment complete on server hdp2/192.168.0.102:2222, sessionid = 0x2411286cc460001, negotiated timeout = 180000
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdp2:2222,hdp1:2222,hdp0:2222 sessionTimeout=180000 watcher=hconnection
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Opening socket connection to server hdp0/192.168.0.100:2222
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Socket connection established to hdp0/192.168.0.100:2222, initiating session
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: Session establishment complete on server hdp0/192.168.0.100:2222, sessionid = 0x41128c6aaf0002, negotiated timeout = 180000
13/09/12 22:25:14 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x41128c6aaf0002
13/09/12 22:25:14 INFO zookeeper.ClientCnxn: EventThread shut down
13/09/12 22:25:14 INFO zookeeper.ZooKeeper: Session: 0x41128c6aaf0002 closed
delete the table
13/09/12 22:25:14 INFO client.HBaseAdmin: Started disable of CPU
13/09/12 22:25:16 INFO client.HBaseAdmin: Disabled CPU
13/09/12 22:25:17 INFO client.HBaseAdmin: Deleted CPU
13/09/12 22:25:18 INFO input.FileInputFormat: Total input paths to process : 1
13/09/12 22:25:18 INFO mapred.JobClient: Running job: job_201309122212_0001
13/09/12 22:25:19 INFO mapred.JobClient:  map 0% reduce 0%
13/09/12 22:25:31 INFO mapred.JobClient: Task Id : attempt_201309122212_0001_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   at MR_Lab2.AddData.<clinit>(AddData.java:12)
   at MR_Lab2.mymapper.map(mymapper.java:20)
   at MR_Lab2.mymapper.map(mymapper.java:1)
   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)

13/09/12 22:25:40 INFO mapred.JobClient: Task Id : attempt_201309122212_0001_m_000000_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
   at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
   at MR_Lab2.AddData.<clinit>(AddData.java:12)
   at MR_Lab2.mymapper.map(mymapper.java:20)
   at MR_Lab2.mymapper.map(mymapper.java:1)
   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)

13/09/12 22:25:49 INFO mapred.JobClient: Task Id : attempt_201309122212_0001_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
   at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
   at MR_Lab2.AddData.<clinit>(AddData.java:12)
   at MR_Lab2.mymapper.map(mymapper.java:20)
   at MR_Lab2.mymapper.map(mymapper.java:1)
   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)

13/09/12 22:25:58 INFO mapred.JobClient: Job complete: job_201309122212_0001
13/09/12 22:25:58 INFO mapred.JobClient: Counters: 4
13/09/12 22:25:58 INFO mapred.JobClient:   Job Counters
13/09/12 22:25:58 INFO mapred.JobClient:     Rack-local map tasks=1
13/09/12 22:25:58 INFO mapred.JobClient:     Launched map tasks=4
13/09/12 22:25:58 INFO mapred.JobClient:     Data-local map tasks=3
13/09/12 22:25:58 INFO mapred.JobClient:     Failed map tasks=1
Not Complete!


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: 抓取Hbase中的資料
文章發表於 : 2013-09-13, 01:15 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
請問您有把 hbase 的 jar 檔放進 hadoop 的 lib 目錄嘛?
若有,放進去以後,有重開 JobTracker 跟 TaskTracker 嘛?

- Jazz


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: 抓取Hbase中的資料
文章發表於 : 2013-09-13, 09:54 
離線

註冊時間: 2013-09-10, 13:25
文章: 6
原先是使用external jar的方式直接加到map/reduce project中
後來直接將hbase中的jar複製到hadoop/lib目錄中
hadoop,hbase,eclipse重開後已可正常執行
感謝jazz版主指點!!


回頂端
 個人資料 E-mail  
 
顯示文章 :  排序  
發表新文章 回覆主題  [ 3 篇文章 ] 

所有顯示的時間為 UTC + 8 小時


誰在線上

正在瀏覽這個版面的使用者:沒有註冊會員 和 4 位訪客


不能 在這個版面發表主題
不能 在這個版面回覆主題
不能 在這個版面編輯您的文章
不能 在這個版面刪除您的文章
不能 在這個版面上傳附加檔案

搜尋:
前往 :  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
正體中文語系由 竹貓星球 維護製作