Taiwan Hadoop Forum

台灣 Hadoop 技術討論區
現在的時間是 2022-07-02, 10:18

所有顯示的時間為 UTC + 8 小時




發表新文章 回覆主題  [ 6 篇文章 ] 
發表人 內容
 文章主題 : fairscheduler編譯失敗
文章發表於 : 2014-02-20, 17:49 
離線

註冊時間: 2014-02-20, 16:50
文章: 3
我試著修改Fairscheduler中的LoadManager 想增加ㄧ些負載平衡的機制

當修改完想進行測試的時候,編譯的部分一直出錯,爬了許多文,也沒辦法解決

所以想請問有人知道為甚麼嗎?

以下為編譯的過程

代碼:
hduser@master:/opt/hadoop$ ant compile-contrib
Buildfile: /opt/hadoop/build.xml

clover.setup:

clover.info:
     [echo]
     [echo]      Clover not found. Code coverage reports disabled.
     [echo]   

clover:

ivy-init-dirs:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

init:
    [touch] Creating /tmp/null1717781464
   [delete] Deleting: /tmp/null1717781464
     [copy] Copying 1 file to /opt/hadoop/conf
     [copy] Copying /opt/hadoop/conf/fair-scheduler.xml.template to /opt/hadoop/conf/fair-scheduler.xml
     [exec] src/saveVersion.sh: 1: src/saveVersion.sh: svn: not found
     [exec] src/saveVersion.sh: 1: src/saveVersion.sh: svn: not found

record-parser:

compile-rcc-compiler:
    [javac] /opt/hadoop/build.xml:473: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

compile-core-classes:
    [javac] /opt/hadoop/build.xml:497: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

compile-hdfs-classes:
    [javac] /opt/hadoop/build.xml:593: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 2 source files to /opt/hadoop/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6
    [javac] 1 warning

compile-mapred-classes:
Trying to override old definition of task jsp-compile
    [javac] /opt/hadoop/build.xml:550: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

create-native-configure:
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
     [exec] libtoolize: copying file `config/config.guess'
     [exec] libtoolize: copying file `config/config.sub'
     [exec] libtoolize: copying file `config/install-sh'
     [exec] libtoolize: copying file `config/ltmain.sh'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
     [exec] libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:310: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] Makefile.am:32: shell echo $$OS_NAME | tr [A-Z] [a-z]: non-POSIX variable name
     [exec] Makefile.am:32: (probably a GNU make extension)

compile-core-native:

create-c++-configure:

check-c++-makefiles:

create-c++-pipes-makefile:

create-c++-utils-makefile:

compile-c++-utils:

compile-c++-pipes:

compile-c++:

compile-core:

jar:
      [tar] Nothing to do: /opt/hadoop/build/classes/bin.tgz is up to date.
      [jar] Building jar: /opt/hadoop/build/hadoop-core-1.2.1.jar

compile-tools:
    [javac] /opt/hadoop/build.xml:614: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

tools-jar:

check-c++-libhdfs:

check-c++-makefile-libhdfs:

create-c++-libhdfs-makefile:

compile-c++-libhdfs:

compile-contrib:

compile:

check-contrib:

init:
     [echo] contrib: capacity-scheduler

init-contrib:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

compile:
     [echo] contrib: capacity-scheduler
    [javac] /opt/hadoop/src/contrib/build-contrib.xml:190: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

check-contrib:

init:
     [echo] contrib: datajoin

init-contrib:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

compile:
     [echo] contrib: datajoin
    [javac] /opt/hadoop/src/contrib/build-contrib.xml:190: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

check-contrib:
     [echo] eclipse.home unset: skipping eclipse plugin

init:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

compile:

check-contrib:

init:
     [echo] contrib: failmon

init-contrib:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

compile:
     [echo] contrib: failmon
    [javac] /opt/hadoop/src/contrib/build-contrib.xml:190: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

check-contrib:

init:
     [echo] contrib: fairscheduler

init-contrib:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

compile:
     [echo] contrib: fairscheduler
    [javac] /opt/hadoop/src/contrib/build-contrib.xml:190: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

check-libhdfs-fuse:

check-libhdfs-exists:

compile:

check-contrib:

init:
     [echo] contrib: gridmix

init-contrib:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:
[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve]       ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]       ::          UNRESOLVED DEPENDENCIES         ::
[ivy:resolve]       ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]       :: org.mortbay.jetty#jsp-2.1;6.1.14: configuration not found in org.mortbay.jetty#jsp-2.1;6.1.14: 'master'. It was required from org.apache.hadoop#gridmix;working@master common
[ivy:resolve]       ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/opt/hadoop/build.xml:723: The following error occurred while executing this line:
/opt/hadoop/src/contrib/build.xml:30: The following error occurred while executing this line:
/opt/hadoop/src/contrib/build-contrib.xml:493: impossible to resolve dependencies:
   resolve failed - see output for details

Total time: 13 seconds


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: fairscheduler編譯失敗
文章發表於 : 2014-02-24, 18:39 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
若修改的範圍不大,
建議使用 jar update 的方式把某個 java class 更新即可。

範例:http://trac.nchc.org.tw/cloud/wiki/III140118/Lab9

- Jazz


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: fairscheduler編譯失敗
文章發表於 : 2014-02-25, 23:12 
離線

註冊時間: 2014-02-20, 16:50
文章: 3
謝謝jazz大的回覆

不好意思,我編譯這邊一直搞不清楚

請問是用javac直接將LoadManager編譯成.class檔嗎?

如果裡面的宣告有用到其他.java檔,要怎麼一起編譯呢?


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: fairscheduler編譯失敗
文章發表於 : 2014-02-26, 00:35 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
zxc54832 寫:
謝謝jazz大的回覆
不好意思,我編譯這邊一直搞不清楚
請問是用javac直接將LoadManager編譯成.class檔嗎?
如果裡面的宣告有用到其他.java檔,要怎麼一起編譯呢?


我的作法是把有改到的 java 檔放在 src
然後因為其他的 java 檔在 hadoop-core-*.jar 裏面有 .class
所以 namespace 都找得到,編譯時拿 hadoop-core-*.jar 當 CLASSPATH 就編譯得過。
編譯完成後,得到有改的 class 檔,然後用 jar -u 的方式更新回 hadoop-core-*.jar

大意是這樣,我的範例是用 ant 去做這整個流程,所以快很多。

PS. 若還是不清楚,那就看要不要弄成壓縮檔或放上 github 供測試。

- Jazz


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: fairscheduler編譯失敗
文章發表於 : 2014-02-26, 15:51 
離線

註冊時間: 2014-02-20, 16:50
文章: 3
謝謝jazz大,運行成功了。
我想再請教ㄧ個問題,由於修改過後想觀看運行中參數的變化
所以試著用System.out.print()印出來,可是終端機上沒有顯示
也沒有stdout,觀察log檔也沒出現,要怎麼樣才能印出來?


回頂端
 個人資料 E-mail  
 
 文章主題 : Re: fairscheduler編譯失敗
文章發表於 : 2014-02-28, 09:01 
離線

註冊時間: 2009-11-09, 19:52
文章: 2897
zxc54832 寫:
謝謝jazz大,運行成功了。
我想再請教ㄧ個問題,由於修改過後想觀看運行中參數的變化
所以試著用System.out.print()印出來,可是終端機上沒有顯示
也沒有stdout,觀察log檔也沒出現,要怎麼樣才能印出來?


不管是偽分散模式(pseudo-distributed mode)或全分散模式(Full Distributed Mode)都不會印到您目前的終端機。
若要觀察 System.out.println() 的結果,請到 http://JobTracker:50030 查到該 Job ID
然後逐一看每個 Mapper 跟 Reducer 的 Task Logs
Ex. http://hadoop.nchc.org.tw:50030/taskdetails.jsp?jobid=job_201402251125_0016&tipid=task_201402251125_0016_m_000000
(註:hadoop.nchc.org.tw 因為 Task Logs 只能在內網查詢,必須連 ssh 進 hadoop.nchc.org.tw 然後用 w3m "網址" 才看得到 Task Logs )

應該會看到類似以下結果,您在 map() 與 reduce() 所呼叫的 System.out.println() 會出現在 stdout logs 的地方:
代碼:
<html>
<title>Task Logs: 'attempt_201402251125_0016_m_000000_0'</title>
<body>
<h1>Task Logs: 'attempt_201402251125_0016_m_000000_0'</h1><br>
<br><b><u>stdout logs</u></b><br>
<pre>
Computing keypoints on the two images...
12928 ASIFT keypoints are detected.
Keypoints computation accomplished in 7 seconds.
Computing keypoints on the two images...
8972 ASIFT keypoints are detected.
Keypoints computation accomplished in 6 seconds.
Matching the keypoints...
The two images match! 909 matchings are identified. log(nfa)=-1499.52.
Keypoints matching accomplished in 2 seconds.
</pre></td></tr></table><hr><br>
<br><b><u>stderr logs</u></b><br>
<pre>
</pre></td></tr></table><hr><br>
<br><b><u>syslog logs</u></b><br>
<pre>
2014-02-27 12:54:22,252 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2014-02-27 12:54:22,440 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2014-02-27 12:54:22,528 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2014-02-27 12:54:22,528 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2014-02-27 12:54:38,891 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2014-02-27 12:54:38,950 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2014-02-27 12:54:38,961 INFO org.apache.hadoop.mapred.TaskRunner: Task:attempt_201402251125_0016_m_000000_0 is done. And is in the process of commiting
2014-02-27 12:54:38,995 INFO org.apache.hadoop.mapred.TaskRunner: Task 'attempt_201402251125_0016_m_000000_0' done.
</pre></td></tr></table><hr><br>
</body></html>


回頂端
 個人資料 E-mail  
 
顯示文章 :  排序  
發表新文章 回覆主題  [ 6 篇文章 ] 

所有顯示的時間為 UTC + 8 小時


誰在線上

正在瀏覽這個版面的使用者:沒有註冊會員 和 2 位訪客


不能 在這個版面發表主題
不能 在這個版面回覆主題
不能 在這個版面編輯您的文章
不能 在這個版面刪除您的文章
不能 在這個版面上傳附加檔案

搜尋:
前往 :  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
正體中文語系由 竹貓星球 維護製作