c++的wordcount範例build失敗

sophie · **註冊時間:** 2013-09-01, 23:34 **文章:** 6

遇到build failed問題，新手我想請教各位前輩大大

我嘗試用http://wiki.apache.org/hadoop/C%2B%2BWordCount網站的範例code來進行build
我將wordcount.cpp檔案放在hadoop下的根目錄

其內容如下：

代碼:

#include "hadoop/Pipes.hh"
#include "hadoop/TemplateFactory.hh"
#include "hadoop/StringUtils.hh"

class WordCountMap: public HadoopPipes::Mapper {
public:
  WordCountMap(HadoopPipes::TaskContext& context){}
  void map(HadoopPipes::MapContext& context) {
    std::vector<std::string> words =
      HadoopUtils::splitString(context.getInputValue(), " ");
    for(unsigned int i=0; i < words.size(); ++i) {
      context.emit(words[i], "1");
    }
  }
};

class WordCountReduce: public HadoopPipes::Reducer {
public:
  WordCountReduce(HadoopPipes::TaskContext& context){}
  void reduce(HadoopPipes::ReduceContext& context) {
    int sum = 0;
    while (context.nextValue()) {
      sum += HadoopUtils::toInt(context.getInputValue());
    }
    context.emit(context.getInputKey(), HadoopUtils::toString(sum));
  }
};

int main(int argc, char *argv[]) {
  return HadoopPipes::runTask(HadoopPipes::TemplateFactory<WordCountMap,
                              WordCountReduce>());
}

之後我執行以下指令：

代碼:

ant -Dcompile.c++=yes examples

錯誤訊息：

代碼:

Buildfile: /opt/hadoop/build.xml

clover.setup:

clover.info:
     [echo] 
     [echo]      Clover not found. Code coverage reports disabled.
     [echo]   

clover:

ivy-download:
      [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: /opt/hadoop/ivy/ivy-2.1.0.jar
      [get] Not modified - so not downloaded

ivy-init-dirs:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /opt/hadoop/ivy/ivysettings.xml

init:
    [touch] Creating /tmp/null504780470
   [delete] Deleting: /tmp/null504780470
     [exec] src/saveVersion.sh: 1: src/saveVersion.sh: svn: not found
     [exec] src/saveVersion.sh: 1: src/saveVersion.sh: svn: not found

record-parser:

compile-rcc-compiler:
    [javac] /opt/hadoop/build.xml:472: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

compile-core-classes:
    [javac] /opt/hadoop/build.xml:496: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

compile-hdfs-classes:
    [javac] /opt/hadoop/build.xml:592: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 2 source files to /opt/hadoop/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6
    [javac] 1 warning

compile-mapred-classes:
Trying to override old definition of task jsp-compile
    [javac] /opt/hadoop/build.xml:549: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

create-native-configure:
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
     [exec] libtoolize: copying file `config/config.guess'
     [exec] libtoolize: copying file `config/config.sub'
     [exec] libtoolize: copying file `config/install-sh'
     [exec] libtoolize: copying file `config/ltmain.sh'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
     [exec] libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] configure.ac:42: warning: AC_COMPILE_IFELSE was called before AC_USE_SYSTEM_EXTENSIONS
     [exec] ../../lib/autoconf/specific.m4:314: AC_GNU_SOURCE is expanded from...
     [exec] configure.ac:42: the top level
     [exec] Makefile.am:32: shell echo $$OS_NAME | tr [A-Z] [a-z]: non-POSIX variable name
     [exec] Makefile.am:32: (probably a GNU make extension)

compile-core-native:

create-c++-configure:
     [exec] libtoolize: putting auxiliary files in `.'.
     [exec] libtoolize: copying file `./config.guess'
     [exec] libtoolize: copying file `./config.sub'
     [exec] libtoolize: copying file `./install-sh'
     [exec] libtoolize: copying file `./ltmain.sh'
     [exec] libtoolize: putting macros in `m4'.
     [exec] libtoolize: copying file `m4/libtool.m4'
     [exec] libtoolize: copying file `m4/ltoptions.m4'
     [exec] libtoolize: copying file `m4/ltsugar.m4'
     [exec] libtoolize: copying file `m4/ltversion.m4'
     [exec] libtoolize: copying file `m4/lt~obsolete.m4'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
     [exec] libtoolize: putting auxiliary files in `.'.
     [exec] libtoolize: copying file `./config.guess'
     [exec] libtoolize: copying file `./config.sub'
     [exec] libtoolize: copying file `./install-sh'
     [exec] libtoolize: copying file `./ltmain.sh'
     [exec] libtoolize: putting macros in `../utils/m4'.
     [exec] libtoolize: copying file `../utils/m4/libtool.m4'
     [exec] libtoolize: copying file `../utils/m4/ltoptions.m4'
     [exec] libtoolize: copying file `../utils/m4/ltsugar.m4'
     [exec] libtoolize: copying file `../utils/m4/ltversion.m4'
     [exec] libtoolize: copying file `../utils/m4/lt~obsolete.m4'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
     [exec] libtoolize: putting auxiliary files in `.'.
     [exec] libtoolize: copying file `./config.guess'
     [exec] libtoolize: copying file `./config.sub'
     [exec] libtoolize: copying file `./install-sh'
     [exec] libtoolize: copying file `./ltmain.sh'
     [exec] libtoolize: putting macros in `../../c++/utils/m4'.
     [exec] libtoolize: copying file `../../c++/utils/m4/libtool.m4'
     [exec] libtoolize: copying file `../../c++/utils/m4/ltoptions.m4'
     [exec] libtoolize: copying file `../../c++/utils/m4/ltsugar.m4'
     [exec] libtoolize: copying file `../../c++/utils/m4/ltversion.m4'
     [exec] libtoolize: copying file `../../c++/utils/m4/lt~obsolete.m4'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.

check-c++-libhdfs:

create-c++-configure-libhdfs:

check-c++-makefiles:

create-c++-pipes-makefile:
     [exec] checking for a BSD-compatible install... /usr/bin/install -c
     [exec] checking whether build environment is sane... yes
     [exec] checking for a thread-safe mkdir -p... /bin/mkdir -p
     [exec] checking for gawk... no
     [exec] checking for mawk... mawk
     [exec] checking whether make sets $(MAKE)... yes
     [exec] checking for style of include used by make... GNU
     [exec] checking for gcc... gcc
     [exec] checking whether the C compiler works... yes
     [exec] checking for C compiler default output file name... a.out
     [exec] checking for suffix of executables... 
     [exec] checking whether we are cross compiling... no
     [exec] checking for suffix of object files... o
     [exec] checking whether we are using the GNU C compiler... yes
     [exec] checking whether gcc accepts -g... yes
     [exec] checking for gcc option to accept ISO C89... none needed
     [exec] checking dependency style of gcc... gcc3
     [exec] checking how to run the C preprocessor... gcc -E
     [exec] checking for grep that handles long lines and -e... /bin/grep
     [exec] checking for egrep... /bin/grep -E
     [exec] checking for ANSI C header files... yes
     [exec] checking for sys/types.h... yes
     [exec] checking for sys/stat.h... yes
     [exec] checking for stdlib.h... yes
     [exec] checking for string.h... yes
     [exec] checking for memory.h... yes
     [exec] checking for strings.h... yes
     [exec] checking for inttypes.h... yes
     [exec] checking for stdint.h... yes
     [exec] checking for unistd.h... yes
     [exec] checking minix/config.h usability... no
     [exec] checking minix/config.h presence... no
     [exec] checking for minix/config.h... no
     [exec] checking whether it is safe to define __EXTENSIONS__... yes
     [exec] checking for special C compiler options needed for large files... no
     [exec] checking for _FILE_OFFSET_BITS value needed for large files... no
     [exec] checking pthread.h usability... yes
     [exec] checking pthread.h presence... yes
     [exec] checking for pthread.h... yes
     [exec] checking for pthread_create in -lpthread... yes
     [exec] checking for HMAC_Init in -lssl... /opt/hadoop/src/c++/pipes/configure: line 421: test: please: integer expression expected
     [exec] /opt/hadoop/src/c++/pipes/configure: line 424: $4: Bad file descriptor
     [exec] configure: error: check
     [exec] /opt/hadoop/src/c++/pipes/configure: line 310: return: please: numeric argument required
     [exec] /opt/hadoop/src/c++/pipes/configure: line 320: exit: please: numeric argument required
     [exec] no

BUILD FAILED
/opt/hadoop/build.xml:2164: exec returned: 255

系統回傳build failed想請問是什麼原因，這個真的困擾我很久>_<
我該怎麼改正？謝謝各位大大!!!!!!!!!!

jazz · **註冊時間:** 2009-11-09, 19:52 **文章:** 2897

雖然有點答非所問，不過建議不要在這個問題上打轉。(牽涉範圍很廣，您卡住的地方是 configure 而已，還沒 make)

我在 hadoop.nchc.org.tw 的環境底下可以正常編出您的 wordcount pipes 範例

這裡用的是 cloudera CDH2 的 hadoop-0.20-pipes 套件

代碼:

h998@hadoop:~$ dpkg -L hadoop-0.20-pipes
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/hadoop-0.20-pipes
/usr/share/doc/hadoop-0.20-pipes/changelog.Debian.gz
/usr/share/doc/hadoop-0.20-pipes/copyright
/usr/share/doc/hadoop-0.20-pipes/changelog.gz
/usr/include
/usr/include/hadoop
/usr/include/hadoop/TemplateFactory.hh
/usr/include/hadoop/StringUtils.hh
/usr/include/hadoop/Pipes.hh
/usr/include/hadoop/SerialUtils.hh
/usr/lib
/usr/lib/libhadooputils.a
/usr/lib/libhadooppipes.a

編譯時需要用到 libhadooputils.a libhadooppipes.a 跟 libpthread.a

代碼:

h998@hadoop:~$ dpkg -S libpthread.a
libc6-dev-i386: /usr/lib32/libpthread.a

代碼:

h998@hadoop:~$ cat wordcount.cpp 
#include "hadoop/Pipes.hh"
#include "hadoop/TemplateFactory.hh"
#include "hadoop/StringUtils.hh"

class WordCountMap: public HadoopPipes::Mapper {
   public:
      WordCountMap(HadoopPipes::TaskContext& context){}
      void map(HadoopPipes::MapContext& context) {
         std::vector<std::string> words =
            HadoopUtils::splitString(context.getInputValue(), " ");
         for(unsigned int i=0; i < words.size(); ++i) {
            context.emit(words[i], "1");
         }
      }
};

class WordCountReduce: public HadoopPipes::Reducer {
   public:
      WordCountReduce(HadoopPipes::TaskContext& context){}
      void reduce(HadoopPipes::ReduceContext& context) {
         int sum = 0;
         while (context.nextValue()) {
            sum += HadoopUtils::toInt(context.getInputValue());
         }
         context.emit(context.getInputKey(), HadoopUtils::toString(sum));
      }
};

int main(int argc, char *argv[]) {
   return HadoopPipes::runTask(HadoopPipes::TemplateFactory<WordCountMap,
         WordCountReduce>());
}

h998@hadoop:~$ g++ wordcount.cpp -o wordcount -lhadooputils -lhadooppipes -lpthread

h998@hadoop:~$ file wordcount
wordcount: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, not stripped

不過這個範例程式好像沒給 input output 目錄，我不確定該怎麼跑

- Jazz

sophie · **註冊時間:** 2013-09-01, 23:34 **文章:** 6

To Jazz:
謝謝jazz的回答，不過我還是有所疑惑>_<

1.我的版本是hadoop-1.2.1版的，所以我需要退回安裝為0.20的版本嗎？

2.我試著用你的指令編譯

代碼:

g++ wordcount.cpp -o wordcount -lhadooputils -lhadooppipes -lpthread

但是發生了問題...
hadoop/Pipes.hh：沒有此一檔案或目錄
但是我在我的目錄下有看到該檔案，在

代碼:

/opt/hadoop/src/c++/pipes/api/hadoop
/opt/hadoop/c++/Linux-amd64-64/include/hadoop
/opt/hadoop/c++/Linux-i386-32/include/hadoop

為什麼他無法找到Pipes.hh，我該怎麼辦呢？？

3.我看很多網站上都是只打如下的指令

代碼:

ant -Dcompile.c++=yes examples

然後就直接upload c++ binary files to HDFS了

代碼:

bin/hadoop fs -put build/c++-examples/Linux-i386-32/bin /examples/bin

我以為只要把ant -Dcompile.c++=yes example執行完成，就可以上傳HDFS執行了。所以我不是很懂jazz這句話的意思->（卡住的地方是 configure 而已，還沒 make）

此外執行此範例的指令為下

代碼:

bin/hadoop pipes -conf src/examples/pipes/conf/word.xml -input in-dir -output out-dir

該範例網站並沒有對input多加贅述，所以我想應該是在hadoop安裝檔下一開始就有的範例input

附上網站網址：http://wiki.apache.org/hadoop/C%2B%2BWordCount

thanks, jazz.

jazz · **註冊時間:** 2009-11-09, 19:52 **文章:** 2897

sophie 寫:

1.我的版本是hadoop-1.2.1版的，所以我需要退回安裝為0.20的版本嗎？

不用退回 0.20 的版本，可以改用對應 1.2.1 的 CDH3 或甚至更新的 CDH4，
主要是因為 CDH 有編好的 libhadooputils.a 跟 libhadooppipes.a

sophie 寫:

2.我試著用你的指令編譯

代碼:

g++ wordcount.cpp -o wordcount -lhadooputils -lhadooppipes -lpthread

但是發生了問題...
hadoop/Pipes.hh：沒有此一檔案或目錄
但是我在我的目錄下有看到該檔案，在

代碼:

/opt/hadoop/src/c++/pipes/api/hadoop
/opt/hadoop/c++/Linux-amd64-64/include/hadoop
/opt/hadoop/c++/Linux-i386-32/include/hadoop

為什麼他無法找到Pipes.hh，我該怎麼辦呢？？

預設找的是 /usr/include 底下的，所以用 CDH 套件就不會指定 Pipes.hh 的路徑。
如果要自己指定路徑必須用 -I 指令
g++ wordcount.cpp -o wordcount -I/opt/hadoop/src/c++/pipes/api/hadoop ....

sophie 寫:

所以我不是很懂jazz這句話的意思->（卡住的地方是 configure 而已，還沒 make）

代碼:

ant -Dcompile.c++=yes examples

會先執行 pipes 原始碼底下的 configure 產生 Makefile 然後才會再跑 make

代碼:

~/hadoop/src/c++/pipes$ ls
aclocal.m4  config.guess  configure.ac  impl        Makefile.am
api         config.sub    debug         install-sh  Makefile.in
compile     configure     depcomp       ltmain.sh   missing

C 的編譯本來就會比較複雜，特別是這裡又牽涉到函式庫的 linking。
所以不單單 ant -Dcompile.c++=yes examples 就可以搞定

sophie 寫:

此外執行此範例的指令為下

代碼:

bin/hadoop pipes -conf src/examples/pipes/conf/word.xml -input in-dir -output out-dir

該範例網站並沒有對input多加贅述，所以我想應該是在hadoop安裝檔下一開始就有的範例input
附上網站網址：http://wiki.apache.org/hadoop/C%2B%2BWordCount

既然是 wordcount，input 目錄一般就是內含任意純文字檔即可。

- Jazz

sophie · **註冊時間:** 2013-09-01, 23:34 **文章:** 6

To jazz:
透過這個網頁，我的問題已經順利解決
http://hadoopintouch.blogspot.tw/2012/04/write-your-first-program-in-hadoop-in-c.html

Step 1:
compile the program using g++
following is the command to compile

代碼:

g++ -I/opt/hadoop/c++/Linux-amd64-64/include -c wordcount.cpp
g++ wordcount.o -o wordcount -L/opt/hadoop/c++/Linux-amd64-64/lib -lnsl -lpthread -lhadooppipes -lhadooputils

it will create WordCount binary into your current folder.

Step 2:
Now pur this binary into hadoop hdfs by using following command:

代碼:

bin/hadoop fs -mkdir /user/test
bin/hadoop fs -put WordCount /user/test/

Step 3:
Now make a input file for your test.
use the following command

代碼:

vim input.txt
press i
write few lines
press esc then press wq.

Step 4:
keep your input inside the hadoop by using the following command:

代碼:

bin/hadoop fs -put input.txt /user/test/

Step 5:
Now run your program using following command:

代碼:

bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -program /user/test/WordCount -input /user/test/input.txt -output /user/test/output

Hopefully it will get run now you can see your output in the output directory which we have given at the time of run /user/test/output.

to see the output type following command:

代碼:

bin/hadoop fs -text /user/test/output/part-00000

jazz · **註冊時間:** 2009-11-09, 19:52 **文章:** 2897

Thanks for sharing. I'm sure that someone might benefit from you.

- Jazz

Taiwan Hadoop Forum

c++的wordcount範例build失敗

誰在線上