job.setOutputFormatClass(TableOutputFormat.class); 为什么报错 如下：
The method setOutputFormatClass(Class<? extends OutputFormat>) in the type Job is not applicable for the arguments (Class<TableOutputFormat>)
注释掉这句话运行就报：
Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.
import 忽略
public class HdfsAppend2HbaseUtil extends Configured implements Tool{
public static class HdfsAdd2HbaseMapper extends Mapper<Text, Text, ImmutableBytesWritable, Put>{
public void map(Text ikey, Text ivalue, Context context)
throws IOException, InterruptedException {
//将新的追加到hbase中现存的数据中
String oldIdList = HBaseHelper.getValueByKey(ikey.toString());
StringBuffer sb = new StringBuffer(oldIdList);
String newIdList = ivalue.toString();
sb.append("\t" + newIdList);
Put p = new Put(ikey.toString().getBytes());
p.addColumn("idFam".getBytes(), "idsList".getBytes(), sb.toString().getBytes());
context.write(new ImmutableBytesWritable(), p);
}
}
public int run(String[] paths) throws Exception {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", "master,salve1");
conf.set("hbase.zookeeper.property.clientPort", "2181");
Job job = Job.getInstance(conf,"AppendToHbase");
job.setJarByClass(cn.edu.hadoop.util.HdfsAppend2HbaseUtil.class);
job.setInputFormatClass(KeyValueTextInputFormat.class);
job.setMapperClass(HdfsAdd2HbaseMapper.class);
job.setNumReduceTasks(0);
// job.setOutputFormatClass(TableOutputFormat.class);
job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE, "reachableTable");
FileInputFormat.setInputPaths(job, new Path(paths[0]));
job.setOutputKeyClass(ImmutableBytesWritable.class);
job.setOutputValueClass(Put.class);
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(String[] args) throws Exception {
System.out.println("Append Start: ");
long time1 = System.currentTimeMillis();
long time2;
String[] pathsStr = {Const.TwoDegreeReachableOutputPathDetail};
int exitCode = ToolRunner.run(new HdfsAppend2HbaseUtil(), pathsStr);
time2 = System.currentTimeMillis();
System.out.println("Append Cost " + "\t" + (time2-time1)/1000 +" s");
System.exit(exitCode);
}
}
jazz 寫:
yantaiGao 寫:
在map中，需要将hdfs文件，追加到一个hbase表中，这个表中有数据，目标是根据hdfs中数据，更新这个hbase中数据 ，当hbase 表是空表时候没有问题，但是当hbase表中有数据时候，往其中更新数据，在程序运行之初会出这个一个问题：
Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.
錯誤訊息只有提到沒有設定輸出目錄，不過原因很難猜。
其次，想透過 map() 將 hdfs 檔案追加倒 hbase 資料表，那該檔案是 HFile 格式嗎？
感覺還是透過 HBase API 會比較正規一點。
- Jazz
登入
註冊
問答集
搜尋
