hadoop - 如何调试map-reduce失败的地方？

Question

我编译了可以成功运行的程序，但mapper reducer失败了；我需要代码帮助以成功运行映射器和化简器。

我猜代码中有一些解析问题。附带的是错误，下面是代码。如何找到解决问题的方法？

01/23 03:27:33 INFO mapreduce.Job:  map 0% reduce 0%
19/01/23 03:27:50 INFO mapreduce.Job:  map 100% reduce 0%
19/01/23 03:27:51 INFO mapreduce.Job:  map 0% reduce 0%
19/01/23 03:27:52 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_0, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

19/01/23 03:28:21 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_1, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

19/01/23 03:28:32 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_2, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

19/01/23 03:28:51 INFO mapreduce.Job:  map 100% reduce 100%
19/01/23 03:28:53 INFO mapreduce.Job: Job job_1548178978946_0002 failed with state FAILED due to: Task failed task_1548178978946_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

19/01/23 03:28:53 INFO mapreduce.Job: Counters: 13
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=67635
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=67635
        Total time spent by all reduce tasks (ms)=0
        Total vcore-milliseconds taken by all map tasks=67635
        Total vcore-milliseconds taken by all reduce tasks=0
        Total megabyte-milliseconds taken by all map tasks=69258240
        Total megabyte-milliseconds taken by all reduce tasks=0
19/01/23 03:28:53 INFO mapreduce.Job: Running job: job_1548178978946_0002
19/01/23 03:28:53 INFO mapreduce.Job: Job job_1548178978946_0002 running in uber mode : false
19/01/23 03:28:53 INFO mapreduce.Job:  map 100% reduce 100%
19/01/23 03:28:53 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_0, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

19/01/23 03:28:54 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_1, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

19/01/23 03:28:54 INFO mapreduce.Job: Task Id : attempt_1548178978946_0002_m_000000_2, Status : FAILED
Error: java.lang.NumberFormatException: For input string: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
    at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.lang.Double.parseDouble(Double.java:538)
    at java.lang.Double.<init>(Double.java:608)
    at com.hadoop.imcdp.MA$Map.partitionData(MA.java:69)
    at com.hadoop.imcdp.MA$Map.map(MA.java:58)
    at com.hadoop.imcdp.MA$Map.map(MA.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

19/01/23 03:28:54 INFO mapreduce.Job: Job job_1548178978946_0002 failed with state FAILED due to: Task failed task_1548178978946_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

19/01/23 03:28:54 INFO mapreduce.Job: Counters: 13
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=67635
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=67635
        Total time spent by all reduce tasks (ms)=0
        Total vcore-milliseconds taken by all map tasks=67635
        Total vcore-milliseconds taken by all reduce tasks=0
        Total megabyte-milliseconds taken by all map tasks=69258240
        Total megabyte-milliseconds taken by all reduce tasks=0

移动平均值示例代码

public class MA extends Configured implements Tool{

// For production the windowlength  would be a commandline or other argument        

         static double windowlength = 3.0;

         static int thekey = (int)windowlength/2;

// used for handling the circular list. 

         static boolean initialised=false;

// Sample window

         static ArrayList<Double> window = new ArrayList<Double>() ;



// The Map method processes the data one point at a time and passes the circular list to the 

// reducer. 

       public static class Map extends Mapper<LongWritable, Text, Text, Text> {

         private final static IntWritable one = new IntWritable(1);

         private Text word = new Text();



         public void map(LongWritable key, Text value, Context context

                         ) throws IOException , InterruptedException

                         {

           double wlen = windowlength;

// creates windows of samples and sends them to the Reducer        

           partitionData(value, context, wlen);



         }



// Create sample windows starting at each sata point and sends them to the reducer



        private void partitionData(Text value,

                Context context, double wlen)

                throws IOException , InterruptedException {

            String line = value.toString();

// the division must be done this way in the mapper. 

               Double ival = new Double(line)/wlen;

// Build initial sample window

               if(window.size() < windowlength)

                  {

                    window.add(ival);

                  }

// emit first window

              if(!initialised && window.size() == windowlength)

                  {

                  initialised = true;

                  emit(thekey, window, context);

                  thekey++;

                  return;

                  }

// Update and emit subsequent windows

              if(initialised)

                  {

// remove oldest datum                

                      window.remove(0);

// add new datum

                      window.add(ival);

                      emit(thekey, window, context);

                      thekey++;

                  }

        }

       }



// Transform list to a string and send to reducer. Text to be replaced by ObjectWritable

// Problem: Hadoop apparently requires all output formats to be the same so

//          cannot make this output collector differ from the one the reducer uses.



       public static void emit(int key, ArrayList<Double> value, Context context) throws IOException , InterruptedException

       {

           Text tx = new Text();

           tx.set(new Integer(key).toString());



           String outstring = value.toString();

// remove the square brackets Java puts in

           String tidied = outstring.substring(1,outstring.length()-1).trim();



           Text out = new Text();      

           out.set(tidied);



           context.write(tx,out);          

       }



       public static class Reduce extends Reducer<Text, Text, Text, Text> 

        {

         public void reduce(Text key, 

                            Iterator<Text> values,

                            Context context

                            ) throws IOException , InterruptedException

           {



               while (values.hasNext()) 

                   { 

                        computeAverage(key, values, context);



                  }



             }



// computes the average of each window and sends to ouptut collector.      

        private void computeAverage(Text key, Iterator<Text> values,

                Context context)

                throws IOException , InterruptedException {

            double sum = 0;

            String thevalue = values.next().toString();

            String[] thenumbers = thevalue.split(",");

            for( String temp: thenumbers)

                {

                // need to trim the string because the constructor does not trim.

                Double ds = new Double(temp.trim());    

                sum += ds;



                }              

      Text out = new Text();

      String outstring = Double.toString(sum);

      out.set(outstring);

      context.write (key, out);

        }

       }

     @Override

    public int run(String[] args) throws Exception {

        Configuration conf = new Configuration();

        args = new GenericOptionsParser(conf, args).getRemainingArgs();

        String input = args[0];

        String output = args[1];



        Job job = new Job(conf, "MA");

        job.setJarByClass(MA.class);

        job.setInputFormatClass(TextInputFormat.class);

        job.setMapperClass(Map.class);

        job.setMapOutputKeyClass(Text.class);

        job.setMapOutputValueClass(Text.class);

         job.setCombinerClass(Reduce.class);

        job.setReducerClass(Reduce.class);

        job.setOutputFormatClass(TextOutputFormat.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(Text.class);



        FileInputFormat.setInputPaths(job, new Path(input));

        Path outPath = new Path(output);

        FileOutputFormat.setOutputPath(job, outPath);

        outPath.getFileSystem(conf).delete(outPath, true);



        job.waitForCompletion(true);

        return (job.waitForCompletion(true) ? 0 : 1);

    }
    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new MA(), args);
        System.exit(exitCode);
    }
      // public static void main(String[] args) throws Exception {
       //  JobConf conf = new JobConf(MA.class);
       //  conf.setJobName("MA");
       //  conf.setOutputKeyClass(Text.class);  
       //  conf.setOutputValueClass(Text.class);
      //   conf.setMapperClass(Map.class);
       //  conf.setCombinerClass(Reduce.class);
       //  conf.setReducerClass(Reduce.class);
       //  conf.setInputFormat(TextInputFormat.class);
       //  conf.setOutputFormat(TextOutputFormat.class);
//       FileInputFormat.setInputPaths(conf, new Path(args[0]));

//       FileOutputFormat.setOutputPath(conf, new Path(args[1]));
      //   FileInputFormat.setInputPaths(conf, new Path("input/movingaverage.txt"));
      //   FileOutputFormat.setOutputPath(conf, new Path("output/smoothed"));
      //   JobClient.runJob(conf);
      // }
    }

Answer 1

Double ival = new Double(line)/wlen;

抛出错误，因为 line是 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20，无法解析为Double。假设您希望每个数字都是 double 数，则需要执行以下操作:

List<Double> ivals = new ArrayList<>();
String[] numbers = line.split(",");
for (int i = 0; i < numbers.length(); i++) {
    ivals.add(new Double(numbers[i])/wlen);
}