java - 如何从 JavaPairRDD 中过滤记录

Question

我正在 Apache Spark 中做一个简单的 WordCount 示例，现在我终于得到了单词数我只想从中过滤唯一的单词。

public class SparkClass {
    public static void main(String[] args) {

    String file = "/home/bhaumik/Documents/my";
    JavaSparkContext sc = new JavaSparkContext("local", "SimpleApp");
    JavaRDD<String> lines = sc.textFile("/home/bhaumik/Documents/myText", 5)
            .flatMap(new FlatMapFunction<String, String>() {

                @Override
                public Iterable<String> call(String t) throws Exception {
                    // TODO Auto-generated method stub
                    return Arrays.asList(t.split(" "));
                }
            });

    JavaPairRDD<String, Integer> pairs = lines.mapToPair(new PairFunction<String, String, Integer>() {

        @Override
        public Tuple2<String, Integer> call(String t) throws Exception {
            // TODO Auto-generated method stub
            return new Tuple2<String, Integer>(t, 1);
        }
    });

    JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer, Integer>() {

        @Override
        public Integer call(Integer v1, Integer v2) throws Exception {
            // TODO Auto-generated method stub
            return v1 + v2;
        }
    });
}

}

Answer 1

在 counts 中，您有一个包含 key 及其出现次数的 RDD。你现在不能得到最小值，所以你应该减少

Tuple2<String, Integer> minApp = counts.reduce((a, b) -> (a._2 > b._2)? b : a);