python - 如何在 AppEngine MapReduce 中获取计数器结果？

Question

我正在使用谷歌mapreduce lib处理我的数据。在处理数据时，在映射器函数中使用计数器。但我不知道如何在 final方法中获取计数器结果。

def mapper(obj):
    yield obj
    yield operation.counters.Increment("process-obj")


class Test(base_handler.PipelineBase):
    """A pipeline to ingest log as CSV in Google Storage
    """

    def run(self, setting_id):
        filepath = yield mapreduce_pipeline.MapperPipeline(
            "test",
            "mapper",
            "mapreduce.input_readers.DatastoreInputReader",
            output_writer_spec="mapreduce.output_writers.FileOutputWriter",
            params={

            },
            shards=10
        )
    def finalized(self):
        # how to read the counter process-obj
        # how to get the setting_id
        pass

Answer 1

命名输出可能就是您正在寻找的。您可以查看更多详情here .

这是您的代码，使用命名输出来获取各种计数器，包括您定义的计数器:

def mapper(obj):
    yield obj
    yield operation.counters.Increment("process-obj")


class Test(base_handler.PipelineBase):
    """A pipeline to ingest log as CSV in Google Storage
    """

    output_names = ['counters']

    def run(self, setting_id):
        results = yield mapreduce_pipeline.MapperPipeline(
          "test",
          "mapper",
          "mapreduce.input_readers.DatastoreInputReader",
          output_writer_spec="mapreduce.output_writers.FileOutputWriter",
          params={

          },
          shards=10
        )
        yield MapreduceResult(results.counters)

    def finalized(self):
        print 'Counters here: ', self.outputs.counters


class MapreduceResult(base_handler.PipelineBase):

    def run(self, counters):
        self.fill(self.outputs.counters, counters)