- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我们有一个默认的 VPC。试图运行数据流作业。初始步骤(读取文件)设法处理 1/2 个步骤。获取 JOB_MESSAGE_ERROR: SDK harness sdk-0-0 disconnected
错误消息,但日志中没有其他内容。已尝试设置角色和 vpc 防火墙规则。
我想使用 Geobeam 图像(Apache Beam Python 3.9 SDK 2.41.0)运行数据流作业。我已将作业定义如下:
def run(pipeline_args, known_args):
import apache_beam as beam
from apache_beam.io.gcp.internal.clients import storage
from apache_beam.options.pipeline_options import PipelineOptions
from geobeam.io import GeoJSONSource, filebasedsource
from geobeam.fn import format_record, make_valid, filter_invalid
pipeline_options = PipelineOptions([
] + pipeline_args)
with beam.Pipeline(options=pipeline_options) as p:
(p
| beam.io.Read(GeoJSONSource(known_args.gcs_url, encoding='utf-8'))
| 'FilterCords' >> beam.Filter(lambda x: len(x[-1]["coordinates"]) > 1)
| 'MakeValid' >> beam.Map(make_valid)
| 'FilterInvalid' >> beam.Filter(filter_invalid)
| 'FormatRecords' >> beam.Map(format_record)
| beam.io.WriteToText(known_args.gcs_write_url)
)
if __name__ == '__main__':
import logging
import argparse
logging.getLogger().setLevel(logging.INFO)
parser = argparse.ArgumentParser()
parser.add_argument('--gcs_url')
parser.add_argument('--gcs_write_url')
known_args, pipeline_args = parser.parse_known_args()
run(pipeline_args, known_args)
我使用以下命令运行作业:
python -m main --runner DataflowRunner --project [[project_id]] \
--temp_location gs://[[temp_bucket_name]]/tmp \
--gcs_url gs://[[inputbucket_name]]/[[filename]].geojson \
--region europe-north1 --sdk_container_image gcr.io/dataflow-geobeam/example \
--gcs_write_url gs://gs://[[outputbucket_name]]/[[filename]]_processed.geojson \
--subnetwork [[full_link_to_subnet]]
我们已经设置了自定义默认 VPC,并且我在 GCP 中为计算虚拟机资源添加了入口/导出防火墙规则的推荐范围。我还为用于数据流作业的默认服务帐户赋予了以下角色:
我还在服务帐户上提供了我的用户角色:
它说作业已停止,但那是因为作业不会继续进行。我得到以下日志输出
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-10-18_05_33_31-17288646308046950877 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:31.708Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:32.780Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-10-18_05_33_31-17288646308046950877. The number of workers will be between 1 and 1000.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:32.803Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-10-18_05_33_31-17288646308046950877.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:34.374Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in europe-north1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.092Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.109Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.141Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.160Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step WriteToText/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.184Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.200Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.226Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.243Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/InitializeWrite into WriteToText/Write/WriteImpl/DoOnce/Map(decode)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.262Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>) into WriteToText/Write/WriteImpl/DoOnce/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.278Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/DoOnce/Map(decode) into WriteToText/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.294Z: JOB_MESSAGE_DETAILED: Fusing consumer Read/Map(<lambda at iobase.py:908>) into Read/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.310Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/PairWithRestriction into Read/Map(<lambda at iobase.py:908>)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.325Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/SplitWithSizing into ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/PairWithRestriction
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.340Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterCords into ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/ProcessElementAndRestrictionWithSizing
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.356Z: JOB_MESSAGE_DETAILED: Fusing consumer MakeValid into FilterCords
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.372Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterInvalid into MakeValid
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.387Z: JOB_MESSAGE_DETAILED: Fusing consumer FormatRecords into FilterInvalid
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.402Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn) into FormatRecords
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.417Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/WriteBundles into WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.432Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/Pair into WriteToText/Write/WriteImpl/WriteBundles
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.447Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/GroupByKey/Write into WriteToText/Write/WriteImpl/Pair
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.464Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToText/Write/WriteImpl/Extract into WriteToText/Write/WriteImpl/GroupByKey/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.489Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.504Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.519Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.535Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.624Z: JOB_MESSAGE_DEBUG: Executing wait step start19
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.655Z: JOB_MESSAGE_BASIC: Executing operation Read/Impulse+Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/PairWithRestriction+ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/SplitWithSizing
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.668Z: JOB_MESSAGE_BASIC: Executing operation WriteToText/Write/WriteImpl/DoOnce/Impulse+WriteToText/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>)+WriteToText/Write/WriteImpl/DoOnce/Map(decode)+WriteToText/Write/WriteImpl/InitializeWrite
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.682Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:33:35.696Z: JOB_MESSAGE_BASIC: Starting 1 workers in europe-north1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-10-18_05_33_31-17288646308046950877 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:34:21.585Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:37:30.456Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:42:40.315Z: JOB_MESSAGE_BASIC: Finished operation Read/Impulse+Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/PairWithRestriction+ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6/SplitWithSizing
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:42:40.354Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_6-split-with-sizing-out3" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-18T12:42:42.422Z: JOB_MESSAGE_ERROR: SDK harness sdk-0-0 disconnected.
然后它再次尝试将 worker 数量增加到 1,然后它立即收到 JOB_MESSAGE_ERROR: SDK harness sdk-0-0 disconnected.
一遍又一遍。附带说明 - 在管道实际启动之前还需要大约 10 分钟。
我设法让它与 DirectRunner
选项一起工作。我不知道去哪里看?会不会跟VPC有关?
我尝试在原生/默认图像和地理束图像上运行字数统计示例,它适用于原生/默认图像,但不适用于地理束图像。
为什么会这样?
最佳答案
经过反复试验,我发现geobeam 基础镜像的python 版本必须与您机器上的本地python 版本相匹配,否则将无法运行。在回答时,这是 python 3.8。
关于python - Apache Beam Pipeline 与 DirectRunner 一起运行,但在初始读取步骤期间因 DataflowRunner(SDK 线束 sdk-0-0 断开连接)而失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/74111801/
我是在项目中使用 keras 的新手。我一直在我的模型中使用generator。 我真的很困惑我应该输入什么值 1) In fit_generator : steps_per_epoch & vali
假设我们有如下情况: A has to give $10 to B. B has to give $20 to C. C has to give $10 to D. 现在这种情况可以简化为: A lo
我正在尝试对特定列(在工作表“OA”中)进行相对引用,我需要在 110 的步骤中检索新工作表中的单元格内容 例如, =OA!$AB217 =OA!$AB327 =OA!$AB437 与其在每个单元格中
我的 PowerShell 控制台启动时间很慢(总是等待超过 5 秒),并且希望获得有关故障排除步骤的建议,以找出瓶颈可能在哪里? 我已经阅读了关于运行脚本的内容,-NoProfile防止模块等加载很
我在 NativeScript 应用程序中使用 slider 小部件,我想知道是否有步骤属性。在我的例子中,小部件代表金钱,我希望以 5 美元的增量滑动。 我查看了文档,但找不到任何对这种情况有帮助的
我在 NativeScript 应用程序中使用 slider 小部件,我想知道是否有步骤属性。在我的例子中,小部件代表金钱,我希望以 5 美元的增量滑动。 我查看了文档,但找不到任何对这种情况有帮助的
这是我的code : &n
为什么 (2) c.ERR(模棱两可)?第一个方法参数 - char ('a') 被扩展为 float => 匹配。 如果找到匹配项,是否无需继续执行第 2 步(装箱/拆箱)或第 3 步(尝试可变参数
我有一个函数,它处理一个包含 6100 个列表项的列表。当列表只有 300 个项目时,该代码可以正常工作。但是立即与 6100 崩溃。有没有一种方法可以遍历这 6100 个项目,一次说 30 个,然后
1.制作PHP安装程序的原理 其实PHP程序的安装原理无非就是将数据库结构和内容导入到相应的数据库中,从这个过程中重新配置连接数据库的参数和文件,为了保证不被别人恶意使用安装文件,当安装
我创建了一个类似于 primeNG page 的步骤组件我想把他放在一个 dynamic dialog 里面但在应用它之后,“第 1 步”和“第 2 步”不会呈现。 查看代码,我发现关键部分是我们打开
我在理解描述的 MixColumns 步骤时遇到问题 here . 我知道扩散,这一切都是有道理的,因为它指出每列都被视为多项式并乘以 GF(2^8) 的模。 但是..乘以GF(2 ^ 8)。尽管域仍
根据我对 TeamCity 工作原理的观察,我注意到在所有步骤执行完毕后评估构建失败条件。这很烦人,因为如果满足任何构建失败条件,我不能有一个不会执行的步骤。 我不是指常见的构建失败条件,例如“至少一
基于这篇试图在我的环境中测试管道代码的帖子。但它给出了以下错误消息。如何修复他的管道代码? ERROR: Unable to find project for artifact copy: test
我参与了一个项目,需要向我的一位同事提供生产数据的子集(日期范围),以进行故障排除。我想将经过清理的生产数据子集插入新的数据库表中我的同事可以访问。请提出实现此目标的最佳方法。 最佳答案 最简单的方法
我有这样的场景: 鉴于我去这个页面 当我输入 cucumber 时 然后我点击 然后我应该看到文字 我不应该看到这条线 如果我运行这个场景,它将执行所有 5 个步骤。但是我想跳过第4步(然后我应该看到
是否有任何功能可以避免 m 文件的绘图输出? 我的意思是我在文件的开头放置了一个函数(如 clc),然后所有绘图函数都被阻止。 最佳答案 您可以使用自己的(嵌套在您的函数内或同一目录中)重载内置绘图函
我是小 cucumber 语言的新手,这在我看来是非常基本的问题,但我找不到答案。 我知道可以在 Gherking 中编写多行步骤参数,如下所示: Given a blog post named "R
即使其中一个步骤失败,有没有办法继续执行 Cucumber Steps。在我当前的设置中,当一个步骤失败时, cucumber 会跳过剩余的步骤......我想知道是否有某种方法可以设置 cucumb
start-step-stop 码是一种数据压缩技术,用于压缩相对较小的数字。 该代码的工作原理如下:它具有三个参数,start、step 和 stop。 Start 确定用于计算前几个数字的位数。
我是一名优秀的程序员,十分优秀!