- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我正在尝试将 csv 转换为 Parquet。我正在使用 python 3.6 和 Spark 2.3.1 64 位。我无法找到给定回溯的解决方案。我也在使用 64 位 python。
我有这个 csv:
Corp,Vathanya Beck
Corp,Mario Bazile
Open,Hasom Bennitt-traflet
Open,Jonathon Berry
Corp,Ayinde Amezquita
Corp,Carol Airiofolo
Corp,Wilfredo Brozo
我可以使用 pandas 函数 to_parquet 将 csv 制作为 Parquet ,但不知何故 Spark 无法正常工作。在 pandas 中,我使用 pyarrow 引擎进行转换。我正在使用以下 Spark 代码将 csv 转换为 Parquet:
from pyspark import SparkContext, SparkConf
conf = SparkConf()
sc = SparkContext(conf=conf)
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
import pyspark
from pyspark.sql.session import SparkSession
from pyspark.sql.types import StructType ,StructField,StringType
SparkSession.builder.config(conf=conf).appName("BLEH").getOrCreate()
schema = StructType([StructField('type', StringType(), True),
StructField('name1', StringType(), True)])
df = sqlContext.read.csv('cv_transactions.csv',schema)
df.show()
以下是读取 Spark 数据帧中的 csv 后的给定输出。
+----+--------------------+
|type| name1|
+----+--------------------+
|Corp| Vathanya Beck|
|Corp| Mario Bazile|
|Open|Hasom Bennitt-tra...|
|Open| Jonathon Berry|
|Corp| Ayinde Amezquita|
|Corp| Carol Airiofolo|
|Corp| Wilfredo Brozo|
+----+--------------------+
但是当我尝试使用以下代码转换为 Parquet 时:
df.write.parquet('r.parquet')
它给了我以下错误:
Py4JJavaError: An error occurred while calling o347.parquet.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:224)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:154)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:225)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:547)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 1 times, most recent failure: Lost task 0.0 in stage 25.0 (TID 25, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\rohan_pawar\Documents\parquet\r\_temporary\0\_temporary\attempt_20181214173824_0025_m_000000_0\part-00000-e076c220-6226-4617-abf9-14e7f3a2ce81-c000.snappy.parquet
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:241)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:342)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:302)
at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.newOutputWriter(FileFormatWriter.scala:367)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:378)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:269)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:267)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1414)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:272)
... 8 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1602)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1590)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1589)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1589)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1823)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1772)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1761)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:194)
... 31 more
Caused by: org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\rohan_pawar\Documents\parquet\r\_temporary\0\_temporary\attempt_20181214173824_0025_m_000000_0\part-00000-e076c220-6226-4617-abf9-14e7f3a2ce81-c000.snappy.parquet
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:241)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:342)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:302)
at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.newOutputWriter(FileFormatWriter.scala:367)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:378)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:269)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:267)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1414)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:272)
... 8 more
最佳答案
我能够很好地运行你的代码。我的 Spark 版本如下。
$ pyspark --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_191
查看Py4JJavaError: An error occurred while calling o26.parquet. (Reading Parquet file)看看你是否有同样的问题。请检查您正在运行的 pyspark 版本,如果问题仍然存在,请使用完整的堆栈跟踪更新问题。
关于python - 使用 pyspark 将 csv 文件转换为 parquet 文件 : Py4JJavaError: An error occurred while calling o347. parquet 错误,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/53779327/
谁能用一个例子来解释 BooleanQuery 中 lucene 中的 BooleanClause.Occur.Must 和 BooleanClause.Occur.SHOULD 之间的区别? 最佳答
正如标题所说,multiset 在所有相同值的范围末尾插入一个值。 (例如:在多重集 1,2,2,3 中插入 2 使其成为 1,2,2,/*new*/2,3)。 如何在所有相同值范围的开头插入新值?
所以这是与此(Inserting in a multiset: before the first occurence of that value instead of after the last o
我试图从我的 WCF .Net Framework 4.5 向 API rest 发布一个文件。这是我的代码: public string CreateConclusion(string[] inst
我的 SQL 查询获取固件的错误修复验证列表,例如def-456 是一张票,要求我对产品进行固件测试。 def-456 有几个记录结果的子任务。结果记录为:id:abc-123、abc-124、abc
我想删除文件中多次出现的行,但想保留某些行。我该怎么做? 这是我的文件的一部分,我想更改它: §M: 1, K: 2 name, time, cycle, instr, L1-mi
我正在 SSMS 中测试 SQL 2016 Live Query Stats,每次尝试时都会收到错误消息“执行批处理时出错。错误消息是:发生一个或多个错误。”并且不返回任何结果集。一位同事试过了,对他
我们在 JBoss 4.2 上设置了一个水平集群。在我们将缓存模式从 REPL_ASYNC 更改为 REPL_SYNC 以解决问题之前, session 复制工作正常。我们开始看到一些 session
我正在尝试将 MVC 网站发布为 Azure 网络角色。 当我在本地运行它时,一切正常。 但是当我将其发布到 Azure 并浏览某些 MVC 操作时,我收到此错误: Server Error in '
假设一个静态库 libfoo 依赖于另一个静态库 libbar 的某些功能。这些和我的应用程序都是用 D 编写的。如果我的应用程序只直接使用 libfoo,并且只调用 libfoo 中的函数而不引用
我正在尝试在 Eclipse Helios 上安装 SVN 客户端, 我已经从 Collaboration 节点安装了所有 SVN 模块(更新中), 现在重启后我可以选择一个连接器 出现“颠覆性连接器
我在 cakephp 中有一些代码会产生错误。 这是 PHP Controller : $this->loadModel( 'Vote' ); //Newly added by amit start
我需要有关 Java 代码的帮助。 这就是问题所在: 输入示例:AaaaaAa 输出:A 出现 7。 问题是我需要它来忽略案例。 请帮助我,我的代码工作正常,只是它不忽略大小写。 import jav
我正在为 J2ME 开发一个应用程序,有时它完全卡住并且 AMS 需要相当长的时间来关闭它.在我看来,这像是一个死锁问题。 你能告诉我什么会导致死锁吗?例如,如果对象调用其自身的另一个同步方法,调用对
尝试将 DEXguard 安装到 Eclipse 中的简单应用程序时出现以下错误: Errors occurred during the build. Errors running builder '
在 SAS 中,假设我有一个名为“person_groups”的数据集。它有两个变量,名为“person”和“group”。该数据集只是将每个人分配到一个组。 我如何从这个数据集中删除所有在他们的组中
有人知道如何在表达式中找到第 n 次出现的字符串以及如何用正则表达式替换它吗? 例如我有以下字符串 txt sub("(^(.*?-){4}.*?)-(.*?-.*?)-", "\\1|\\3||"
是否有一个包允许我为同一个缓冲区设置多个 Occur 结果缓冲区(例如 grep-a-lot: http://www.emacswiki.org/emacs/grep-a-lot.el )。 我在分析
我一直在寻找这个,但似乎无法找到它。 我有一个带有 try {} catch {} 语句的脚本。如果没有发生错误,我想添加一个操作。 例如 try { something } catch { "Err
我正在从 iPhone 应用程序将照片上传到 Facebook。我已经让它工作了,只是有时它会返回“发生未知错误”。我不确定问题是什么。这种情况发生的概率约为 75%。 其他人也遇到过这种情况吗? 最
我是一名优秀的程序员,十分优秀!