pyspark - unix_timestamp 是否截断或舍入毫秒？

Question

来自reference :

Convert time string with given pattern (‘yyyy-MM-dd HH:mm:ss’, by default) to Unix time stamp (in seconds), using the default timezone and the default locale, return null if fail.

我发现这会减少 DataFrame 时间戳列的毫秒数。我只是想知道它是否只是简单地截断或将时间戳四舍五入到最接近的秒。

Answer 1

没有文档备份，但在 @spark 2.2.0 中，它是截断，这是一个演示:

from pyspark.sql import Row
import pyspark.sql.functions as F
r = Row('datetime')
lst = [r('2017-10-29 10:20:30.102'), r('2017-10-29 10:20:30.999')]

df = spark.createDataFrame(lst)

(df.withColumn('trunc_datetime', F.unix_timestamp(F.col('datetime')))
   .withColumn('seconds', F.from_unixtime(F.col('trunc_datetime'), 'ss'))
   .show(2, False))

+-----------------------+--------------+-------+
|datetime               |trunc_datetime|seconds|
+-----------------------+--------------+-------+
|2017-10-29 10:20:30.102|1509286830    |30     |
|2017-10-29 10:20:30.999|1509286830    |30     |
+-----------------------+--------------+-------+