python - 如何将参数传递给 Hive 中的 Python 流脚本？

Question

Hive 用户可以通过脚本流式传输表以转换该数据:

ADD FILE replace-nan-with-zeros.py;

SELECT
  TRANSFORM (...)
  USING 'python replace-nan-with-zeros.py'
  AS (...)
FROM some_table;

我有一个简单的 Python 脚本:

#!/usr/bin/env python
import sys


kFirstColumns= 7

def main(argv):

    for line in sys.stdin:
        line = line.strip();
        inputs = line.split('\t')

        # replace NaNs with zeros
        outputs = [ ]
        columnIndex = 1;
        for value in inputs:
            newValue = value
            if columnIndex > kFirstColumns:
                newValue = value.replace('NaN','0.0')
            outputs.append(newValue)
            columnIndex = columnIndex + 1

        print '\t'.join(outputs)

if __name__ == "__main__":
    main(sys.argv[1:])

如何使 kFirstColumns 成为此 Python 脚本的命令行或其他类型的参数？

谢谢!

Answer 1

解决方案真的很简单。使用

ADD FILE replace-nan-with-zeros.py;

SELECT
  TRANSFORM (...)
  USING 'python replace-nan-with-zeros.py 7'
  AS (...)
FROM some_table;

不仅仅是

  ...
  USING 'python replace-nan-with-zeros.py'
  ...

它对我来说很好。

Python 脚本应更改为:

kFirstColumns= int(sys.argv[1])