python - 溢出错误: size does not fit in an int

Question

我正在编写一个在 AzureML 中使用的 python 脚本。我的数据集相当大。我有一个数据集，其中包含名为 ID(int) 和 DataType(text) 的列。我想将这些值连接到只有一列，其中包含 ID 和 DataType 并用逗号分隔的文本。

执行此操作时如何避免出现错误。我的代码有错误吗？

当我运行此代码时，出现以下错误:

Error 0085: The following error occurred during script evaluation, please view the output log for more information:
---------- Start of error message from Python interpreter ----------
data:text/plain,Caught exception while executing function: Traceback (most recent call last):
File "C:\server\invokepy.py", line 167, in batch
idfs.append(rutils.RUtils.RFileToDataFrame(infile))
File "C:\server\RReader\rutils.py", line 15, in RFileToDataFrame
rreader = RReaderFactory.construct_from_file(filename, compressed)
File "C:\server\RReader\rreaderfactory.py", line 25, in construct_from_file
return _RReaderFactory.construct_from_stream(stream)
File "C:\server\RReader\rreaderfactory.py", line 46, in construct_from_stream
return RReader(BinaryReader(RFactoryConstants.big_endian, stream.read()))
File "C:\pyhome\lib\gzip.py", line 254, in read
self._read(readsize)
File "C:\pyhome\lib\gzip.py", line 313, in _read
self._add_read_data( uncompress )
File "C:\pyhome\lib\gzip.py", line 329, in _add_read_data
self.crc = zlib.crc32(data, self.crc) & 0xffffffffL
OverflowError: size does not fit in an int

我的代码如下:

# The script MUST contain a function named azureml_main
# which is the entry point for this module.
#
# The entry point function can contain up to two input arguments:
#   Param<dataframe1>: a pandas.DataFrame
#   Param<dataframe2>: a pandas.DataFrame

def azureml_main(dataframe1):
import pandas as pd
dataframe1['SignalID,DataType'] = dataframe1['ID'] + " , " + dataframe1['DataType']
dataframe1 = dataframe1.drop('DataType')
dataframe1 = dataframe1.drop('ID')
# Return value must be of a sequence of pandas.DataFrame
return dataframe1

当我在 AzureML 中运行默认 python 代码时，出现相同的错误。所以我很确定我的数据不适合数据框。

默认脚本如下:

# The script MUST contain a function named azureml_main
# which is the entry point for this module.
#
# The entry point function can contain up to two input arguments:
#   Param<dataframe1>: a pandas.DataFrame
#   Param<dataframe2>: a pandas.DataFrame
def azureml_main(dataframe1 = None, dataframe2 = None):

    # Execution logic goes here
    print('Input pandas.DataFrame #1:\r\n\r\n{0}'.format(dataframe1))

    # If a zip file is connected to the third input port is connected,
    # it is unzipped under ".\Script Bundle". This directory is added
    # to sys.path. Therefore, if your zip file contains a Python file
    # mymodule.py you can import it using:
    # import mymodule

    # Return value must be of a sequence of pandas.DataFrame
    return dataframe1,

Answer 1

如果您需要将整数 ID 和字符串 DataType 列连接到新列 SignalID，请使用 astype 进行转换。然后你就可以drop列 DataType 和 ID 添加参数 axis=1:

import pandas as pd

def azureml_main(dataframe1):
    dataframe1['SignalID'] = dataframe1['ID'].astype(str) 
                                      + " , " 
                                      + dataframe1['DataType']

    dataframe1 = dataframe1.drop(['DataType', 'ID'], axis=1)
    # Return value must be of a sequence of pandas.DataFrame
    return dataframe1