python - 内存使用过多 xarray `to_dataframe()`

Question

我正在使用 xarray 读取一个相当小的 NetCDF 文件 (5.4MB)，并想将其转换为 Pandas 数据框:

import xarray as xr
f = xr.open_dataset('file.nc')

到目前为止，Python 使用的内存非常少(~75 MB)，但是一旦我调用:

f2 = f.to_dataframe()

内存使用量激增 (>12 GB)。有谁知道为什么会这样？我尝试使用 to_dask_dataframe()，但这会导致我在 NetCDF 文件中的某些数据类型上出错。

我上传了 NetCDF 文件 here ，由于我无法分发原始数据，因此将原始数据替换为随机数。

---

根据评论中的要求:

In [3]: f
Out[3]: 
<xarray.Dataset>
Dimensions:          (day_in_time_interval: 3652, nv: 2, time: 175296)
Coordinates:
  * time             (time) datetime64[ns] 2001-01-01 2001-01-01T00:30:00 ...
Dimensions without coordinates: day_in_time_interval, nv
Data variables:
    iso_dataset      |S1 ...
    product          |S1 ...
    station_details  |S1 ...
    date             (time) int32 ...
    valid_dates      (day_in_time_interval) int8 ...
    time_bnds        (time, nv) float32 ...
    C020             (time) float32 ...
    C060             (time) float32 ...
    C120             (time) float32 ...
    C200             (time) float32 ...
Attributes:
    institution:              Royal Netherlands Meteorological Institute (KNMI)
    comment:                  none
    Conventions:              CF-1.4
    location:                 CESAR observatory, the Netherlands
    file_creation_date_time:  20161130 09:34:56 (UTC)

和原始文件的ncdump:

netcdf ecnco2 {
dimensions:
    time = UNLIMITED ; // (175296 currently)
    nv = 2 ;
    day_in_time_interval = 3652 ;
variables:
    char iso_dataset ;
        iso_dataset:hierarchyLevel = "dataset" ;
        iso_dataset:url = "http://www.cesar-database.nl" ;
        iso_dataset:protocol = "website" ;
        iso_dataset:topic = "climatologyMeteorologyAtmosphere" ;
        iso_dataset:westbound_longitude = "4.926" ;
        iso_dataset:eastbound_longitude = "4.926" ;
        iso_dataset:southbound_latitude = "51.97" ;
        iso_dataset:northbound_latitude = "51.97" ;
        iso_dataset:datasetDateType = "publication" ;
        iso_dataset:code = "28992" ;
        iso_dataset:codeSpace = "EPSG" ;
        iso_dataset:accessConstraints = "CESAR data policy" ;
        iso_dataset:useLimitation = "None" ;
        iso_dataset:organisationName_dataset = "Royal Netherlands Meteorological Institute (KNMI)" ;
        iso_dataset:email_dataset = "fred.bosveld@knmi.nl" ;
        iso_dataset:role_dataset = "Principle Investigator" ;
        iso_dataset:organisationName_metadata = "Royal Netherlands Meteorological Institute (KNMI)" ;
        iso_dataset:role_metadata = "Principle Investigator" ;
        iso_dataset:email_metadata = "fred.bosveld@knmi.nl" ;
        iso_dataset:url_metadata = "http://www.knmi.nl/~bosveld" ;
        iso_dataset:metadataDateType = "creation" ;
        iso_dataset:language = "eng" ;
        iso_dataset:metadataStandardName = "ISO-19115" ;
        iso_dataset:metadataStandardNameVersion = "Nederlands profiel op ISO 19115 voor geografie, v1.2" ;
    char product ;
        product:format_version = "netCDF,3.6" ;
        product:originator = "Bosveld, F.C., KNMI" ;
        product:software_version = "see http://www.knmi.nl/~bosveld -> software -> Mobibase" ;
        product:command_line = " ncselect.x ecnco2 a30 [M]cesar,[o]ecnco2 2001,2010 -fecnco2.nc" ;
        product:date_start_of_data = "2001-01-01T00:00Z" ;
        product:date_end_of_data = "2010-12-31T23:59Z" ;
        product:revision_date = "2016-11-30" ;
    char station_details ;
        station_details:name = "CESAR observatory" ;
        station_details:latitude = "51.97" ;
        station_details:longitude = "4.926" ;
        station_details:elevation = "-0.7" ;
        station_details:WMO_id = "06348" ;
        station_details:address = "Zijdeweg 1" ;
        station_details:postal_code = "3411 MH" ;
        station_details:city = "Lopik" ;
        station_details:administration_area = "Utrecht" ;
        station_details:country = "the Netherlands" ;
    float time(time) ;
        time:units = "hours since 2001-01-01 00:00:00 0:00" ;
        time:long_name = "hours since 2001-01-01 00:00:00 (UTC)" ;
        time:standard_name = "time" ;
        time:axis = "T" ;
        time:bounds = "time_bnds" ;
    int date(time) ;
        date:long_name = "yyyymmdd" ;
    byte valid_dates(day_in_time_interval) ;
        valid_dates:comment = "indicates whether any data are included for a particular day: 0=none, 1=data, index runs from date indicated by \"units\" attribute of the time variable" ;
    float time_bnds(time, nv) ;
    float C020(time) ;
        C020:units = "ppm" ;
        C020:long_name = "CO2 concentration ECN at 20 m" ;
        C020:_FillValue = -9999.f ;
        C020:cell_methods = "time: mean" ;
    float C060(time) ;
        C060:units = "ppm" ;
        C060:long_name = "CO2 concentration ECN at 60 m" ;
        C060:_FillValue = -9999.f ;
        C060:cell_methods = "time: mean" ;
    float C120(time) ;
        C120:units = "ppm" ;
        C120:long_name = "CO2 concentration ECN at 120 m" ;
        C120:_FillValue = -9999.f ;
        C120:cell_methods = "time: mean" ;
    float C200(time) ;
        C200:units = "ppm" ;
        C200:long_name = "CO2 concentration ECN at 200 m" ;
        C200:_FillValue = -9999.f ;
        C200:cell_methods = "time: mean" ;

// global attributes:
        :institution = "Royal Netherlands Meteorological Institute (KNMI)" ;
        :comment = "none" ;
        :Conventions = "CF-1.4" ;
        :location = "CESAR observatory, the Netherlands" ;
        :file_creation_date_time = "20161130 09:34:56 (UTC)" ;
        :_Format = "classic" ;
}

Answer 1

发生这种情况是因为您的数据集具有多个维度，并且要在一个数据框中表示所有这些，必须进行大量广播。为了说明，让我们减小数据集的大小:

In [8]: ds_small = ds.isel(time=slice(0, 4), day_in_time_interval=slice(0, 2))

In [9]: ds_small
Out[9]:
<xarray.Dataset>
Dimensions:          (day_in_time_interval: 2, nv: 2, time: 4)
Coordinates:
  * time             (time) datetime64[ns] 2001-01-01 2001-01-01T00:30:00 ...
Dimensions without coordinates: day_in_time_interval, nv
Data variables:
    iso_dataset      |S1 ...
    product          |S1 ...
    station_details  |S1 ...
    date             (time) int32 ...
    valid_dates      (day_in_time_interval) int8 ...
    time_bnds        (time, nv) float32 ...
    C020             (time) float32 ...
    C060             (time) float32 ...
    C120             (time) float32 ...
    C200             (time) float32 ...
Attributes:
    institution:              Royal Netherlands Meteorological Institute (KNMI)
    comment:                  none
    Conventions:              CF-1.4
    location:                 CESAR observatory, the Netherlands
    file_creation_date_time:  20161130 09:34:56 (UTC)

In [10]: ds_small.to_dataframe()
Out[10]:
                                            iso_dataset product station_details      date  valid_dates  time_bnds      C020      C060      C120      C200
day_in_time_interval nv time
0                    0  2001-01-01 00:00:00         b''     b''             b''  20010101            0        0.0  0.749853  0.311870  0.644066  0.231409
                        2001-01-01 00:30:00         b''     b''             b''  20010101            0        0.5  0.758620  0.948448  0.089245  0.632072
                        2001-01-01 01:00:00         b''     b''             b''  20010101            0        1.0  0.649947  0.542748  0.422275  0.555378
                        2001-01-01 01:30:00         b''     b''             b''  20010101            0        1.5  0.972251  0.766816  0.180199  0.441256
                     1  2001-01-01 00:00:00         b''     b''             b''  20010101            0        0.5  0.749853  0.311870  0.644066  0.231409
                        2001-01-01 00:30:00         b''     b''             b''  20010101            0        1.0  0.758620  0.948448  0.089245  0.632072
                        2001-01-01 01:00:00         b''     b''             b''  20010101            0        1.5  0.649947  0.542748  0.422275  0.555378
                        2001-01-01 01:30:00         b''     b''             b''  20010101            0        2.0  0.972251  0.766816  0.180199  0.441256
1                    0  2001-01-01 00:00:00         b''     b''             b''  20010101            0        0.0  0.749853  0.311870  0.644066  0.231409
                        2001-01-01 00:30:00         b''     b''             b''  20010101            0        0.5  0.758620  0.948448  0.089245  0.632072
                        2001-01-01 01:00:00         b''     b''             b''  20010101            0        1.0  0.649947  0.542748  0.422275  0.555378
                        2001-01-01 01:30:00         b''     b''             b''  20010101            0        1.5  0.972251  0.766816  0.180199  0.441256
                     1  2001-01-01 00:00:00         b''     b''             b''  20010101            0        0.5  0.749853  0.311870  0.644066  0.231409
                        2001-01-01 00:30:00         b''     b''             b''  20010101            0        1.0  0.758620  0.948448  0.089245  0.632072
                        2001-01-01 01:00:00         b''     b''             b''  20010101            0        1.5  0.649947  0.542748  0.422275  0.555378
                        2001-01-01 01:30:00         b''     b''             b''  20010101            0        2.0  0.972251  0.766816  0.180199  0.441256

因此，在我的示例中，您的 3 维数据集最终会生成长度等于维度乘积 (4*2*2) 的 3 级 MultiIndex。

您可能想要通过删除一些虚假变量/维度来稍微清理您的数据集。以下行适用于您共享的小型或完整数据集:

In [14]: ds[['C020', 'C060', 'C120', 'C200']].to_dataframe().head()
Out[14]:
                         C020      C060      C120      C200
time
2001-01-01 00:00:00  0.749853  0.311870  0.644066  0.231409
2001-01-01 00:30:00  0.758620  0.948448  0.089245  0.632072
2001-01-01 01:00:00  0.649947  0.542748  0.422275  0.555378
2001-01-01 01:30:00  0.972251  0.766816  0.180199  0.441256
2001-01-01 02:00:00  0.302412  0.389957  0.702632  0.497293