python - 如何向数据行添加标题属性？

Question

我有一个从 igra2 weather dataset 获得的天气数据集

文件的典型部分如下所示:

#ICXUAE05424 1909 04 22 99 0935    6          erac-hud  657000      -180000
30 -9999  -9999   100 -9999 -9999 -9999   120    49
30 -9999  -9999   350 -9999 -9999 -9999   119   110
30 -9999  -9999   750-9999 -9999 -9999   149    97
30 -9999  -9999  1250-9999 -9999 -9999   136   123
30 -9999  -9999  1750-9999 -9999 -9999   104   121
30 -9999  -9999  2250-9999 -9999 -9999   117   171
#ICXUAE05424 1909 04 22 99 1820    3          erac-hud  657000      -180000
30 -9999  -9999   100 -9999 -9999 -9999   120    53
30 -9999  -9999   350A -9999 -9999 -9999   111    69
30 -9999  -9999   750B-9999 -9999 -9999   102    55
#ICXUAE05424 1909 04 23 99 0845    5          erac-hud  657000      -180000
30 -9999  -9999   100 -9999 -9999 -9999    31     9
30 -9999  -9999   350 -9999 -9999 -9999   102    62
30 -9999  -9999   750 -9999 -9999 -9999   103   132
30 -9999  -9999  1250 -9999 -9999 -9999    98   120
30 -9999  -9999  1750 -9999 -9999 -9999   101   100

我需要通过将部分(或全部)标题属性附加到其数据行来预处理数据，然后将其转换为 csv 文件。如果不是 Python Pandas，如何在 linux bash 中使用 sed 实现此目的输出 csv 文件应类似于以下形式:

lvl12,etime,press,gph,temp,rh,dpdp,wdir,wspd,hour,lattitude,longitude
21,-9999,96900A,234,270A,742,-9999,-9999,-9999,12,316333,748667
20,-9999,95000,-9999,290A,484,-9999,-9999,-9999,12,316333,748667
20,-9999,88700,-9999,290A,454,-9999,-9999,-9999,12,316333,748667
10,-9999,85000,1384A,260A,446,-9999,-9999,-9999,12,316333,748667
10,-9999,70000,3055A,130A,506,-9999,-9999,-9999,12,316333,748667
20,-9999,58400,-9999,0A,690,-9999,-9999,-9999,12,316333,748667
20,-9999,55900,-9999,0A,312,-9999,-9999,-9999,12,316333,748667
10,-9999,50000,5772A,-65A-9999,-9999,320,850,,12,316333,748667

其他数据集信息:

以 # 为前缀的行是标题，后面的行是数据。

标题属性是:

station code, year, month, day, etc

第七个属性空格分隔为npv，表示后面的数据行数。

数据列是:

lvl12, etime, press, gph, temp, rh, dpdp, wdir, wspd

Answer 1

您需要手动逐行解析文件，并注意 header 的位置。

我假设750-9999之类的数据实际上有一个空格750 -9999？如果不是这种情况，则需要固定宽度的方法:

这可以仅使用 Python 的 CSV 库来完成，如下所示:

import csv

header = ["lvl12", "etime", "press", "gph", "temp", "rh", "dpdp", "wdir", "wspd", "hour", "lattitude", "longitude"]    
data = []

with open('weather.txt', newline='') as f_input, open('output.csv', 'w', newline='') as f_output:
    csv_input = csv.reader(f_input, delimiter=' ', skipinitialspace=True)
    csv_output = csv.writer(f_output)
    csv_output.writerow(header)

    for row in csv_input:
        if row[0].startswith('#'):
            header = row
        else:
            csv_output.writerow(row + [header[5]] + header[-2:])

或者，如果您也想使用 Pandas:

import pandas as pd
import csv

data = []

with open('weather.txt', newline='') as f_input:
    csv_input = csv.reader(f_input, delimiter=' ', skipinitialspace=True)

    for row in csv_input:
        if row[0].startswith('#'):
            header = row
        else:
            data.append(row + [header[5]] + header[-2:])

columns = ["lvl12", "etime", "press", "gph", "temp", "rh", "dpdp", "wdir", "wspd", "hour", "lattitude", "longitude"]
df = pd.DataFrame(data, columns=columns)
print(df) 

给你:

   lvl12  etime  press   gph   temp     rh   dpdp wdir wspd  hour lattitude longitude
0     30  -9999  -9999   100  -9999  -9999  -9999  120   49  0935    657000   -180000
1     30  -9999  -9999   350  -9999  -9999  -9999  119  110  0935    657000   -180000
2     30  -9999  -9999   750  -9999  -9999  -9999  149   97  0935    657000   -180000
3     30  -9999  -9999  1250  -9999  -9999  -9999  136  123  0935    657000   -180000
4     30  -9999  -9999  1750  -9999  -9999  -9999  104  121  0935    657000   -180000
..   ...    ...    ...   ...    ...    ...    ...  ...  ...   ...       ...       ...
9     30  -9999  -9999   100  -9999  -9999  -9999   31    9  0845    657000   -180000
10    30  -9999  -9999   350  -9999  -9999  -9999  102   62  0845    657000   -180000
11    30  -9999  -9999   750  -9999  -9999  -9999  103  132  0845    657000   -180000
12    30  -9999  -9999  1250  -9999  -9999  -9999   98  120  0845    657000   -180000
13    30  -9999  -9999  1750  -9999  -9999  -9999  101  100  0845    657000   -180000

[14 rows x 12 columns]

使用 Python 3.x 进行测试。如果您使用的是 Python 2.x，请更改以下内容:

with open('weather.txt', 'rb') as f_input, open('output.csv', 'wb') as f_output: