python - 如何从 inFile 读取 header 并将 header 写入 outFile？ ( python 3)

Question

我在文件读取中使用了 next 来解析数据而不是标题行，因此读取完全避免了第一行。如何在读取时拾取标题(但不解析标题行)，然后在写入操作中写入标题？

我想要执行此操作的实际数据集是 30 列和 80k 行，因此我试图在一次读取操作中完成此操作。

测试数据:

date, animal, color
3/14/2015, cat, blue
3/24/2015, dog, green

代码:

from dateutil.parser import *
import csv

with open('testin.csv', 'r', encoding='utf-8') as inFile, open('testout.csv', 'w', encoding='utf-8') as outFile:
    exampleReader = csv.reader(inFile)
    next(exampleReader, 1)
    exampleData = list(exampleReader)
    exampleWriter = csv.writer(outFile)
    # print a few to see what it's doing
    print('the list', exampleData)
    for item in exampleData:
        item[0] = str(parse(item[0])) # converting date format for sqlite
        del item[2] # dropping column that is not needed
        print('date corrected', item) 
        exampleWriter.writerow(item)

Answer 1

我会使用 pandas对于如此大量的数据:

import io
import pandas as pd

data = """\
date, animal, color, junk
3/14/2015, cat, blue, aaa
3/24/2015, dog, green, bbb
"""
num_cols = 4
all_cols = set(range(num_cols))
skip_cols = set([2,3])

# replace `io.StringIO(data)` with the CSV filename    
df = pd.read_csv(io.StringIO(data),
                 sep=',',
                 skipinitialspace=True,
                 parse_dates=[0],
                 usecols=(all_cols - skip_cols))
print(df)

# save DF as CSV file
df.to_csv('/path/to/new.csv', index=False)

# save DF to SQLite DB
import sqlalchemy
engine = sqlalchemy.create_engine('sqlite:///my_db.sqlite')
df.to_sql('my_table', engine, if_exists='replace')

示例:

In [150]: data = """\
   .....: date, animal, color, junk
   .....: 3/14/2015, cat, blue, aaa
   .....: 3/24/2015, dog, green, bbb
   .....: """

In [151]: num_cols = 4

In [152]: all_cols = set(range(num_cols))

In [153]: skip_cols = set([2,3])

In [154]: df = pd.read_csv(io.StringIO(data),
   .....:                  sep=',',
   .....:                  skipinitialspace=True,
   .....:                  parse_dates=['date'],
   .....:                  usecols=(all_cols - skip_cols))

In [155]: print(df)
        date animal
0 2015-03-14    cat
1 2015-03-24    dog