python - 使用关键字提取在 Pandas 中动态创建列

Question

我有一个看起来像这样的 pandas DataFrame

Col1,Col2,Col3
1,"this is a text","more text"
2,"this is another text","even more"
3,"here is one more", "something also here"
4,"let's get another one","we are close"
5,"one last text","finally"

然后我在这些文本上应用了名称实体识别，我提取了一些重要的关键词。像这样

def get_entities(ocr, title):
    doc = nlp(' '.join([ocr, title]))
    entities = []
    for ent in doc.ents:
        entity = '_'.join([ent.label_, ent.text])
        entities.append(entity)
    return set(entities)

df['entities'] = df.apply(lambda row: get_entities( row.Col2, row.Col3), axis = 1)

上面创建了一个名为 entities 的新列，该列的行值是不同关键字的列表。假设是这样

Col1,Col3
1,['key1', 'key2']
2,['key3', 'key2']
3,['key4', 'key1']
4,['key3', 'key4']
5,['key5', 'key2']

现在我尝试在该列上应用 get_dummies 并创建所有可能的关键字，行值为 0-1。以上就是

Col1,Col3,key1,key2,key3,key4,key5
1,['key1', 'key2'],1,1,0,0,0
2,['key3', 'key2'],0,1,1,0,0
3,['key4', 'key1'],1,0,0,1,0
4,['key3', 'key4'],0,0,1,1,0
5,['key5', 'key2'],0,1,0,0,1

当然，直接在列表列上应用get_dummies是行不通的

df = pd.concat([df,pd.get_dummies(df['entities'], prefix='entities')],axis=1)

我会很感激任何想法

Answer 1

最简单的解决方案是更改您的函数返回的内容。 get_dummies可以处理分隔符分隔的字符串，这很容易从 get_entities 返回方法。

---

def get_entities(ocr, title):
    ...
    return ','.join(set(entities))

现在您可以使用 get_dummies直接在结果上。以您的第二个示例框架为例，您将得到:

df['Col3'].str.get_dummies(',')

   key1  key2  key3  key4  key5
0     1     1     0     0     0
1     0     1     1     0     0
2     1     0     0     1     0
3     0     0     1     1     0
4     0     1     0     0     1

---

如果您不想更改函数的返回值，请在尝试之前添加另一个步骤 get_dummies使用 str.join

---

df['Col3'].str.join(',').str.get_dummies(',')

   key1  key2  key3  key4  key5
0     1     1     0     0     0
1     0     1     1     0     0
2     1     0     0     1     0
3     0     0     1     1     0
4     0     1     0     0     1