python - 如何为每个句子(行)创建标记化单词(列)的数据框？

Question

我有以下文字:

“大家好，我叫sam!我喜欢麻辣拉面。我也很喜欢打游戏。”

我的目标是将此段落转换为每个句子的标记化单词数据框。 (其中行数等于句子数，列数等于最长句子中的单词数)。

我开始创建标记化句子的数据框:

from nltk.tokenize import sent_tokenize, word_tokenize

df = pd.DataFrame({"sentences": sent_tokenize(paragraph)})

结果是:

    sentences
0   Hi there, my name is sam!
1   I love spicy hand pulled noodles.
2   I also like to game alot.

然后我将每个句子(行)转换为标记化单词列表:

df["tokens"] = df.sentences.apply(word_tokenize)

结果是(如果我单独打印该列):

0    [Hi, there, ,, my, name, is, sam, !]
1    [I, love, spicy, hand, pulled, noodles, .]
2    [I, also, like, to, game, alot, .]

接下来我希望发生的事情是这样的(这里需要帮助):

      w1   w2     w3      w4     w5       w6       w7     w8
0     Hi   there  ,       my     name     is       sam    !
1     I    love   spicy   hand   pulled   noodles  .      NaN
2     I    also   like    to     game     alot     .      NaN

其中列数等于最长word_tokenized句子的长度。对于比最长的句子短的句子，我希望空列包含 NaN 值(甚至 0.0)。有没有办法用 pandas 命令实现这一点？

Answer 1

如果第一个前缀列以 1 (w1) 开头:

In [350]: df.join(pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)])).fillna(np.nan)               
Out[350]: 
                           sentences                                      tokens  w1     w2     w3    w4      w5       w6   w7   w8
0          Hi there, my name is sam!        [Hi, there, ,, my, name, is, sam, !]  Hi  there      ,    my    name       is  sam    !
1  I love spicy hand pulled noodles.  [I, love, spicy, hand, pulled, noodles, .]   I   love  spicy  hand  pulled  noodles    .  NaN
2          I also like to game alot.          [I, also, like, to, game, alot, .]   I   also   like    to    game     alot    .  NaN

---

如果您需要它作为一个单独的数据框:

In [352]: pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)]).fillna(np.nan)                        
Out[352]: 
   w1     w2     w3    w4      w5       w6   w7   w8
0  Hi  there      ,    my    name       is  sam    !
1   I   love  spicy  hand  pulled  noodles    .  NaN
2   I   also   like    to    game     alot    .  NaN