database - Sentiment140 预处理

Question

我一直在尝试对 Kaggle 上的 Sentiment140 数据库进行一些预处理:https://www.kaggle.com/kazanova/sentiment140

我使用的代码是这样的:

import os
from nltk.stem.lancaster import LancasterStemmer
from nltk.tokenize import RegexpTokenizer

Base_location = ''
dataset_location = os.path.join(Base_location, 'Sentiment140.csv')
corpus = []
labels = []

# Parse tweets and sentiments
with open(dataset_location, 'r', encoding='latin-1') as df:
    for i, line in enumerate(df):
        parts = line.strip().split(',')

        # Sentiment (0 = Negative, 1 = Positive)
        labels.append(str(parts[0].strip()))

        # Tweet
        tweet = parts[5].strip()
        if tweet.startswith('"'):
            tweet = tweet[1:]
        if tweet.endswith('"'):
            tweet = tweet[::-1]

        corpus.append(tweet.strip().lower())

print('Corpus size: {}'.format(len(corpus)))

# Tokenize and stem
tkr = RegexpTokenizer('[a-zA-Z0-9@]+')
stemmer = LancasterStemmer()

tokenized_corpus = []

for i, tweet in enumerate(corpus):
    tokens = [stemmer.stem(t) for t in tkr.tokenize(tweet) if not t.startswith('@')]
    tokenized_corpus.append(tokens)

print(tokenized_corpus)

但是，我不断收到此错误:

TypeError: '_io.TextIOWrapper' object is not subscriptable

谁能帮助我了解如何解决这个问题？

提前致谢

Answer 1

长话短说

要读取 .csv 或结构化数据集，请使用 pandas https://pandas.pydata.org/或任何其他数据框库。

---

长:

而不是做:

Base_location = ''
dataset_location = os.path.join(Base_location, 'Sentiment140.csv')
corpus = []
labels = []

# Parse tweets and sentiments
with open(dataset_location, 'r', encoding='latin-1') as df:
    for i, line in enumerate(df):
        parts = line.strip().split(',')

        # Sentiment (0 = Negative, 1 = Positive)
        labels.append(str(parts[0].strip()))

        # Tweet
        tweet = parts[5].strip()
        if tweet.startswith('"'):
            tweet = tweet[1:]
        if tweet.endswith('"'):
            tweet = tweet[::-1]

        corpus.append(tweet.strip().lower())

您可以简单地用 pandas 读取 .csv 文件，例如

import pandas as pd
corpus = pd.read_csv('training.1600000.processed.noemoticon.csv', encoding='latin-1')

然后使用.apply()函数处理推文:

"""
Columns
====

target: the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)
ids: The id of the tweet ( 2087)
date: the date of the tweet (Sat May 16 23:58:44 UTC 2009)
flag: The query (lyx). If there is no query, then this value is NO_QUERY.
user: the user that tweeted (robotickilldozr)
text: the text of the tweet (Lyx is cool)
"""

from nltk.stem.lancaster import LancasterStemmer
from nltk.tokenize import RegexpTokenizer

import pandas as pd


df = pd.read_csv('training.1600000.processed.noemoticon.csv', 
                 header=None, 
                 names=['target', 'ids', 'date', 'flag', 'user', 'text'],
                 encoding='latin-1')


tokenizer = RegexpTokenizer('[a-zA-Z0-9@]+')
stemmer = LancasterStemmer()

def process_tweet(tweet):
    return [stemmer.stem(token) if not token.startswith('@') else token 
            for token in tokenizer.tokenize(tweet)]

# 1. Cast the column type to string 
# 2. Lowercase it
# 3. Iterate throw each row and get the output from process_tweet()
# 4. # 3. Keep in a new column call `tokenized_text`
df['tokenized_text']= df['text'].str.lower().apply(process_tweet)