python - 如何在Python中迭代一个大文本文件(906.2MB)？

Question

这是我正在使用的代码片段

tweets_data=[]
with open('/home/surya/tweet2.txt','r') as f:
   while True:
      next_n_lines= list(islice(f,10000))
      if not next_n_lines:
         break
      for line in next_n_lines:
         try:
             tweet= json.loads(line)
             tweets_data.append(tweet)
         except:
             continue
print(len(tweets_data))

在这里，我使用了 Python 中 itertools 包中的 islice 来逐行循环 block ，而不是一次性循环 whloe 文件。我使用了一个数组 tweets_data，因为稍后我计划用它来使用 matplotlib 进行一些分析。

我已经在 119.2 MB 的同一文件上测试了代码，大约需要 30 秒，但在 906.2 MB 的文件上，我的 Ubuntu 挂起，我在等待很长时间后被迫终止进程。

这是我的 tweet.txt 文件的格式

{"created_at":"Thu Jun 16 15:44:22 +0000 2016","id":743469273415311360,"id_str":"743469273415311360","text":"RT @itvnews: This is how the fans in Lens reacted as Daniel Sturridge scored the winning #ENG goal #ENGWAL #EURO2016\nhttps:\/\/t.co\/7uta44gRh1","source":"\u003ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":29464323,"id_str":"29464323","name":"Darren Roberts","screen_name":"Darren081969","location":"Kent England","url":null,"description":null,"protected":false,"verified":false,"followers_count":14,"friends_count":49,"listed_count":0,"favourites_count":9,"statuses_count":9,"created_at":"Tue Apr 07 14:58:50 +0000 2009","utc_offset":3600,"time_zone":"London","geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"1A1B1F","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme9\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme9\/bg.gif","profile_background_tile":false,"profile_link_color":"2FC2EF","profile_sidebar_border_color":"181A1E","profile_sidebar_fill_color":"252429","profile_text_color":"666666","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/742699562217943041\/e87qqP_5_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/742699562217943041\/e87qqP_5_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/29464323\/1465908742","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Thu Jun 16 15:39:45 +0000 2016","id":743468111282659328,"id_str":"743468111282659328","text":"This is how the fans in Lens reacted as Daniel Sturridge scored the winning #ENG goal #ENGWAL #EURO2016\nhttps:\/\/t.co\/7uta44gRh1","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":21866939,"id_str":"21866939","name":"ITV News","screen_name":"itvnews","location":"United Kingdom","url":"http:\/\/www.itv.com\/news","description":"Breaking news and the biggest stories from the UK and around the world. Bulletins weekdays at 1:30pm, 6:30pm and 10pm on ITV. Email yourstory@itv.com","protected":false,"verified":true,"followers_count":1447643,"friends_count":1371,"listed_count":5097,"favourites_count":245,"statuses_count":142011,"created_at":"Wed Feb 25 13:51:23 +0000 2009","utc_offset":3600,"time_zone":"London","geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"006473","profile_background_image_url":"http:\/\/pbs.twimg.com\/profile_background_images\/766810747\/a53a8d9148ca55ff82b713b6a9631fe7.jpeg","profile_background_image_url_https":"https:\/\/pbs.twimg.com\/profile_background_images\/766810747\/a53a8d9148ca55ff82b713b6a9631fe7.jpeg","profile_background_tile":true,"profile_link_color":"006473","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDFFCC","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/3105933720\/17bab587f151975ca4ae9f6a1cab7c56_normal.png","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/3105933720\/17bab587f151975ca4ae9f6a1cab7c56_normal.png","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/21866939\/1398434757","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"retweet_count":53,"favorite_count":67,"entities":{"hashtags":[{"text":"ENG","indices":[76,80]},{"text":"ENGWAL","indices":[86,93]},{"text":"EURO2016","indices":[94,103]}],"urls":[{"url":"https:\/\/t.co\/7uta44gRh1","expanded_url":"https:\/\/amp.twimg.com\/v\/a4cdb597-bde4-4523-8cf6-591d823723c8","display_url":"amp.twimg.com\/v\/a4cdb597-bde\u2026","indices":[104,127]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en"},"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"ENG","indices":[89,93]},{"text":"ENGWAL","indices":[99,106]},{"text":"EURO2016","indices":[107,116]}],"urls":[{"url":"https:\/\/t.co\/7uta44gRh1","expanded_url":"https:\/\/amp.twimg.com\/v\/a4cdb597-bde4-4523-8cf6-591d823723c8","display_url":"amp.twimg.com\/v\/a4cdb597-bde\u2026","indices":[117,140]}],"user_mentions":[{"screen_name":"itvnews","name":"ITV News","id":21866939,"id_str":"21866939","indices":[3,11]}],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1466091862430"}

我试图用代码实现的目标是尝试找出 tweet.txt 文件中的推文数量。欢迎任何有关更快迭代方式或不同方法的建议。预先感谢。

Answer 1

如果内存使用有问题，请迭代文件本身:

for line in f:
    # your code here

如果您有大量内存，请迭代 .readlines() 列表:

for line in f.readlines():
    # your code here