python - 根据日期键的值过滤字典

Question

我想从某个日期起从世界各地尽可能多的来源导入文章。

import requests
url = ('https://newsapi.org/v2/top-headlines?'
       'country=us&'
       'apiKey=de9e19b7547e44c4983ad761c104278f')
response = requests.get(url)

response_dataframe = pd.DataFrame(response.json())

articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2019-01-04T11:30:00Z'}
print(articles)

但我得到:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-84-0f21f2f50907> in <module>
      2 response_dataframe['articles'][1]['publishedAt']
      3 
----> 4 articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2018-01-04T11:30:00Z'}
      5 print(articles)

<ipython-input-84-0f21f2f50907> in <setcomp>(.0)
      2 response_dataframe['articles'][1]['publishedAt']
      3 
----> 4 articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2018-01-04T11:30:00Z'}
      5 print(articles)

TypeError: unhashable type: 'dict'

因此如何通过选择这些键来选择一系列文章？预期输出是一个按日期和报纸对文章进行排序的数据框。

              The New York Times                                The Washington Post                                The Financial Times  
2007-01-01    . What Sticks from '06. Somalia Orders Islamis... New Ebola Vaccine Gives 100 Percent Protecti...
2007-01-02    . Heart Health: Vitamin Does Not Prevent Death... Flurry of Settlements Over Toxic Mortgages M...
2007-01-03    . Google Answer to Filling Jobs Is an Algorith... Jason Miller Backs Out of White House Commun...
2007-01-04    . Helping Make the Shift From Combat to Commer... Wielding Claims of ‘Fake News,’ Conservative...
2007-01-05    . Rise in Ethanol Raises Concerns About Corn a... When One Party Has the Governor’s Mansion an
...

我的Python版本是3.6.6

Answer 1

您正在过滤字典，然后尝试将它们放入集合中。您的预期结果不需要您去重复任何内容，因此避免错误的最简单方法是使用列表理解；只需将 {...} 大括号替换为方括号即可:

articles = [article for article in response_dataframe['articles'] if article['publishedAt'] >= '2019-01-04T11:30:00Z']

但是，如果您要将数据放入数据帧中进行处理，那么最好使用 pandas.io.json.json_normalize() function ;它可以根据通常从 JSON 源加载的列表和字典结构为您生成数据帧。

首先将您想要的文章数据加载到数据框中，然后您可以从那里进行过滤和重新排列；以下代码将所有数据加载到具有新的 date 列的单个数据框中，该列源自 publishAt 信息:

import pandas as pd
from pandas.io.json import json_normalize

df = json_normalize(response.json(), 'articles')

# make the datetime column a native type, and add a date-only column
df['publishedAt'] = pd.to_datetime(df['publishedAt'])
df['date'] = df['publishedAt'].dt.date

# move source dictionary into separate columns rather than dictionaries
source_columns = df['source'].apply(pd.Series).add_prefix('source_')
df = pd.concat([df.drop(['source'], axis=1), source_columns], axis=1)

这为您提供了一个包含所有文章信息的数据框，作为具有 native 类型的完整数据框，其中包含列 author、content、description、publishedAt、date、title、url、urlToImage 和 source 映射中的 >source_id 和 source_name 列。

我注意到API已经允许您按日期进行过滤，我会依赖它而不是在本地进行过滤，因为您可以通过让API为您提供较小的数据集来节省时间和带宽。 API 还允许您应用排序，这又是一个好主意。

要按日期和源名称对行进行分组，您必须 pivot the dataframe ;日期应该是索引，列是源名称，标题是值:

df.pivot(index='date', columns='source_name', values='title')

但是这会失败，因为这种格式每天没有足够的空间容纳每个来源多个标题:

ValueError: Index contains duplicate entries, cannot reshape

在提供给我的 JSON 数据中，有多篇 CNN 和福克斯新闻今天的文章。

您可以将多个标题聚合到列表中:

pd.pivot_table(df,
    index='date', columns='source_name', values='title',
    aggfunc=list)

对于“今天”的默认 20 个结果，这给了我:

>>> pd.pivot_table(
...     df, index='date', columns='source_name', values='title',
...     aggfunc=list
... )
source_name                                            Bbc.com                        ...                                                                Youtube.com
date                                                                                  ...
2019-01-05   [Paul Whelan: Russia rules out prisoner swap f...                        ...                          [Bears Buzz: Eagles at Bears - Wildcard Round ...

[1 rows x 18 columns]

就我个人而言，我只是将数据框限制为日期、标题和源名称，并带有日期索引:

>>> df[['date', 'source_name', 'title']].set_index('date').sort_values(['date', 'source_name'])
                    source_name                                              title
date
2019-01-05              Bbc.com  Paul Whelan: Russia rules out prisoner swap fo...
2019-01-05            Bloomberg  Russia Says FBI Arrested Russian Citizen on Pa...
2019-01-05                  CNN  Pay raises frozen for Pence, Cabinet members u...
2019-01-05                  CNN  16 big questions on Robert Mueller's Russia in...
2019-01-05            Colts.com  news What They're Saying: Colts/Texans, Wild C...
2019-01-05             Engadget  Pandora iOS update adds offline playback for A...
2019-01-05             Espn.com  Roger Federer wins Hopman Cup with Switzerland...
2019-01-05             Fox News  Japanese 'Tuna King' pays record $3M for prize...
2019-01-05             Fox News  Knicks' Turkish star Enes Kanter to skip Londo...
2019-01-05          Latimes.com  Flu toll mounts in California, with 42 deaths ...
2019-01-05             NBC News  After the fire: Blazes pose hidden threat to t...
2019-01-05           Newser.com  After Backlash, Ellen Not Ditching Support for...
2019-01-05              Npr.org  Three Dead After Fight Escalates Into Shooting...
2019-01-05              Reuters  French 'yellow vests' rail against unrepentant...
2019-01-05             The Hill  Trump: 'I don’t care' that most federal employ...
2019-01-05  The Huffington Post  5 Children Dead After Church Van Crashes On Wa...
2019-01-05            The Verge  Apple seeks to end bent iPad Pro controversy w...
2019-01-05    Thisisinsider.com  Kanye West surprised Kim Kardashian with a $14...
2019-01-05            USA Today  See 'Mean Girls' co-stars Lindsay Lohan and Jo...
2019-01-05          Youtube.com  Bears Buzz: Eagles at Bears - Wildcard Round -...

以上内容按日期和来源排序，因此来自同一来源的多个标题被分组。