作者热门文章
- c - 在位数组中找到第一个零
- linux - Unix 显示有关匹配两种模式之一的文件的信息
- 正则表达式替换多个文件
- linux - 隐藏来自 xtrace 的命令
我有一个 11gb 的 json
文件,我无法在 pandas 中加载它。(来源:http://jmcauley.ucsd.edu/data/amazon/)上述链接中的元数据是我正在使用的文件。
元数据:元数据包括描述、价格、销售排名、品牌信息和共同购买链接:
它具有以下模式 -
{ "asin": "0000031852", "title": "Girls Ballet Tutu Zebra Hot Pink", "price": 3.17, "imUrl": "http://ecx.images-amazon.com/images/I/51fAmVkTbyL._SY300_.jpg", "related": { "also_bought": ["B00JHONN1S", "B002BZX8Z6", "B00D2K1M3O", "0000031909", "B00613WDTQ", "B00D0WDS9A", "B00D0GCI8S", "0000031895", "B003AVKOP2", "B003AVEU6G", "B003IEDM9Q", "B002R0FA24", "B00D23MC6W", "B00D2K0PA0", "B00538F5OK", "B00CEV86I6", "B002R0FABA", "B00D10CLVW", "B003AVNY6I", "B002GZGI4E", "B001T9NUFS", "B002R0F7FE", "B00E1YRI4C", "B008UBQZKU", "B00D103F8U", "B007R2RM8W"], "also_viewed": ["B002BZX8Z6", "B00JHONN1S", "B008F0SU0Y", "B00D23MC6W", "B00AFDOPDA", "B00E1YRI4C", "B002GZGI4E", "B003AVKOP2", "B00D9C1WBM", "B00CEV8366", "B00CEUX0D8", "B0079ME3KU", "B00CEUWY8K", "B004FOEEHC", "0000031895", "B00BC4GY9Y", "B003XRKA7A", "B00K18LKX2", "B00EM7KAG6", "B00AMQ17JA", "B00D9C32NI", "B002C3Y6WG", "B00JLL4L5Y", "B003AVNY6I", "B008UBQZKU", "B00D0WDS9A", "B00613WDTQ", "B00538F5OK", "B005C4Y4F6", "B004LHZ1NY", "B00CPHX76U", "B00CEUWUZC", "B00IJVASUE", "B00GOR07RE", "B00J2GTM0W", "B00JHNSNSM", "B003IEDM9Q", "B00CYBU84G", "B008VV8NSQ", "B00CYBULSO", "B00I2UHSZA", "B005F50FXC", "B007LCQI3S", "B00DP68AVW", "B009RXWNSI", "B003AVEU6G", "B00HSOJB9M", "B00EHAGZNA", "B0046W9T8C", "B00E79VW6Q", "B00D10CLVW", "B00B0AVO54", "B00E95LC8Q", "B00GOR92SO", "B007ZN5Y56", "B00AL2569W", "B00B608000", "B008F0SMUC", "B00BFXLZ8M"], "bought_together": ["B002BZX8Z6"] }, "salesRank": {"Toys & Games": 211836}, "brand": "Coxlures", "categories": [["Sports & Outdoors", "Other Sports", "Dance"]] }
另一方面,前 10 行有以下数据:
["{'asin': '0001048791', 'salesRank': {'Books': 6334800}, 'imUrl': 'http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg', 'categories': [['Books']], 'title': 'The Crucible: Performed by Stuart Pankin, Jerome Dempsey & Cast'}\n", "{'asin': '0000143561', 'categories': [['Movies & TV', 'Movies']], 'description': '3Pack DVD set - Italian Classics, Parties and Holidays.', 'title': 'Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays', 'price': 12.99, 'salesRank': {'Movies & TV': 376041}, 'imUrl': 'http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif', 'related': {'also_viewed': ['B0036FO6SI', 'B000KL8ODE', '000014357X', 'B0037718RC', 'B002I5GNVU', 'B000RBU4BM'], 'buy_after_viewing': ['B0036FO6SI', 'B000KL8ODE', '000014357X', 'B0037718RC']}}\n", "{'asin': '0000037214', 'related': {'also_viewed': ['B00JO8II76', 'B00DGN4R1Q', 'B00E1YRI4C']}, 'title': 'Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory', 'price': 6.99, 'salesRank': {'Clothing': 1233557}, 'imUrl': 'http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg', 'brand': 'Big Dreams', 'categories': [['Clothing, Shoes & Jewelry', 'Girls'], ['Clothing, Shoes & Jewelry', 'Novelty, Costumes & More', 'Costumes & Accessories', 'More Accessories', 'Kids & Baby']]}\n", "{'asin': '0000032069', 'title': 'Adult Ballet Tutu Cheetah Pink', 'price': 7.89, 'imUrl': 'http://ecx.images-amazon.com/images/I/51EzU6quNML._SX342_.jpg', 'related': {'also_bought': ['0000032050', 'B00D0DJAEG', '0000032042', 'B00D0F450I', 'B00D2JTMS2', 'B00D0FDUAY', 'B00D2JSRFQ', '0000032034', 'B00D0D5F6S', 'B00D2JRWWA', 'B00D0FIIJM', 'B00D0FCQQI', 'B00EXVN9PU', 'B0041EOTJO', 'B004PYEE8G', 'B001GTKPDQ', 'B00EON0SJ2', 'B005HMHOQ4', 'B002XZMGGQ'], 'also_viewed': ['B00D0F450I', '0000032050', 'B00D2JTMS2', '0000032042', 'B004PYEE8G', 'B00JHNSNSM', 'B00D0DJAEG', 'B00D2JSRFQ', 'B00D0FCQQI', 'B00D2JRWWA', 'B003AVNY6I', 'B0071KR2LC', 'B00GOR07RE', 'B00D0FIIJM', 'B005F50FXC', 'B0079MCIMU', 'B00D0FDUAY', 'B00H3RYN3I', 'B005C4Y4F6', 'B007IEFT84', 'B00D0D5F6S', 'B002BZX8Z6', 'B00JHONN1S', 'B008F0SU0Y', 'B00FNNFXAG', 'B007R2RM8W', 'B007VM3AMK', 'B00C0PLENA', 'B00BJGG6VG', 'B00E1YRI4C', 'B00IIK61WA', 'B009UC638W', 'B00KZN6RVI', 'B00CSFEENY', 'B002GZGI4E', 'B00HSOJJ94', 'B00LIPP4VG', 'B009RXWNSI', 'B00E87F196', 'B005HMHOQY', 'B00J6S9MSS', '0000032034', 'B00CJQGNJK', 'B008FCA0F0', 'B0056LG7GY', 'B00DPQWCZ2', 'B00I3PV0US', 'B00KZN6IVW', 'B0054TBWKO', 'B00I2S01I8', 'B00BXF12P8', 'B00GVHU678', 'B005NWENGC', 'B003AVKOP2', 'B00JK8MQ4Q', 'B00FZIMVQS', 'B008BB19VE', 'B00GTEXPOE', 'B009WPT2RQ', 'B00E37SBBG'], 'bought_together': ['0000032050', 'B00D0DJAEG', '0000032042', 'B00D0F450I']}, 'brand': 'BubuBibi', 'categories': [['Sports & Outdoors', 'Other Sports', 'Dance', 'Clothing', 'Girls', 'Skirts']]}\n", "{'asin': '0000031909', 'related': {'also_bought': ['B002BZX8Z6', 'B00JHONN1S', '0000031895', 'B00D2K1M3O', '0000031852', 'B00D0WDS9A', 'B00D10CLVW', 'B00D103F8U', 'B003AVEU6G', 'B00D2K0PA0', 'B002GZGI4E', 'B00D0ZF44Y', 'B008F0SMUC', 'B00D0GCI8S', 'B008F0SU0Y', 'B002YSCPZY', '0448408775', 'B002R0FABA', 'B008GHWNWC', 'B002R0FA24', 'B001GTKPEK', 'B006XA7KZO', 'B001GZUQ9S', 'B00613VNL0', 'B003IEDM9Q', 'B003LTOZK8', 'B003AVNY6I', 'B008UBQZKU', 'B001AQD8VQ', 'B003ILA0L2', 'B00AFDOPDA', 'B002R0F7FE'], 'also_viewed': ['B002BZX8Z6', 'B00JHONN1S', 'B008F0SU0Y', 'B00E1YRI4C', 'B00AFDOPDA', 'B002GZGI4E', 'B00CEUWY8K', 'B003IEDM9Q', 'B00HSOJB9M', '0000031895', 'B003AVKOP2', 'B005C4Y4F6', 'B008F0SMUC', 'B00362QGW0', 'B008UD01L2', 'B00FAZ5ZE6', 'B008F0SY6O', 'B00DPLLQR2', 'B00CEUWUZC', 'B004PYEE8G', 'B003AVNY6I', 'B00CEUX0D8', 'B00JHNSNSM', 'B00D10CLVW', 'B00D23MC6W', 'B007XAI53E', 'B008X6CBS2', 'B004PEI45U', 'B005HMHOQ4', 'B002C3Y6WG', 'B00HSC8O74', 'B008BMGHM4', 'B00CEUWTFS', 'B00FNNFXAG', 'B00CYBU84G', 'B00D9C32NI', 'B0046W9T8C', 'B008UBG5IW', 'B001YHX45G', 'B00CEV8366', 'B00I2UHSZA', 'B009RXWNSI', 'B008FCA0F0', 'B001GTKPEK', 'B004TU1VPU', 'B00CBPIO7S', 'B00CHHXJ0M', 'B00538F5OK', 'B005F50FXC', 'B00CEUX4QQ', 'B003XRKA7A', '0000031852', 'B002C3R5XI', 'B00C6Q1Z6E'], 'bought_together': ['B002BZX8Z6']}, 'title': 'Girls Ballet Tutu Neon Pink', 'price': 7.0, 'salesRank': {'Toys & Games': 201847}, 'imUrl': 'http://ecx.images-amazon.com/images/I/41xBoP0FVzL._SY300_.jpg', 'brand': 'Unknown', 'categories': [['Sports & Outdoors', 'Other Sports', 'Dance']], 'description': 'High quality 3 layer ballet tutu. 12 inches in length'}\n", "{'asin': '0000032034', 'title': 'Adult Ballet Tutu Yellow', 'price': 7.87, 'imUrl': 'http://ecx.images-amazon.com/images/I/21GNUNIa1CL.jpg', 'related': {'also_bought': ['B00D2JSRFQ', '0000032042', '0000032050', 'B00D2JTMS2', 'B00D0FDUAY', 'B00D0FIIJM', 'B00D2JRWWA', 'B00D0F450I', 'B00D0FCQQI', 'B00H3RYN3I', 'B002I55DT8', 'B00498HUQ6', 'B001YZCF1M', 'B00FNNFXAG', 'B00EON0SJ2', 'B000J09OV2', 'B0048WRX5G', 'B00I2EOG92', 'B003UM99FC', 'B00D0DJAEG', '0000032069', 'B00I2S01I8', 'B003AVKOP2', 'B003CPDAUW', 'B005HMHOQ4', 'B00JHONN1S', 'B00GOR07RE', 'B007TMMVJA', 'B00DPPRW2G', 'B0089ND408', 'B0046W9T8C', 'B005HMHOQE', 'B00EOOR812', 'B00CLZWXYI', 'B008AU29UQ', 'B00BNRKT6E', 'B004YHFSIO', 'B00EB5WN9Q', 'B008UBQZKU', 'B00D0D5F6S', 'B004PYEE8G', 'B00FQU9ZUA', 'B008AABRPO', 'B007BZ5CUA', 'B00I5SCG7E', 'B0036LOTNO', 'B009WPT2SA', 'B009QVCTTY', 'B00KZN5J8U', 'B008B81LN8', 'B00E1YRI4C', 'B004SVOVSE', 'B002I4ZJ1Q', 'B005AZMN3C', 'B00BBQFGWO', 'B009QVQZ3K', 'B005C4Y4F6', 'B008CLA6HG', 'B0085D9V1S', 'B000M55BDY', 'B00FE9DIHO', 'B009QVWIQ8', 'B00LIPP114', 'B001YHX45G', 'B00BN6S8WC', 'B009MDB6FE', 'B007AK1KTS', 'B00J6LZ16M', 'B002FRPE9I', 'B002RHLXKU', 'B006F404KQ', 'B00362OQQI', 'B003UNHJ4Y', 'B00D10CLVW', 'B002BZX8Z6', 'B0041EOTJO', 'B00F3KZUPC', 'B0055A1F4A', 'B0035BGVYU', 'B000P18LZ0', 'B007F2H4PU', 'B004XHVUE6', 'B00KF54D6W', 'B0097B1D8G', 'B00840TWES', 'B0050GAHKC', 'B00I9JSUO2', 'B003HCYEQY', 'B0075CNY7M', 'B00AFDOPDA', 'B008FCA0F0', 'B000IRG356', 'B00DSVBR7S', 'B00DYIQ8E2', 'B0041BVA80', 'B009M7FWBE'], 'also_viewed': ['B00D2JSRFQ', '0000032050', 'B00JHNSNSM', '0000032042', 'B00D2JTMS2', 'B003AVKOP2', 'B004YHFSIO', 'B00GOR07RE', 'B00D0FDUAY', 'B004PYEE8G', 'B00D0FCQQI', 'B009WPT2SA', 'B003AVNY6I', 'B00EON0SJ2', 'B005C4Y4F6', 'B00D0FIIJM', 'B00FNNFXAG', 'B00D0F450I', 'B00D2JRWWA', 'B008F0SU0Y', 'B00JHONN1S', 'B00FE9DIHO', 'B0071KR2LC', 'B00H3RYN3I', 'B00IIK61WA', 'B00D0DJAEG', 'B00KZN6RVI', 'B0054TBWKO', 'B00GEDG8D0', 'B00JMX4CCS', 'B00K18LJ6U', 'B0079MCIMU', 'B005HMHOQY', 'B009RXWNSI', 'B007XAI4LW', 'B007IEFTO8', 'B00E1YRI4C', 'B007R2RM8W', 'B002BZX8Z6', 'B00IIK61UW', 'B008F0SMUC', 'B00KF54D6W', 'B00E1Q66BG', 'B003WFSLBA', 'B00IJVASUE', 'B00DPPRW2G', 'B00HAVJ48G', 'B002C3Y6WG', 'B00I5RLL4Y', 'B003AVEU6G', 'B00E95LC8Q', 'B005F50FXC', 'B002U03H1M', 'B00E87F196', 'B008A7TFK6', 'B00HSOJB9M', 'B008A7TFGK', 'B00DPYOB2Q', '0375851682', 'B00CSFEENY'], 'bought_together': ['B00D2JSRFQ', '0000032050', '0000032042', 'B00D2JTMS2']}, 'brand': 'BubuBibi', 'categories': [['Sports & Outdoors', 'Other Sports', 'Dance', 'Clothing', 'Girls', 'Skirts']]}\n", '{\'asin\': \'0000589012\', \'title\': "Why Don\'t They Just Quit? DVD Roundtable Discussion: What Families and Friends need to Know About Addiction and Recovery", \'price\': 15.95, \'imUrl\': \'http://ecx.images-amazon.com/images/I/519%2B1kseM3L._SY300_.jpg\', \'related\': {\'also_bought\': [\'B000Z3N1HQ\', \'0578045427\', \'B007VI5AQ8\', \'B003AC98V2\', \'B004V4RW8O\', \'B000I0QL7I\', \'B000J10F8C\', \'B0007CEXYK\', \'B000ERVK4Y\', \'B000XSKDBA\', \'B002UNMWTC\', \'B00008MTXI\', \'B007TSV4GK\', \'B0052ADP6Y\', \'B00EUENWIY\', \'B003YKYX9M\', \'B004RD3YFE\', \'B007Y9F6RW\', \'B00004UEDQ\', \'B0039Y774Q\', \'B0006IIKRG\', \'B00JAGF9HE\', \'6305162026\', \'6305692572\', \'B001D7T460\', \'B0018QOIWG\', \'B002Y7ZELW\', \'B0045HCJ08\', \'0830907394\', \'B000LAZDPG\', \'B00A2H9QN8\', \'B001O5CLXY\', \'B000JBXXYK\', \'B003B3NGS6\', \'B0037SR3N4\', \'B00641Y2ZS\', \'0470903953\', \'0977977315\', \'B00049QQHI\', \'B000E6ESU8\', \'0470402741\', \'061565732X\', \'0615763146\', \'B000VZPTH8\', \'B003JO6OPO\', \'B00787BTEO\', \'B004R1Q7YQ\', \'B001GG6GKK\', \'B0015VQAZM\', \'1592854869\', \'B000QRIL08\', \'B000GQLA8O\', \'B000MPM3TE\', \'0979021804\', \'1608823407\', \'159285821X\', \'B00005Q4CS\', \'B0000549B1\', \'6305594333\', \'B00AFEXRME\', \'B004FN25AG\', \'0830906363\', \'0470402768\', \'1118414756\', \'B009SV4O2M\', \'1481106694\', \'1572306254\', \'B0013MOLPO\', \'B00009Y3QI\', \'B003NMOL2U\', \'B001AKBI8C\', \'0981708803\', \'1572306394\', \'B00B9LNPA6\', \'B005BYBZEK\', \'B004D7SBMU\', \'B00CQMADIO\', \'0470405511\', \'B00CHEHHT4\', \'B000ESUWY2\', \'0792838068\', \'B00AWE09Z0\', \'B00E4XZZEK\', \'0830914870\', \'B00GFZLEF4\', \'083090459X\', \'1402218443\', \'1893007170\', \'1893277046\', \'B005CKI7H6\', \'B0001LQL6K\', \'B000067S10\', \'0890425558\', \'B00114KYC8\', \'1466221224\', \'0943158508\', \'B00A7ID5BG\', \'0671765582\', \'B000B8IH10\', \'1568381395\'], \'buy_after_viewing\': [\'B003AC98V2\', \'B007VI5AQ8\', \'B000ERVK4Y\', \'B0007CEXYK\']}, \'salesRank\': {\'Movies & TV\': 1084845}, \'categories\': [[\'Movies & TV\', \'Movies\']]}\n', "{'asin': '0001048775', 'description': 'William Shakespeare is widely regarded as the greatest playwright the world has seen. He produced an astonishing amount of work; 37 plays, 154 sonnets, and 5 poems. He died on 23rd April 1616, aged 52, and was buried in the Holy Trinity Church, Stratford.', 'title': 'Measure for Measure: Complete & Unabridged', 'imUrl': 'http://ecx.images-amazon.com/images/I/5166EBHDQYL.jpg', 'salesRank': {'Books': 13243226}, 'categories': [['Books']]}\n", "{'asin': '0000031852', 'related': {'also_bought': ['B00JHONN1S', 'B002BZX8Z6', 'B00D2K1M3O', '0000031909', 'B00613WDTQ', 'B00D0WDS9A', 'B00D0GCI8S', '0000031895', 'B003AVKOP2', 'B003AVEU6G', 'B003IEDM9Q', 'B002R0FA24', 'B00D23MC6W', 'B00D2K0PA0', 'B00538F5OK', 'B00CEV86I6', 'B002R0FABA', 'B00D10CLVW', 'B003AVNY6I', 'B002GZGI4E', 'B001T9NUFS', 'B002R0F7FE', 'B00E1YRI4C', 'B008UBQZKU', 'B00D103F8U', 'B007R2RM8W'], 'also_viewed': ['B002BZX8Z6', 'B00JHONN1S', 'B008F0SU0Y', 'B00D23MC6W', 'B00AFDOPDA', 'B00E1YRI4C', 'B002GZGI4E', 'B003AVKOP2', 'B00D9C1WBM', 'B00CEV8366', 'B00CEUX0D8', 'B0079ME3KU', 'B00CEUWY8K', 'B004FOEEHC', '0000031895', 'B00BC4GY9Y', 'B003XRKA7A', 'B00K18LKX2', 'B00EM7KAG6', 'B00AMQ17JA', 'B00D9C32NI', 'B002C3Y6WG', 'B00JLL4L5Y', 'B003AVNY6I', 'B008UBQZKU', 'B00D0WDS9A', 'B00613WDTQ', 'B00538F5OK', 'B005C4Y4F6', 'B004LHZ1NY', 'B00CPHX76U', 'B00CEUWUZC', 'B00IJVASUE', 'B00GOR07RE', 'B00J2GTM0W', 'B00JHNSNSM', 'B003IEDM9Q', 'B00CYBU84G', 'B008VV8NSQ', 'B00CYBULSO', 'B00I2UHSZA', 'B005F50FXC', 'B007LCQI3S', 'B00DP68AVW', 'B009RXWNSI', 'B003AVEU6G', 'B00HSOJB9M', 'B00EHAGZNA', 'B0046W9T8C', 'B00E79VW6Q', 'B00D10CLVW', 'B00B0AVO54', 'B00E95LC8Q', 'B00GOR92SO', 'B007ZN5Y56', 'B00AL2569W', 'B00B608000', 'B008F0SMUC', 'B00BFXLZ8M'], 'bought_together': ['B002BZX8Z6']}, 'title': 'Girls Ballet Tutu Zebra Hot Pink', 'price': 3.17, 'salesRank': {'Toys & Games': 211836}, 'imUrl': 'http://ecx.images-amazon.com/images/I/51fAmVkTbyL._SY300_.jpg', 'brand': 'Coxlures', 'categories': [['Sports & Outdoors', 'Other Sports', 'Dance']], 'description': 'TUtu'}\n", '{\'asin\': \'0001048236\', \'categories\': [[\'Books\']], \'description\': ""One thing is certain, Sherlockians, put aside your Baring-GouldAnnotated, your Folio SocietyIllustrated-for the time being, the Oxford is the edition to curl up with on a winter\'s night"--The Chicago Tribune"An incomparable gift book; or, should you find it impossible to surrender up such treasures, the best of gifts to oneself"--USA Today"To the true Sherlockian, this will be a treasure; to otherwise diverted detective story fans, it is a rich lode for discovery"--Denver Post"The complete and authentic adventures of the legendary detective--expertly edited and annotated by a team of Holmes scholars....in a handsome, boxed set....A lovely gift"--The Christian Science Monitor"Here in nine volumes...are all the adventures of Holmes and Watson. Each book has an introduction, something new and fascinating for even the most devoted Holmesians plus a series of intelligent notes at the back of each volume."--Oxford Times"TheOxford Sherlock Holmes, a new edition of the stories, is a splendid piece of publishing. Nine compact volumes, beautifully produced, each with a stimulating introduction; clear type, accurate texts, a handy chronology, a helpful bibliography. And, most valuable of all, explanatory notes running to 50 pages or more per volume." --John Gross, writing inSunday Telegraph--This text refers to thePaperbackedition.", \'title\': \'The Sherlock Holmes Audio Collection\', \'price\': 9.26, \'salesRank\': {\'Books\': 8973864}, \'imUrl\': \'http://ecx.images-amazon.com/images/I/51DH145C5JL.jpg\', \'related\': {\'also_viewed\': [\'1442300191\', \'9626349786\', \'1602837155\', \'1598879162\', \'1400115159\', \'1478396202\', \'1408426250\', \'B007PM2A4A\', \'1609980603\'], \'buy_after_viewing\': [\'0312089457\']}}\n']
如何编写python脚本来加载pandas中的json文件?
最佳答案
我真的不明白这个问题。是内存消耗还是文件格式问题?使用 pd.read_json()
时发生了什么错误?
如果是文件格式问题(例如ValueError:尾随数据
):
基于 the source link you posted ,该文件似乎不是普通的 json 文件(请参阅“读取数据”部分)。这是一个以行分隔的 json 文件(每行都是一个 json 对象),一种广泛用于 json streaming 的格式。 .
要读取以行分隔的 json 文件,将 lines=True
传递给 pd.read_json()
:
pd.read_json('path/to/file',lines=True)
请注意,您需要 Pandas 版本。 0.19.0+ 使用这个。
关于python - 如何在特定模式的 pandas/python 中加载大于 10gb 的 json 文件,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47708128/
我是一名优秀的程序员,十分优秀!