python-3.x - python如何获得多列下所有行对之间的差异

Question

我有两个 CSV 文件，并且两个文件都有多个列和行，我期待在两个文件的所有行之间获得差异。让我们假设文件之间的 Asset Tag Number 是否存在差异，然后以任何形式突出显示差异(可能是粗体值或适当的东西)，此外，我们有一个键是 Serial Number，它在两个文件中都是唯一的。因此，最好将行的差异放入 new.csv 文件中，并在删除相同行的同时突出显示差异。

仅供引用，我的两个文件都有超过 100 列..

我的实际数据列在两个 csv 文件中都如下所示。

Columns: [Asset Tag Number_a, Serial Number_a, System Name_a, Domain_a, System manufacturer_a, Model Name_a, System Type_a, Critical Level_a, Purpose Level 1_a, Purpose2_a, ShareIndv_a, Site_a, Building_a, Room_a, Rack_a, serverCostCenter_a, User ID   BU Grp Mgr_a, OS Name_a, OS Version_a, OS Type_a, Service Pack_a, Notification Group_a, Off The Network_a, First Name_a, Last Name_a, Manager Name_a, Status_a, BU Cost Center_a, BU CC Description_a, Organization Name_a, Higher Level BU_a, Business Contact_a, Description_a, Asset Type_a, System Type SW_a, Server _a, Host ID(Unix)_a, IP Address_a, MAC Address_a, Installed RAM_a, Disk Capacity_a, Installed Disk_a, Server Status _a, High Level Status_a, Lifecycle Status_a, EndOfLifeDate_a, Last Audit_a, AltVersion_a, BIOS Vendor_a, BIOS Version_a, BIOS Release Date_a, SMBIOS Enabled_a, SMBios Version_a, Region_a, Currency_a, Acquisition Cost USD_a, Net Book Value USD_a, CPU Type_a, CPU Speed_a, Acquisition Date_a, Age_a, DateModified_a, Altiris Exception_a, Inventory Owner_a, Last Logon User_a, Inventory Owner Last Logon User_a, Client Date_a, Reporting Status_a, Contact Status_a, Comments_a, Exception Reason_a, DNR_a, Asset Tag Number_b, Serial Number_b, System Name_b, Domain_b, System manufacturer_b, Model Name_b, System Type_b, Critical Level_b, Purpose Level 1_b, Purpose2_b, ShareIndv_b, Site_b, Building_b, Room_b, Rack_b, serverCostCenter_b, User ID   BU Grp Mgr_b, OS Name_b, OS Version_b, OS Type_b, Service Pack_b, Notification Group_b, Off The Network_b, First Name_b, Last Name_b, Manager Name_b, Status_b, BU Cost Center_b, ...]
Index: []

作为 Pandas 新手学习者，我应用了很少的代码方法，但似乎不太合适，因此寻求慷慨的帮助和建议。

1)第一个代码尝试..

#!/grid/common/pkgs/python/v3.6.1/bin/python3
import pandas as pd

A = pd.read_csv('a.csv', index_col=0)
B = pd.read_csv('b.csv', index_col=0)

C = pd.merge(left=A,right=B, how='outer', left_index=True, right_index=True, suffixes=['_a', '_b'])

not_in_a = C.drop( A.index )
not_in_b = C.drop( B.index )

not_in_a.to_csv('not_in_a.csv')
not_in_b.to_csv('not_in_b.csv')

2) 尝试了另一种代码，但输出的宽度太大，难以阅读，而此代码段应删除重复项，并仅打印有差异的那个..

from __future__ import print_function
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
import csv
import pandas as pd


##### Python pandas, widen output display to see more columns. ####
pd.set_option('display.height', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('expand_frame_repr', True)

a = pd.read_csv('a.csv')
b = pd.read_csv('b.csv')
c = pd.concat([a,b], axis=0)

c.drop_duplicates(keep='first', inplace=True)
c.reset_index(drop=True, inplace=True)
print(c)

我做了一些谷歌搜索，发现了一些关于这个主题的堆栈溢出讨论。但是，该线程中有一些不错的解决方案，但我认为没有任何解决方案可以满足我的要求，因此我在这里发布。

3)另一个与python集一起使用的代码，它部分工作..

#!/grid/common/pkgs/python/v3.6.1/bin/python3
import os
orig = open('aa.csv','r')
new = open('bb.csv','r')
bigb = set(new) - set(orig)
print(bigb)
# Write to output file
with open('different.csv', 'w') as file_out:
    for line in bigb:
        file_out.write(line)
    orig.close()
    new.close()
    file_out.close()

我有以下两个示例文件供引用，它们看起来与我的数据相似，我们可以将 Serial Number 作为输出逻辑和代码的关键。

下面是我的两个 csv 文件 file1.csv & file2.csv

File1:

wrkStaId                     Asset Tag Number  Serial Number System Name
                                                              mac-ymatsuok2
                                                              PC-ABNER-W10
                                                              PC-ADAMLIN-W10                                                                                              
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}    1234     ser123         sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}    3456     ser124         10210277      
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}    456      ser345         A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}    4485     ser900         dgs1sj

Sample File2:

    wrkStaId                Asset Tag Number Serial Number  System Name
                                                            mac-ymatsuok2
                                                            PC-Karn-W10
                                                            PC-ADAMLIN-W10
                                                            PC-ADRIANA-W10
   {ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}   1234 ser123     sfreder
   {8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}   3456 ser124     10210277
   {E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}   1709 ser345     A313819
   {445EC096-A70C-47D1-91FF-0DFE747F762A}   4485 ser900     dgs1sj

Desired Result: How do you want the difference represented, as these are non-numeric values. Do you want to print both rows in case they differ into a new file, and drop them if they are the same?

答案: Yes
想要的输出..

文件 1 中的差异不在文件 2 中

wrkStaId                     Asset Tag Number  Serial Number System Name
                                                              PC-ABNER-W10                                                                                                
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}    456      ser345         A313819

文件 1 中没有的文件 2 中的差异

    wrkStaId                Asset Tag Number Serial Number  System Name
                                                            PC-Karn-W10
                                                            PC-ADRIANA-W10
   {E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}   1709 ser345     A313819

非常感谢@w-m，但是我仍然希望从 SO 的专家那里透露更多的想法。

Answer 1

您的数据似乎包含两部分:一个 System Name 列表，然后是一个行表。由于结构完全不同，我建议您将数据拆分为 System Name s 和完整行的列表，并分别处理它们。

首先提取 System Name 列表:

l1 = df1[df1.wrkStaId == ""].System_Name
l2 = df2[df2.wrkStaId == ""].System_Name

您可以使用 Python 集差异代码获取差异:

>>> set(l1).difference(set(l2))
{'PC-ABNER-W10'}
>>> set(l2).difference(set(l1))
{'PC-ADRIANA-W10', 'PC-Karn-W10'}

现在删除空的 wrkStaId 条目:

df1 = df1[df1.wrkStaId != ""].set_index("wrkStaId")
df2 = df2[df1.wrkStaId != ""].set_index("wrkStaId")

其余数据现在包含以 wrkStaId 作为索引的完整行。

df1:

                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}            1234.0        ser123     sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}            3456.0        ser124    10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}             456.0        ser345     A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}            4485.0        ser900      dgs1sj

df2:

                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}            1234.0        ser123     sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}            3456.0        ser124    10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}            1709.0        ser345     A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}            4485.0        ser900      dgs1sj

您现在可以像 this 一样对 Pandas df 进行设置差异:

>>> df1[~df1.isin(df2).all(1)]
                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}             456.0        ser345     A313819

>>> df2[~df2.isin(df1).all(1)]
                                            Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}            1709.0        ser345     A313819

您可能需要稍微调整一下代码才能得到您想要的内容，但我希望这能让您继续前进。