python - 如何在 Pandas 数据框中的所有列中获取唯一值

Question

我想列出 Pandas 数据框中所有列中的所有唯一值，并将它们存储在另一个数据框中。我已经尝试过这个但是它是按行附加的，我希望它是按列的。我怎么做？

raw_data = {'student_name': ['Miller', 'Miller', 'Ali', 'Miller'], 
        'test_score': [76, 75,74,76]}
      df2 = pd.DataFrame(raw_data, columns = ['student_name', 'test_score'])


      newDF = pd.DataFrame() 

      for column in df2.columns[0:]:
          dat = df2[column].drop_duplicates()
          df3 = pd.DataFrame(dat)
          newDF = newDF.append(df3)

print(newDF)


Expected Output:
student_name  test_score
Ali          74
Miller       75
             76

Answer 1

我想你可以使用 drop_duplicates .

如果想检查某些列并保留第一行，如果被欺骗:

newDF = df2.drop_duplicates('student_name')
print(newDF)
   student_name  test_score
0        Miller        76.0
1      Jacobson        88.0
2           Ali        84.0
3        Milner        67.0
4         Cooze        53.0
5         Jacon        96.0
6        Ryaner        64.0
7          Sone        91.0
8         Sloan        77.0
9         Piger        73.0
10        Riani        52.0

感谢 @cᴏʟᴅsᴘᴇᴇᴅ 的另一个解决方案:

df2[~df2.student_name.duplicated()]

---

但是如果想一起检查所有列是否有重复项，请保留第一行:

newDF = df2.drop_duplicates()
print(newDF)
   student_name  test_score
0        Miller        76.0
1      Jacobson        88.0
2           Ali        84.0
3        Milner        67.0
4         Cooze        53.0
5         Jacon        96.0
6        Ryaner        64.0
7          Sone        91.0
8         Sloan        77.0
9         Piger        73.0
10        Riani        52.0
11          Ali         NaN

按新样本编辑 - 删除重复项并按两列排序:

newDF = df2.drop_duplicates().sort_values(['student_name', 'test_score'])
print(newDF)
  student_name  test_score
2          Ali          74
1       Miller          75
0       Miller          76

EDIT1:如果想用 NaNs 替换第一列的 dupes:

newDF = df2.drop_duplicates().sort_values(['student_name', 'test_score'])
newDF['student_name'] = newDF['student_name'].mask(newDF['student_name'].duplicated())
print(newDF)
  student_name  test_score
2          Ali          74
1       Miller          75
0          NaN          76

EDIT2:更通用的解决方案:

newDF = df2.sort_values(df2.columns.tolist())
           .reset_index(drop=True)‌
           ​.apply(lambda x: x.drop_duplicates())