r - 大数据集清洗: How to fill in missing data based on multiple categories and searching by row order

Question

这是我的第一篇 StackOverflow 帖子，所以我希望它不会太难理解。

我有一个大型数据集(约 14,000 行)鸟类观察结果。这些数据是通过站在一个地方(点)并数出您在 3 分钟内看到的鸟类来收集的。在每个点计数中，新的鸟类观察结果会成为一个新行，因此存在许多重复的日期、时间、地点和点(地点内的特定位置)。同样，每个点的计算时间为 3 分钟。因此，如果您在第 1 分钟内看到ye低warbler(编码为 YEWA)，那么它将与该特定点计数(日期、站点)的 MINUTE=1 相关联、点和时间)。 ID=观察者缩写，Number=发现的鸟类数量(此处不一定重要)。

但是，如果看到NOBIRDS，则该特定分钟的数据集中会出现“NOBI”。因此，如果整个 3 分钟的点计数都有 NOBI，则它们将是具有相同日期、地点、点和时间的三行，并且三行中每一行的“BIRD”列中都有“NOBI”。

所以我有两个主要问题。 第一个是一些观察者在三分钟内输入一次“NOBI”，而不是三次(每分钟一次)。任何有“分钟”的地方 已留空(成为NA)，并且“BIRD”=“NOBI”，我需要添加三行数据，除“MINUTE”之外的所有列都具有相同的值，这应该是1、2 和 3 分别代表各行。

如果它看起来像这样:

   ID     DATE SITE POINT TIME MINUTE BIRD NUMBER
1  BS 5/9/2018  CW2  U125 7:51     NA NOBI     NA
2  BS 5/9/2018  CW1  D250 8:12      1 YEWA     2
3  BS 5/9/2018  CW1  D250 8:12      2 NOBI     NA
4  BS 5/9/2018  CW1  D250 8:12      3 LABU     1

它应该看起来像这样:

   ID     DATE SITE POINT TIME MINUTE BIRD NUMBER
1  BS 5/9/2018  CW2  U125 7:51      1 NOBI     NA
2  BS 5/9/2018  CW2  U125 7:51      2 NOBI     NA
3  BS 5/9/2018  CW2  U125 7:51      3 NOBI     NA
4  BS 5/9/2018  CW1  D250 8:12      1 YEWA     2
5  BS 5/9/2018  CW1  D250 8:12      2 NOBI     NA
6  BS 5/9/2018  CW1  D250 8:12      3 LABU     1

注意:如果您想将其中一些数据输入 R 控制台，我在本文末尾使用 dput 包含了一些数据，这应该比复制粘贴上面的内容更容易输入

我尝试重现具有多个条件的 if 语句失败了(基于: R multiple conditions in if statement & Ifelse in R with multiple categorical conditions )我尝试通过多种方式编写此内容，包括使用 dplyr 的管道，但请参阅下面的一些代码、注释和错误消息的示例。

>if(PC$BIRD == "NOBI" & PC$MINUTE==NA){PC$Fix<-TRUE}
 Error in if (PC$BIRD == "NOBI" & PC$MINUTE == NA) { : 
   missing value where TRUE/FALSE needed
 In addition: Warning message:
 In if (PC$BIRD == "NOBI" & PC$MINUTE == NA) { :
   the condition has length > 1 and only the first element will be used

## Then I need to do something like this:
>if(PC$Fix<-TRUE){duplicate(row where Fix==TRUE, times=2)} #I know this isn't 
    ### even close, but I want the row to be replicated two more times so 
    ### that there are 3 total rows witht he same values
    ### Fix indicates that a fix is needed in this example
# Then somehow I need to assign a 1 to PC$MINUTE for the first row (original row), 
# a 2 to the next row (with other values from other columns being the same), and a 3 
# to the last duplicated row (still other values from other columns being the same)

第二个问题对我来说似乎更困难，那就是按顺序或以某种方式按日期、地点、点和时间搜索数据集。分钟值应始终从 1... 到 2... 到 3，然后对于下一组日期、时间、地点和点返回到 1。也就是说，每个点计数的所有值应为 1:3。然而，一次计数可能在 MINUTE=1 内有多次目击，因此在 MINUTE=2 之前有 5 或 6(或 20)个 MINUTE=1。但是，同样，当没有 BIRDS (NOBI) 时，此数据集中的一些观察者只是留下一行，而不是为每分钟写一行 BIRD=“NOBI”。也就是说，如果数据集为:

   ID     DATE SITE POINT TIME MINUTE BIRD NUMBER
...
4  BS 5/9/2018  CW2  U125 7:54      1 AMRO      1
5  BS 5/9/2018  CW2  U125 7:54      1 SPTO      1
6  BS 5/9/2018  CW2  U125 7:57      1 AMRO      1
7  BS 5/9/2018  CW2  U125 7:57      1 SPTO      1
8  BS 5/9/2018  CW2  U125 7:57      1 AMCR      3
9  BS 5/9/2018  CW2  U125 7:57      2 SPTO      1
10 BS 5/9/2018  CW2  U125 7:57      2 HOWR      1
11 BS 5/9/2018  CW2  U125 7:57      3 UNBI      1

我们可以看到 7:57 点计数时间已完成(有 1:3 的 MINUTE 值)。然而，7:54 点计数时间在 MINUTE=1 处停止。意思是，我需要在下面再输入两行，它们具有所有相同的日期、地点、点、时间信息，但第一个添加的行为 MINUTE=2 且 BIRD="NOBI"，MINUTE=3 且 BIRD="NOBI "对于第二个添加的行。所以它应该看起来像这样:

   ID     DATE SITE POINT TIME MINUTE BIRD NUMBER
...
4  BS 5/9/2018  CW2  U125 7:54      1 AMRO      1
5  BS 5/9/2018  CW2  U125 7:54      1 SPTO      1
6  BS 5/9/2018  CW2  U125 7:54      2 NOBI      NA
7  BS 5/9/2018  CW2  U125 7:54      3 NOBI      NA
8  BS 5/9/2018  CW2  U125 7:57      1 AMRO      1
9  BS 5/9/2018  CW2  U125 7:57      1 SPTO      1
10 BS 5/9/2018  CW2  U125 7:57      1 AMCR      3
11 BS 5/9/2018  CW2  U125 7:57      2 SPTO      1
12 BS 5/9/2018  CW2  U125 7:57      2 HOWR      1
13 BS 5/9/2018  CW2  U125 7:57      3 UNBI      1

最后，我明白这是一个漫长而复杂的问题，我可能没有表达清楚。如果需要任何说明，请告诉我，我很乐意听到任何建议，即使它不能完全解决我的问题。预先感谢您!

<小时/>

只有当您想将我的数据样本输入 R 时，此行下面的所有内容才对您有用

<小时/>

要将我的数据输入 R 控制台，请复制并粘贴从“结构”函数到代码末尾的所有内容，以将其作为数据框输入 R 控制台，代码为:dataframe<-structure(list...请参阅Example of using dput()寻求帮助。

PC<-read.csv("PC.csv") ### ORIGINAL FILE
dput(PC)
structure(list(ID = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = "BS", class = "factor"), 
DATE = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = "5/9/2018", class = "factor"), 
SITE = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = "CW2", class = "factor"), 
POINT = structure(c(2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("M", "U125"), class = "factor"), 
TIME = structure(c(8L, 8L, 8L, 9L, 9L, 10L, 10L, 10L, 10L, 
10L, 10L, 11L, 1L, 1L, 1L, 2L, 2L, 2L, 3L, 3L, 4L, 4L, 4L, 
4L, 5L, 5L, 5L, 5L, 5L, 5L, 6L, 7L), .Label = c("6:48", "6:51", 
"6:54", "6:57", "7:12", "7:15", "7:18", "7:51", "7:54", "7:57", 
"8:00"), class = "factor"), MINUTE = c(1L, 2L, 3L, 1L, 1L, 
1L, 1L, 1L, 2L, 2L, 3L, 1L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 1L, 
1L, 1L, 2L, 3L, 1L, 1L, 1L, 2L, 3L, 3L, NA, NA), BIRD = structure(c(6L, 
6L, 6L, 2L, 7L, 2L, 7L, 1L, 7L, 5L, 8L, 8L, 6L, 6L, 6L, 6L, 
6L, 6L, 7L, 7L, 7L, 7L, 6L, 8L, 3L, 7L, 9L, 5L, 4L, 2L, 6L, 
6L), .Label = c("AMCR", "AMRO", "BRSP", "DUFL", "HOWR", "NOBI", 
"SPTO", "UNBI", "VESP"), class = "factor"), NUMBER = c(NA, 
NA, NA, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, NA, NA, NA, NA, 
NA, NA, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, NA, 
NA)), class = "data.frame", row.names = c(NA, -32L))


PCc<-read.csv("PC_Corrected.csv")  #### WHAT I NEED MY DATABASE TO LOOK LIKE
dput(PCc)
structure(list(ID = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L), .Label = "BS", class = "factor"), DATE = structure(c(1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = "5/9/2018", class = "factor"), 
SITE = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L), .Label = "CW2", class = "factor"), POINT = structure(c(2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("M", 
"U125"), class = "factor"), TIME = structure(c(8L, 8L, 8L, 
9L, 9L, 9L, 9L, 10L, 10L, 10L, 10L, 10L, 10L, 11L, 11L, 11L, 
1L, 1L, 1L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 4L, 4L, 4L, 4L, 5L, 
5L, 5L, 5L, 5L, 5L, 6L, 6L, 6L, 7L, 7L, 7L), .Label = c("6:48", 
"6:51", "6:54", "6:57", "7:12", "7:15", "7:18", "7:51", "7:54", 
"7:57", "8:00"), class = "factor"), MINUTE = c(1L, 2L, 3L, 
1L, 1L, 2L, 3L, 1L, 1L, 1L, 2L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 
3L, 1L, 2L, 3L, 1L, 1L, 2L, 3L, 1L, 1L, 2L, 3L, 1L, 1L, 1L, 
2L, 3L, 3L, 1L, 2L, 3L, 1L, 2L, 3L), BIRD = structure(c(6L, 
6L, 6L, 2L, 7L, 6L, 6L, 2L, 7L, 1L, 7L, 5L, 8L, 8L, 6L, 6L, 
6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 6L, 6L, 7L, 7L, 6L, 8L, 3L, 
7L, 9L, 5L, 4L, 2L, 6L, 6L, 6L, 6L, 6L, 6L), .Label = c("AMCR", 
"AMRO", "BRSP", "DUFL", "HOWR", "NOBI", "SPTO", "UNBI", "VESP"
), class = "factor"), NUMBER = c(NA, NA, NA, 1L, 1L, NA, 
NA, 1L, 1L, 1L, 1L, 1L, 1L, 1L, NA, NA, NA, NA, NA, NA, NA, 
NA, 1L, 1L, NA, NA, 1L, 1L, NA, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
NA, NA, NA, NA, NA, NA)), class = "data.frame", row.names = c(NA, 
-42L))

Answer 1

这里有一种使用 tidyverse 元包中的 dplyr 和 tidyr 来完成此操作的方法。

# Step one - identify missing rows.
#    For each DATE, SITE, POINT, TIME, count how many of each minute 
library(tidyverse)

# Convert factors to character to make later joining simpler, 
#   and fix missing ID's by assuming prior line should be used,
#   and make NOBI rows have a count of NA
PC_2_clean <- PC %>%
  mutate_if(is.factor, as.character) %>%
  fill(ID, .direction = "up") %>%
  mutate(NUMBER = if_else(BIRD == "NOBI", NA_integer_, NUMBER))


# Create a wide table with spots for each minute. Missing will
#   show up as NA's
# All the NA's here in the 1, 2, and 3 columns represent 
#   missing minutes that we should add.
PC_3_NA_find <- PC_2_clean %>%
  count(ID, DATE, SITE, POINT, TIME, MINUTE) %>%
  spread(MINUTE, n)

PC_3_NA_find
# A tibble: 11 x 9
# ID    DATE     SITE  POINT TIME    `1`   `2`   `3` `<NA>`
# <chr> <chr>    <chr> <chr> <chr> <int> <int> <int>  <int>
#   1 BS    5/9/2018 CW2   M     7:12      3     1     2     NA
# 2 BS    5/9/2018 CW2   M     7:15     NA    NA    NA      1
# 3 BS    5/9/2018 CW2   M     7:18     NA    NA    NA      1
# 4 BS    5/9/2018 CW2   U125  6:48      1     1     1     NA
# 5 BS    5/9/2018 CW2   U125  6:51      1     1     1     NA
# 6 BS    5/9/2018 CW2   U125  6:54      2    NA    NA     NA
# 7 BS    5/9/2018 CW2   U125  6:57      2     1     1     NA
# 8 BS    5/9/2018 CW2   U125  7:51      1     1     1     NA
# 9 BS    5/9/2018 CW2   U125  7:54      2    NA    NA     NA
# 10 BS    5/9/2018 CW2   U125  7:57      3     2     1     NA
# 11 BS    5/9/2018 CW2   U125  8:00      1    NA    NA     NA


# Take the NA minute entries and make the desired line for each
PC_4_rows_to_add <- PC_3_NA_find %>%
  gather(MINUTE, count, `1`:`3`) %>%
  filter(is.na(count)) %>%
  select(-count, -`<NA>`) %>%

  mutate(MINUTE = as.integer(MINUTE),
         BIRD = "NOBI",
         NUMBER = NA_integer_)


# Add these lines to the original,  remove the NA minute rows 
#   (these have been replaced with minute rows), and sort
PC_5_with_NOBIs <- PC_2_clean %>%
  bind_rows(PC_4_rows_to_add) %>%
  filter(MINUTE != "NA") %>%
  arrange(ID, DATE, SITE, POINT, TIME, MINUTE, BIRD)


# Check result
PC_5_with_NOBIs  %>%
  count(ID, DATE, SITE, POINT, TIME, MINUTE) %>%
  spread(MINUTE, n)

PC_5_with_NOBIs



# Now to confirm it matches your desired output.
#   Note, I convert to character to avoid mismatches between factors
PCc_char <- PCc %>%
  mutate_if(is.factor, as.character) %>%
  arrange(ID, DATE, SITE, POINT, TIME, MINUTE, BIRD)

identical(PC_5_with_NOBIs, PCc_char)
# [1] TRUE