r - 有效地找到独特的子集组(例如独特的购物篮)

Question

我有一个数据框，其中一列代表购物篮的索引。对于每个篮子，我有另一列标识该篮子中的项目。在数据集中找到唯一篮子的最有效方法是什么？

这是一个使用 dplyr 的示例:

outer_num <- 10000
tmp_df <-
    data.frame(basket_index = rep(1:(8*outer_num), each = 2),
               items_purchased = rep(rep(c(1, 1, 2, 2, 1, 1, 3, 3), 2), outer_num))

items_purchased_df <-
    data.frame(items_purchased = 1:3, 
               item_name = c("shampoo", "soap", "conditioner"))

tmp_df_2 <-
    tmp_df %>%
    inner_join(items_purchased_df) %>%
    select(basket_index, items_purchased = item_name) 

head(tmp_df_2, 16)
#    basket_index items_purchased
# 1             1         shampoo
# 2             1         shampoo
# 3             2            soap
# 4             2            soap
# 5             3         shampoo
# 6             3         shampoo
# 7             4     conditioner
# 8             4     conditioner
# 9             5         shampoo
# 10            5         shampoo
# 11            6            soap
# 12            6            soap
# 13            7         shampoo
# 14            7         shampoo
# 15            8     conditioner
# 16            8     conditioner

在这个例子中，我们看到只有三个独特的篮子，每个篮子有两个项目。通常，篮子中可能有不同数量的元素，可能有也可能没有重复的元素，在某些情况下，篮子中元素的出现顺序很重要。

以下函数产生可接受的输出:

tmp_fn <- function(tmp_df) {
    tmp_df %>%
        group_by(basket_index) %>%
        mutate(collapsed_purchases = paste0(items_purchased, collapse = ',')) %>%
        group_by(collapsed_purchases) %>%
        filter(basket_index == min(basket_index)) %>%
        ungroup
}

以便

tmp_fn(tmp_df_2)
#   basket_index items_purchased collapsed_purchases    
#           <int> <fct>           <chr>                  
# 1            1 shampoo         shampoo,shampoo        
# 2            1 shampoo         shampoo,shampoo        
# 3            2 soap            soap,soap              
# 4            2 soap            soap,soap              
# 5            4 conditioner     conditioner,conditioner
# 6            4 conditioner     conditioner,conditioner

这不是非常节省时间。将项目因子转换为整数(并假设这是一个瞬时过程!)将其速度提高了近两个数量级，但即使在这个小数据集上仍然需要半秒:

tmp_df_3 <-
    tmp_df_2 %>%
    mutate(items_purchased_old = items_purchased,
           items_purchased = as.integer(factor(items_purchased)))

microbenchmark::microbenchmark(tmp_fn(tmp_df_2), times = 10)
# Unit: seconds
#            expr     min       lq     mean   median       uq      max neval
# tmp_fn(tmp_df_2) 20.6301 20.93541 21.98261 22.24193 22.43473 23.77921    10

microbenchmark::microbenchmark(tmp_fn(tmp_df_3), times = 10)
# Unit: milliseconds
#       expr      min       lq     mean   median       uq      max neval
# tmp_fn(tmp_df_3) 348.3901 358.0814 507.7983 363.7639 387.2384 1566.903    10

Answer 1

更新:我的结果是 stringsAsFactors = F .没有它，与 OP 的 tmp_fn() 相比，没有显着的性能提升。功能。

据我所知，group_by + mutate和 group_by + filter很慢。这是一种避免这种情况的方法-

# for outer_num <- 10000
system.time(
  res <- tmp_df_2 %>%
    group_by(basket_index) %>%
    summarize(collapsed_purchases = paste0(items_purchased, collapse = ',')) %>%
    filter(!duplicated(collapsed_purchases)) 
    # summarize drops one (in this case, the only) grouping level
    # so filter is on ungrouped data which is good; also duplicated() is fast enough
)

# user  system elapsed 
# 4.35    0.00    4.41 

res
# A tibble: 3 x 2
#   basket_index collapsed_purchases    
#          <int> <chr>                  
# 1            1 shampoo,shampoo        
# 2            2 soap,soap              
# 3            4 conditioner,conditioner

# get desired result
tmp_df_2 %>% 
  inner_join(res, by = "basket_index")

#   basket_index items_purchased     collapsed_purchases
# 1            1         shampoo         shampoo,shampoo
# 2            1         shampoo         shampoo,shampoo
# 3            2            soap               soap,soap
# 4            2            soap               soap,soap
# 5            4     conditioner conditioner,conditioner
# 6            4     conditioner conditioner,conditioner

注:使用 data.table可能会提供更快的速度。