r - 包含NA的因子列的唯一值=> “Hash table is full”错误

Question

我有一个具有5千7百万条记录和9列的data.table，当我尝试运行一些摘要统计信息时，其中之一导致出现问题。令人讨厌的列是3699级的一个因素，我从以下代码行收到错误:

    > unique(da$UPC)
    Error in unique.default(da$UPC): hash table is full

现在显然我将只使用: levels(da$UPC)，但我试图将每个组中存在的唯一值作为data.table组语句中多个j参数/表达式的一部分进行计数。

有趣的是 unique(da$UPC[1:1000000])可以按预期工作，但是 unique(da$UPC[1:10000000])不能正常工作。鉴于我的表有5700万条记录，这是一个问题。

我尝试将因子转换为字符，并且按以下方式工作没问题:

    da$UPC = as.character(levels(da$UPC))[da$UPC]
    unique(da$UPC)

这样做确实向我展示了一个额外的“级别”，即 NA。因此，由于我的数据在因子列中包含一些NA，因此唯一功能无法正常工作。我想知道开发人员是否意识到这是需要修复的东西？我在r-devel上找到了以下文章，该文章可能是相关的，但我不确定，并且没有提到 data.table。

Linked article: unique(1:3,nmax=1) freezes R!

    sessionInfo:

    R version 3.0.1 (2013-05-16)
    Platform: x86_64-unknown-linux-gnu (64-bit)

    locale:
     [1] LC_CTYPE=C                    LC_NUMERIC=C
     [3] LC_TIME=en_US.iso88591        LC_COLLATE=C
     [5] LC_MONETARY=en_US.iso88591    LC_MESSAGES=en_US.iso88591
     [7] LC_PAPER=C                    LC_NAME=C
     [9] LC_ADDRESS=C                  LC_TELEPHONE=C
     [11] LC_MEASUREMENT=en_US.iso88591 LC_IDENTIFICATION=C

    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base

    other attached packages:
    [1] plyr_1.8         data.table_1.8.8

Answer 1

此代码段应将您丢失的观察值放到一个常规级别，使其更易于管理。

# Need additional level to place missing into first
levels(da$UPC) <- c(levels(da$UPC), '(NA)')
da$UPC[is.na(da$UPC)] <- '(NA)'

听起来您最终想降低不频繁的水平来进行某种分析。我写了一个函数factorize()，相信可以为您提供帮助。它将不常见的级别存储到“其他”类别中。

这是链接，如果有帮助，请告诉我。

[factorize()] [1]
https://github.com/greenpat/R-Convenience/blob/master/factorize.R

(转载如下)

# This function takes a vector x and returns a factor representation of the same vector.
# The key advantage of factorize is that you can assign levels for infrequent categories,
# as well as empty and NA values. This makes it much easier to perform
# multidimensional/thematic analysis on your largest population subsets.
factorize <- function(
    x,  # vector to be transformed
    min_freq = .01,  # all levels < this % of records will be bucketed
    min_n = 1,  # all levels < this # of records will be bucketed
    NA_level = '(missing)',  # level created for NA values
    blank_level = '(blank)',  # level created for "" values
    infrequent_level = 'Other',  # level created for bucketing rare values
    infrequent_can_include_blank_and_NA = F,  # default NA and blank are not bucketed
    order = T,  # default to ordered
    reverse_order = F  # default to increasing order
) {
    if (class(x) != 'factor'){
        x <- as.factor(x)
    }
    # suspect this is faster than reassigning new factor object
    levels(x) <- c(levels(x), NA_level, infrequent_level, blank_level)

    # Swap out the NA and blank categories
    x[is.na(x)] <- NA_level
    x[x == ''] <- blank_level

    # Going to use this table to reorder
    f_tb <- table(x, useNA = 'always')

    # Which levels will be bucketed?
    infreq_set <- c(
        names(f_tb[f_tb < min_n]),
        names(f_tb[(f_tb/sum(f_tb)) < min_freq])
    )

    # If NA and/or blank were infrequent levels above, this prevents bucketing
    if(!infrequent_can_include_blank_and_NA){
        infreq_set <- infreq_set[!infreq_set %in% c(NA_level, blank_level)]
    }

    # Relabel all the infrequent choices
    x[x %in% infreq_set] <- infrequent_level

    # Return the reordered factor
    reorder(droplevels(x), rep(1-(2*reverse_order),length(x)), FUN = sum, order = order)
}