python - 如何有效计算 pandas 中列的线性组合？

Question

我需要进行以下计算:

priors['user_product'] = priors.eval('product_id + user_id*100000')

其中user_product是我想要生成的新列。然而，由于先验数据帧很大(准确地说有 3000000 行)，因此计算需要花费大量时间

Answer 1

如果你想要快速，你可以使用numpy或numexpr或普通的pandas

Pandas

priors['user_product'] = priors.product_id + 100000 * priors.user_id

numpy

priors['user_product'] = priors.product_id.values + 100000 * priors.user_id.values

numexpr

pid = priors.product_id.values
uid = priors.user_id.values
priors['user_product'] = numexpr.evaluate('pid + 100000 * uid')

时机

n = 3000000
priors = pd.DataFrame(dict(product_id=np.random.rand(n), user_id=np.random.rand(n)))

%timeit priors['user_product'] = priors.eval('product_id + 100000 * user_id')
%timeit priors['user_product'] = priors.product_id.values + 100000 * priors.user_id.values
%timeit priors['user_product'] = priors.product_id + 100000 * priors.user_id

10 loops, best of 3: 31.6 ms per loop
100 loops, best of 3: 17.6 ms per loop
100 loops, best of 3: 18.5 ms per loop

%%timeit
pid = priors.product_id.values
uid = priors.user_id.values
priors['user_product'] = numexpr.evaluate('pid + 100000 * uid')

100 loops, best of 3: 13.6 ms per loop