python - 如何加速从具有位置 (X, Y) 和强度的点创建图像？

Question

我有一个包含列的表格:[X, Y, intensity] 并希望从它们生成图像。这些表可能很大，现在这会花费太多时间。因此，我正在寻找优化代码的方法。

代码使用了如下所示的 pandas 数据帧:

import time
import cv2
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# Create demo version of the dataframes I use, which has similar characteristics as the real data
n = 27231221
df = pd.DataFrame({
    "X": np.random.uniform(low=0.0, high=142.0, size=n), 
    "Y": np.random.uniform(low=0.0, high=142.0, size=n), 
    "intensity": np.random.randint(low=0, high=60, size=n)
})
df.head()

    X           Y           intensity
0   63.643846   105.160795  11
1   123.693543  58.230852   55
2   2.289850    71.002206   42
3   132.666182  16.504936   7
4   99.317168   38.397257   56

代码本身如下:

# Resolution of the image, must stay like this
x_resolution=5e-2,
y_resolution=5e-2

start = time.time()

# Create bins with a certain resolution for the 2D histogram of the points
x_min = df["X"].min()
x_max = df["X"].max()
x_range = x_max - x_min
x_edges = np.linspace(start=x_min, stop=x_max,
                      num=np.ceil(x_range / x_resolution))
y_min = df["Y"].min()
y_max = df["Y"].max()
y_range = y_max - y_min
y_edges = np.linspace(start=y_min, stop=y_max,
                      num=np.ceil(y_range / y_resolution))
bins = (x_edges, y_edges)

# Timing
end = time.time()
print('Created bins in:', end - start)
start = end

# Create an histogram with the average bin intensity
im_n, _, _ = np.histogram2d(x=df["X"], y=df["Y"], bins=bins)  # Number of points in each bin

# Timing
end = time.time()
print('Created hist, part A, in:', end - start)
start = end

im_n += 0.001  # Prevent division by zero is not possible
im_int, x, y = np.histogram2d(x=df["X"], y=df["Y"], bins=bins, weights=df["intensity"]) # Total intensity in each bin
im_mean = (im_int / im_n) # Average intensity in each bin

# Timing
end = time.time()
print('Created hist, part B, in:', end - start)
start = end

# From average intensity to normalized values suitable for displaying 
# Note, there are outlier values which must not be take into consideration, hence the max_intensity
max_intensity = df["intensity"].quantile(0.98)
im_mean = np.clip(a=im_mean, a_min=0, a_max=max_intensity) / max_intensity

# Timing
end = time.time()
print('Created image in:', end - start)
start = end

# Kernel sizes for smoothing the image
close_kernel_size=3
# Connect pixels close to each other using a square in the image
kernel = np.ones((close_kernel_size, close_kernel_size))
im_mean = cv2.morphologyEx(im_mean, cv2.MORPH_CLOSE, kernel)

# Timing
end = time.time()
print('Smoothed image, part A, in:', end - start)
start = end

# Fill areas without high resolution pixels with lower resolution pixels
ellipse_kernel_size = 20
kernel = np.ones((ellipse_kernel_size, ellipse_kernel_size))
closing = cv2.morphologyEx(im_mean, cv2.MORPH_CLOSE, kernel)
idxs = im_mean == 0
im_mean[idxs] = closing[idxs]

# Timing
end = time.time()
print('Smoothed image, part B, in:', end - start)
start = end

# Show image
plt.figure(figsize=(3, 3))
plt.imshow(im_mean, cmap='gray')

输出看起来像这样:

Created bins in: 0.7478666305541992
Created hist, part A, in: 15.96267056465149
Created hist, part B, in: 16.237517833709717
Created image in: 0.426699161529541
Smoothed image, part A, in: 0.056333065032958984
Smoothed image, part B, in: 0.17376041412353516

<matplotlib.image.AxesImage at 0x7f6945f99ac8>

很明显，大部分改进都可以通过改进直方图创建来实现。但我不知道这是否可能或如何做..？

除了调整上面的代码，如果有其他(更快)的方法从这样的 DataFrame 到图像，我很想知道它们。

Answer 1

嗯，所以只需将您的直方图代码修改为:

im_n, _, _ = np.histogram2d(x=df["X"].values, y=df["Y"].values, bins=bins) 

改进时间

Created hist, part A, in: 22.977999925613403

为此:

Created hist, part A, in: 6.108999967575073

因此两次通话都提高了近 3 倍

如果我使用 to_numpy()，我会得到相似的时间:

np.histogram2d(x=df["X"].to_numpy(), y=df["Y"].to_numpy(), bins=bins)

Created hist, part A, in: 6.01200008392334