python - 在 Keras 中定义二进制掩码

Question

我有一个形状为 [X,Y,3] 的输入图像，并且有 2 个坐标 (x,y)。现在我想用这些坐标创建一个蒙版，然后将它与输入图像相乘。掩码应该是一个与图像大小相同的二元矩阵，坐标为 [x:x+p_size,y:y+p_size]，其他位置为零。

我的问题是如何在 Keras(tensorflow 后端)中定义掩码？

请注意，此操作发生在模型内(因此仅使用 numpy 无济于事)。

img = Input(shape=(32,32,3))
xy = Input(shape=(2)) # x and y coordinates for the mask
mask = ?
output = keras.layers.Multiply()([img, mask])

Answer 1

您可以使用 Lambda 完成所有操作实现自定义功能的层:

from keras.models import Model
from keras.layers import Input, Lambda
from keras import backend as K
import numpy as np

# Masking function factory
def mask_img(x_size, y_size=None):
    if y_size is None:
        y_size = x_size
    # Masking function
    def mask_func(tensors):
        img, xy = tensors
        img_shape = K.shape(img)
        # Make indexing arrays
        xx = K.arange(img_shape[1])
        yy = K.arange(img_shape[2])
        # Get coordinates
        xy = K.cast(xy, img_shape.dtype)
        x = xy[:, 0:1]
        y = xy[:, 1:2]
        # Make X and Y masks
        mask_x = (xx >= x) & (xx < x + x_size)
        mask_y = (yy >= y) & (yy < y + y_size)
        # Make full mask
        mask = K.expand_dims(mask_x, 2) & K.expand_dims(mask_y, 1)
        # Add channels dimension
        mask = K.expand_dims(mask, -1)
        # Multiply image and mask
        mask = K.cast(mask, img.dtype)
        return img * mask
    return mask_func

# Model
img = Input(shape=(10, 10, 3))  # Small size for test
xy = Input(shape=(2,))
output = Lambda(mask_img(3))([img, xy])
model = Model(inputs=[img, xy], outputs=output)

# Test
img_test = np.arange(100).reshape((1, 10, 10, 1)).repeat(3, axis=-1)
xy_test = np.array([[2, 4]])
output_test = model.predict(x=[img_test, xy_test])
print(output_test[0, :, :, 0])

输出:

[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0. 24. 25. 26.  0.  0.  0.]
 [ 0.  0.  0.  0. 34. 35. 36.  0.  0.  0.]
 [ 0.  0.  0.  0. 44. 45. 46.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]