python - 如何在 Numba cuda 中对字符串数组执行内核函数？

Question

我有一个从文件中读取的字符串数组，我想将文件的每一行与特定字符串进行比较..文件太大(大约 200 MB 的行)

我已遵循本教程 https://nyu-cds.github.io/python-numba/05-cuda/但它没有准确显示如何处理字符串/字符数组。

import numpy as np
from numba import cuda



@cuda.jit
def my_kernel(io_array):

    tx = cuda.threadIdx.x

    ty = cuda.blockIdx.x

    bw = cuda.blockDim.x

    pos = tx + ty * bw
    if pos < io_array.size:  # Check array boundaries
        io_array[pos]   # i want here to compare each line of the string array to a specific line

def main():
    a = open("test.txt", 'r')  # open file in read mode

    print("the file contains:")
    data = country = np.array(a.read())


    # Set the number of threads in a block
    threadsperblock = 32

    # Calculate the number of thread blocks in the grid
    blockspergrid = (data.size + (threadsperblock - 1)) // threadsperblock

    # Now start the kernel
    my_kernel[blockspergrid, threadsperblock](data)


    # Print the result
    print(data)

if __name__ == '__main__':
        main()

我有两个问题。

首先:如何将我想要将文件的每一行与其进行比较的句子(字符串)发送到内核函数。 (在io_array中，不影响线程计算)

第二:它如何处理字符串数组？当我运行上面的代码时出现此错误

this error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at test2.py (18)

File "test2.py", line 18:
def my_kernel(io_array):
    <source elided>
    if pos < io_array.size:  # Check array boundaries
        io_array[pos]   # do the computation

P.S 我是 Cuda 新手，刚刚开始学习它。

Answer 1

首先:

data = country = np.array(a.read())

并没有像你想象的那样做。它不会产生一个可以像这样索引的 numpy 数组:

io_array[pos]

如果你不相信我，只需在普通的 python 代码中尝试一下:

print(data[0]) 

你会得到一个错误。如果您需要这方面的帮助，只需在 python 或 numpy 标签上提问即可。

因此我们需要一种不同的方法来从磁盘加载字符串数据。为了简单起见，我选择使用 numpy.fromfile() 。此方法要求文件中的所有行具有相同的宽度。我喜欢这个概念。如果您想处理不同长度的行，则需要描述更多信息。

如果我们以这种方式开始，我们可以将数据加载为字节数组，并使用它:

$ cat test.txt
the quick brown fox.............
jumped over the lazy dog........
repeatedly......................
$ cat t43.py
import numpy as np
from numba import cuda

@cuda.jit
def my_kernel(str_array, check_str, length, lines, result):

    col,line = cuda.grid(2)
    pos = (line*(length+1))+col
    if col < length and line < lines:  # Check array boundaries
        if str_array[pos] != check_str[col]:
            result[line] = 0

def main():
    a = np.fromfile("test.txt", dtype=np.byte)
    print("the file contains:")
    print(a)
    print("array length is:")
    print(a.shape[0])
    print("the check string is:")
    b = a[33:65]
    print(b)
    i = 0
    while a[i] != 10:
        i=i+1
    line_length = i
    print("line length is:")
    print(line_length)
    print("number of lines is:")
    line_count = a.shape[0]/(line_length+1)
    print(line_count)
    res = np.ones(line_count)
    # Set the number of threads in a block
    threadsperblock = (32,32)

    # Calculate the number of thread blocks in the grid
    blocks_x = (line_length/32)+1
    blocks_y = (line_count/32)+1
    blockspergrid = (blocks_x,blocks_y)
    # Now start the kernel
    my_kernel[blockspergrid, threadsperblock](a, b, line_length, line_count, res)


    # Print the result
    print("matching lines (match = 1):")
    print(res)

if __name__ == '__main__':
        main()
$ python t43.py
the file contains:
[116 104 101  32 113 117 105  99 107  32  98 114 111 119 110  32 102 111
 120  46  46  46  46  46  46  46  46  46  46  46  46  46  10 106 117 109
 112 101 100  32 111 118 101 114  32 116 104 101  32 108  97 122 121  32
 100 111 103  46  46  46  46  46  46  46  46  10 114 101 112 101  97 116
 101 100 108 121  46  46  46  46  46  46  46  46  46  46  46  46  46  46
  46  46  46  46  46  46  46  46  10]
array length is:
99
the check string is:
[106 117 109 112 101 100  32 111 118 101 114  32 116 104 101  32 108  97
 122 121  32 100 111 103  46  46  46  46  46  46  46  46]
line length is:
32
number of lines is:
3
matching lines (match = 1):
[ 0.  1.  0.]
$