python - 语音到文本 - 将说话者标签映射到 JSON 响应中的相应转录本

Question

每隔一段时间就会出现一段 JSON 数据，它提出了一个挑战，可能需要数小时才能从中提取所需的信息。我从 Speech To Text API 引擎生成了以下 JSON 响应。

它显示每个说话者的转录本、每个单词的发音以及时间戳和说话者标签 speaker 0和 speaker 2在谈话中。

   {
    "results": [
        {
            "alternatives": [
                {
                    "timestamps": [
                        [
                            "the",
                            6.18,
                            6.63
                        ],
                        [
                            "weather",
                            6.63,
                            6.95
                        ],
                        [
                            "is",
                            6.95,
                            7.53
                        ],
                        [
                            "sunny",
                            7.73,
                            8.11
                        ],
                        [
                            "it's",
                            8.21,
                            8.5
                        ],
                        [
                            "time",
                            8.5,
                            8.66
                        ],
                        [
                            "to",
                            8.66,
                            8.81
                        ],
                        [
                            "sip",
                            8.81,
                            8.99
                        ],
                        [
                            "in",
                            8.99,
                            9.02
                        ],
                        [
                            "some",
                            9.02,
                            9.25
                        ],
                        [
                            "cold",
                            9.25,
                            9.32
                        ],
                        [
                            "beer",
                            9.32,
                            9.68
                        ]
                    ],
                    "confidence": 0.812,
                    "transcript": "the weather is sunny it's time to sip in some cold beer "
                }
            ],
            "final": "True"
        },
        {
            "alternatives": [
                {
                    "timestamps": [
                        [
                            "sure",
                            10.52,
                            10.88
                        ],
                        [
                            "that",
                            10.92,
                            11.19
                        ],
                        [
                            "sounds",
                            11.68,
                            11.82
                        ],
                        [
                            "like",
                            11.82,
                            12.11
                        ],
                        [
                            "a",
                            12.32,
                            12.96
                        ],
                        [
                            "plan",
                            12.99,
                            13.8
                        ]
                    ],
                    "confidence": 0.829,
                    "transcript": "sure that sounds like a plan"
                }
            ],
            "final": "True"
        }
    ],
    "result_index":0,
    "speaker_labels": [
        {
            "from": 6.18,
            "to": 6.63,
            "speaker": 0,
            "confidence": 0.475,
            "final": "False"
        },
        {
            "from": 6.63,
            "to": 6.95,
            "speaker": 0,
            "confidence": 0.475,
            "final": "False"
        },
        {
            "from": 6.95,
            "to": 7.53,
            "speaker": 0,
            "confidence": 0.475,
            "final": "False"
        },
        {
            "from": 7.73,
            "to": 8.11,
            "speaker": 0,
            "confidence": 0.499,
            "final": "False"
        },
        {
            "from": 8.21,
            "to": 8.5,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 8.5,
            "to": 8.66,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 8.66,
            "to": 8.81,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 8.81,
            "to": 8.99,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 8.99,
            "to": 9.02,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 9.02,
            "to": 9.25,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 9.25,
            "to": 9.32,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 9.32,
            "to": 9.68,
            "speaker": 0,
            "confidence": 0.472,
            "final": "False"
        },
        {
            "from": 10.52,
            "to": 10.88,
            "speaker": 2,
            "confidence": 0.441,
            "final": "False"
        },
        {
            "from": 10.92,
            "to": 11.19,
            "speaker": 2,
            "confidence": 0.364,
            "final": "False"
        },
        {
            "from": 11.68,
            "to": 11.82,
            "speaker": 2,
            "confidence": 0.372,
            "final": "False"
        },
        {
            "from": 11.82,
            "to": 12.11,
            "speaker": 2,
            "confidence": 0.372,
            "final": "False"
        },
        {
            "from": 12.32,
            "to": 12.96,
            "speaker": 2,
            "confidence": 0.383,
            "final": "False"
        },
        {
            "from": 12.99,
            "to": 13.8,
            "speaker": 2,
            "confidence": 0.428,
            "final": "False"
        }
    ]
}

请原谅缩进问题(如果有的话)，但 JSON 是有效的，我一直在尝试将每个抄本与其相应的演讲者标签进行映射。

我想要类似下面的东西。上面的 JSON 大约有 20,000 行，根据时间戳和单词发音提取说话者标签并将其与 transcript 放在一起是一场噩梦。 .

[
    {
        "transcript": "the weather is sunny it's time to sip in some cold beer ",
        "speaker" : 0
    },
    {
        "transcript": "sure that sounds like a plan",
        "speaker" : 2
    }

]  

到目前为止我尝试了什么:JSON 数据存储在名为 example.json 的文件中.我已经能够将每个单词及其对应的时间戳和说话者标签放入元组列表中(请参见下面的输出):

import json
# with open('C:\\Users\\%USERPROFILE%\\Desktop\\example.json', 'r') as f:
    # data = json.load(f)

l1 = []
l2 = []
l3 = []

for i in data['results']:
    for j in i['alternatives'][0]['timestamps']:
        l1.append(j)

for m in data['speaker_labels']:
     l2.append(m)

for q in l1:
    for n in l2:
        if q[1]==n['from']:
            l3.append((q[0],n['speaker'], q[1], q[2]))
print(l3)

这给出了输出:

 [('the', 0, 6.18, 6.63),
 ('weather', 0, 6.63, 6.95),
 ('is', 0, 6.95, 7.53),
 ('sunny', 0, 7.73, 8.11),
 ("it's", 0, 8.21, 8.5),
 ('time', 0, 8.5, 8.66),
 ('to', 0, 8.66, 8.81),
 ('sip', 0, 8.81, 8.99),
 ('in', 0, 8.99, 9.02),
 ('some', 0, 9.02, 9.25),
 ('cold', 0, 9.25, 9.32),
 ('beer', 0, 9.32, 9.68),
 ('sure', 2, 10.52, 10.88),
 ('that', 2, 10.92, 11.19),
 ('sounds', 2, 11.68, 11.82),
 ('like', 2, 11.82, 12.11),
 ('a', 2, 12.32, 12.96),
 ('plan', 2, 12.99, 13.8)]

但现在我不确定如何根据时间戳比较将单词关联在一起，并“存储”每组单词以再次形成带有说话人标签的文字记录。

我还成功地获得了列表中的文字记录，但现在如何从上面的列表中提取每个文字记录的说话人标签。扬声器标签 speaker 0和 speaker 2不幸的是，我希望每个词都适用 transcript相反。

for i in data['results']:
    l4.append(i['alternatives'][0]['transcript'])

这给出了输出:

["the weather is sunny it's time to sip in some cold beer ",'sure that sounds like a plan']

我已尽力解释问题，但我愿意接受任何反馈，并会在必要时进行更改。另外，我很确定有更好的方法来解决这个问题，而不是制作多个列表，非常感谢任何帮助。

对于更大的数据集，请参阅 pastebin .我希望这个数据集可以有助于性能基准测试。我可以在可用时或需要时提供更大的数据集。

当我处理大型 JSON 数据时，性能是一个重要因素，同样，在重叠转录中准确地实现说话人隔离是另一个要求。

Answer 1

使用 pandas，这是我刚才处理它的方法。

假设数据存储在名为 data 的字典中

import pandas as pd

labels = pd.DataFrame.from_records(data['speaker_labels'])

transcript_tstamps = pd.DataFrame.from_records(
    [t for r in data['results'] 
       for a in r['alternatives'] 
       for t in a['timestamps']], 
    columns=['word', 'from', 'to']
)
# this list comprehension more-efficiently de-nests the dictionary into
# records that can be used to create a DataFrame

df = labels.merge(transcript_tstamps)
# produces a dataframe of speakers to words based on timestamps from & to
# since I knew I wanted to merge on the from & to columns, 
# I named the columns thus when I created the transcript_tstamps data frame
# like this:
    confidence  final   from  speaker     to     word
0        0.475  False   6.18        0   6.63      the
1        0.475  False   6.63        0   6.95  weather
2        0.475  False   6.95        0   7.53       is
3        0.499  False   7.73        0   8.11    sunny
4        0.472  False   8.21        0   8.50     it's
5        0.472  False   8.50        0   8.66     time
6        0.472  False   8.66        0   8.81       to
7        0.472  False   8.81        0   8.99      sip
8        0.472  False   8.99        0   9.02       in
9        0.472  False   9.02        0   9.25     some
10       0.472  False   9.25        0   9.32     cold
11       0.472  False   9.32        0   9.68     beer
12       0.441  False  10.52        2  10.88     sure
13       0.364  False  10.92        2  11.19     that
14       0.372  False  11.68        2  11.82   sounds
15       0.372  False  11.82        2  12.11     like
16       0.383  False  12.32        2  12.96        a
17       0.428  False  12.99        2  13.80     plan

speaker & word data join后，需要将同一speaker的连续词组合在一起，推导出当前speaker。例如，如果扬声器数组看起来像 [2,2,2,2,0,0,0,2,2,2,0,0,0,0]，我们需要将前四个 2 在一起，然后是接下来的三个 0，然后是三个 2，然后是剩余的 0。

按 ['from', 'to'] 对数据进行排序，然后为此设置一个名为 current_speaker 的虚拟变量，如下所示:

df = df.sort_values(['from', 'to'])
df['current_speaker'] = (df.speaker.shift() != df.speaker).cumsum()

从这里开始，按 current_speaker 分组，将单词聚合成一个句子并转换为 json。有一些额外的重命名来修复输出 json 键

transcripts = df.groupby('current_speaker').agg({
   'word': lambda x: ' '.join(x),
   'speaker': min
}).rename(columns={'word': 'transcript'})
transcripts[['speaker', 'transcript']].to_json(orient='records')
# produces the following output (indentation added by me for legibility):
'[{"speaker":0,
  "transcript":"the weather is sunny it\'s time to sip in some cold beer"},    
 {"speaker":2,
  "transcript":"sure that sounds like a plan"}]'

要在转录开始/结束时添加额外的数据，您可以将 from/to 的最小值/最大值添加到 groupby

transcripts = df.groupby('current_speaker').agg({
   'word': lambda x: ' '.join(x),
   'speaker': min,
   'from': min,
   'to': max
}).rename(columns={'word': 'transcript'})

此外，(尽管这不适用于此示例数据集)您或许应该为每个时间片选择具有最高置信度的备选方案。