c# - 如何通过 websocket 将音频发送到 Nexmo Voice

Question

我正在尝试在 .Net Core 2 web api 中使用 websockets 实现 Nexmo 的语音 api。

这个 api 需要:

• 通过Nexmo 接收来自电话的音频
• 使用 Microsoft Cognitive Speech to text接口(interface)
• 将文本发送给机器人
• 使用 Microsoft Cognitive text to speech根据机器人的回复
• 通过他们的语音 api websocket 将语音发送回 nexmo

目前，我正在绕过机器人步骤，因为我首先尝试连接到 websocket。当尝试回声方法(将收到的音频发送回 websocket)时，它可以正常工作。但是，当我尝试将语音从 Microsoft 文本发送到语音时，通话结束了。

我没有找到任何实现不同于回声的文档。

TextToSpeech 和 SpeechToText 方法在 websocket 外部使用时按预期工作。

这是带有语音到文本的 websocket:

public static async Task Echo(HttpContext context, WebSocket webSocket)
    {
        var buffer = new byte[1024 * 4];
        WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
        while (!result.CloseStatus.HasValue)
        {
            while(!result.EndOfMessage)
            {
                result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
            }
            var text = SpeechToText.RecognizeSpeechFromBytesAsync(buffer).Result;
            Console.WriteLine(text);
        }
        await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
    }

这是带有文字转语音功能的 websocket:

public static async Task Echo(HttpContext context, WebSocket webSocket)
    {
        var buffer = new byte[1024 * 4];
        WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
        while (!result.CloseStatus.HasValue)
        {
            var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("Hello, this is a test", "en-US");
            await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, 0, ttsAudio.Length), WebSocketMessageType.Binary, true, CancellationToken.None);

            result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
        }
        await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
    }

---

2019 年 3 月 1 日更新

回复 Sam Machin 的评论我尝试将数组拆分为每个 640 字节的 block (我使用 16000khz 采样率)，但 nexmo 仍然挂断电话，我仍然听不到任何声音。

public static async Task NexmoTextToSpeech(HttpContext context, WebSocket webSocket)
    {
        var ttsAudio = await TextToSpeech.TransformTextToSpeechAsync("This is a test", "en-US");
        var buffer = new byte[1024 * 4];
        WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);

        while (!result.CloseStatus.HasValue)
        {
            await SendSpeech(context, webSocket, ttsAudio);
            result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
        }
        await webSocket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Closing Socket", CancellationToken.None);
    }

    private static async Task SendSpeech(HttpContext context, WebSocket webSocket, byte[] ttsAudio)
    {
        const int chunkSize = 640;
        var chunkCount = 1;
        var offset = 0;
        
        var lastFullChunck = ttsAudio.Length < (offset + chunkSize);
        try
        {
            while(!lastFullChunck)
            {
                await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, chunkSize), WebSocketMessageType.Binary, false, CancellationToken.None);
                offset = chunkSize * chunkCount;
                lastFullChunck = ttsAudio.Length < (offset + chunkSize);
                chunkCount++;
            }

            var lastMessageSize = ttsAudio.Length - offset;
            await webSocket.SendAsync(new ArraySegment<byte>(ttsAudio, offset, lastMessageSize), WebSocketMessageType.Binary, true, CancellationToken.None);
        }
        catch (Exception ex)
        {
        }
    }

这是有时会出现在日志中的异常:

System.Net.WebSockets.WebSocketException (0x80004005): The remoteparty closed the WebSocket connection without completing the closehandshake.

Answer 1

看起来你正在将整个音频剪辑写入 websocket，Nexmo 接口(interface)要求音频在 20ms 帧中每条消息一个，这意味着你需要将你的剪辑分成 320 或 640 字节(取决于on if you're using 8Khz or 16Khz) block 并将每个 block 写入套接字。如果您尝试将太大的文件写入套接字，它将如您所见关闭。

参见 https://developer.nexmo.com/voice/voice-api/guides/websockets#writing-audio-to-the-websocket了解详情。