openai-api - 如何计算整个ChatGPT对话的token？

Question

我正在编写一个与 ChatGPT 对话并将其输出为流的函数，但我发现 ChatGPT 似乎没有提供我使用流输出时使用的 token .

这是流式输出的功能:

/**
 * talk with ChatGPT
 * @param msg
 * @param callback
 * @example chatWithGPTWithSteaming([{role: "system", "content": "You are a helpful assistant."},{role: "system", content: "Hello world"}],(text)=>{console.log(text)})
 */
export async function chatWithGPTWithSteaming(msg: any,callback:Function) {
    const chatCompletion = await openai.createChatCompletion({
        model: 'gpt-3.5-turbo',
        messages: msg,
        stream: true,
    }, {responseType: "stream"});

    chatCompletion.data.on('data', data => {
        const lines = data.toString().split('\n').filter(line => line.trim() !== '');
        for (const line of lines) {
            const message = line.replace(/^data: /, '');
            if (message === '[DONE]') {
                console.log("text is end");
                console.log(chatCompletion);
                // callback(false);
                return; // Stream finished
            }
            try {
                const parsed = JSON.parse(message);
                const text = parsed.choices[0].delta.content;
                data += text;
                if (text) {
                    console.log(text);
                    callback(text);
                }
            } catch (error) {
                console.error('Could not JSON parse stream message', message, error);
            }
        }
    });
    console.log(chatCompletion);
}

这是data值的示例:

但是当我不使用流输出时:

export async function chatWithGPT(msg: any,a) {
    const completion = await openai.createChatCompletion({
        model: "gpt-3.5-turbo",
        messages: [
            {role: "system", "content": "You are a helpful assistant."},
            {role: "user", content: "Hello!"},
        ],
    });
    console.log(completion.data.choices[0].message);
}

此时，我可以从中获取usage.total token

那么在使用流输出时我应该如何获取token？

我注意到 Tokenizer OpenAI提供的可以使用，但是我使用Tokenizer计算出的token值与ChatGPT API返回的值不同。

当我使用此对话时，API 返回 prompt_token 为 19

{role: "system", "content": "You are a helpful assistant."},
{role: "user", content: "Hello!"},

但是Tokenizer提供的token是9。

任何帮助将不胜感激。

Answer 1

我从 this code of Python 得到了答案.

这是Javascript的代码，它需要的依赖是GPT-3-Encoder .

const { encode } = require('gpt-3-encoder');

function numTokensFromMessages(messages, model = "gpt-3.5-turbo-0613") {
    let tokens_per_message = 0;
    let tokens_per_name = 0;
    if (["gpt-3.5-turbo-0613", "gpt-3.5-turbo-16k-0613", "gpt-4-0314", "gpt-4-32k-0314", "gpt-4-0613", "gpt-4-32k-0613"].includes(model)) {
        tokens_per_message = 3;
        tokens_per_name = 1;
    } else if (model == "gpt-3.5-turbo-0301") {
        tokens_per_message = 4;
        tokens_per_name = -1;
    } else if (model.includes("gpt-3.5-turbo")) {
        console.log("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.");
        return numTokensFromMessages(messages, "gpt-3.5-turbo-0613");
    } else if (model.includes("gpt-4")) {
        console.log("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.");
        return numTokensFromMessages(messages, "gpt-4-0613");
    } else {
        throw new Error(`num_tokens_from_messages() is not implemented for model ${model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.`);
    }
    let num_tokens = 0;
    for (let i = 0; i < messages.length; i++) {
        let message = messages[i];
        num_tokens += tokens_per_message;
        for (let key in message) {
            let value = message[key];
            num_tokens += encode(value).length;
            if (key == "name") {
                num_tokens += tokens_per_name;
            }
        }
    }
    num_tokens += 3;
    return num_tokens;
}

// usage:
const testToken = numTokensFromMessages([
    { role: "system", "content": "You are a helpful assistant." },
    { role: "user", content: "Hello!" },
    { role: "assistant", content: "What can I help you with today?" },
    { role: "user", content: "I'd like to book a hotel in Berlin." },
]);

console.log(testToken);