openai-api - gpt-3.5-turbo 中的提示标记添加了 7 个标记

Question

我使用https://api.openai.com/v1/chat/completions api并且在每个响应中7个 token 被添加到prompt_tokens中，但是 token 计算在文档中是不同的(https://platform.openai.com/docs/guides/chat/introduction)

文档中的“content”:“pear”的“prompt_tokens”必须为 1，但 api 响应为 8

Request:
{
     "model": "gpt-3.5-turbo",
    "messages": [
        {
             "role": "user",
            "content": "pear"
         }
    ]
 }
Response:
{
     "id": "chatcmpl-6rQXcOtA0wOoYIbTJEvBrjL9CvN6i",
   "object": "chat.completion",
     "created": 1678191428,
    "model": "gpt-3.5-turbo-0301",
     "usage": {
         "prompt_tokens": 8,
        "completion_tokens": 122,
         "total_tokens": 130
     },
    "choices": [
         {
             "message": {
                "role": "assistant",
                "content": "\n\nA pear is a type of fruit that is commonly eaten in many parts of the world. Pears are usually sweet and juicy with a soft flesh that is rich in vitamins and minerals. It is a good source of dietary fiber, potassium, vitamin C, and copper. Pears come in a variety of shapes, sizes, and colors, with some having a green, yellow, or reddish-brown skin. They are often eaten raw as a snack, used in salads, baked into desserts, or cooked in savory dishes. Pears are also made into juice, jams, and other preserves."
             },
             "finish_reason": "stop",
             "index": 0
         }
     ]
 }

Answer 1

提示 token 不仅包含内容，还包含整个消息。这是:

{
 "role": "user",
 "content": "pear"
}