I just deployed the Nous-Hermes-Llama2-70b parameter on a 2x Nvidia A100 GPU through the Hugging Face Inference endpoints.

我刚刚通过拥抱面孔推理端点在2x NVIDIA A100图形处理器上部署了nous-hermes-lama2-70b参数。

When I tried the following code, the response generations were incomplete sentences that were less than 1 line long.

当我尝试以下代码时，响应生成的是不完整的句子，长度不到1行。

import requests

API_URL = 'https://myendpoint.us-east-1.aws.endpoints.huggingface.cloud'
headers = {
  "Authorization": "Bearer mytoken1234",
  "Content-Type": "application/json"
}

def query(payload):
  response = requests.post(API_URL, headers=headers, json=payload)
  return response.json()
 
output = query({
  "inputs": "### Instruction:\r\nCome up with a joke about cats\r\n### Response:\r\n",
})

The output in this case was:

本例中的输出为：

"Why don't cats play poker in the jungle?

 Because "

As you see, the response stopped after 9 words.

如你所见，回复在9个单词后停止。

Do I need to add more headers to the request like temperature and max token length? How would I do that? What do I need to do to get normal, long responses?

我是否需要向请求添加更多标头，如温度和最大令牌长度？我该怎么做呢？我需要做什么才能得到正常的、长时间的回复？

Here is the model I'm using: https://huggingface.co/NousResearch/Nous-Hermes-Llama2-70b

这是我正在使用的模型：https://huggingface.co/NousResearch/Nous-Hermes-Llama2-70b

Added "max_new_tokens" => 256 as a parameter, fixed it.

新增“max_new_tokens”=>256作为参数，修复