llmware

/

DeepSeek-R1-Distill-Qwen-7B-ov-int4

like 0

Follow

llmware 251

Using the model download from this repo around 2025-11-28 09:07.

When prompting the model, the opening <think> tag is not included, while the closing </think> tag is present. The expectation is that these tags delimit the reasoning part of the response.

$ curl http://localhost:8324/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-R1-Distill-Qwen-7B-ov-int4",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hi there"
      }
    ],
    "max_tokens": 500
  }'
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Okay, the user said \"Hi there.\" I should respond in a friendly and welcoming manner. Maybe say hello and ask how I can help them today. Keep it simple and open-ended so they feel comfortable to share what they need assistance with.\n</think>\n\nHello! How can I assist you today?","role":"assistant","tool_calls":[]}}],"created":1764676360,"model":"DeepSeek-R1-Distill-Qwen-7B-ov-int4","object":"chat.completion","usage":{"prompt_tokens":13,"completion_tokens":61,"total_tokens":74}}

Or in a more readable format:

Alright, the user greeted me with "Hi there." I need to respond in a friendly and helpful manner.

I should acknowledge their greeting and let them know I'm here to assist.

Maybe I'll ask them how I can help today to keep the conversation going.

Keeping it simple and open-ended should encourage them to share what they need.
</think>

Hello! How can I assist you today?

A issue reported to OpenVINO Model Server pointed to a bug with the template: https://github.com/openvinotoolkit/model_server/issues/3829

This apparently has been fixed upstream, and an update of this repo is necessary: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/commit/e2702c98808daaf40b07da72b9f9c327cd2f443c