qgallouedec HF Staff commited on
Commit
f419bf6
·
verified ·
1 Parent(s): 8be3368

Add a response_template for parsing assistant responses

Browse files

Adds a `response_template` to `tokenizer_config.json` so `tokenizer.parse_response(...)` turns raw generated text back into a message dict (transformers >= 5.13, https://github.com/huggingface/transformers/pull/45847):

```python
>>> tokenizer.parse_response("A cat sitting on a mat.<|END_OF_TURN_TOKEN|>", prefix=prompt)
{'role': 'assistant', 'content': 'A cat sitting on a mat.'}
```

The model emits plain content between `<|CHATBOT_TOKEN|>` and `<|END_OF_TURN_TOKEN|>` — no thinking or tool-call fields — so the template is a single `content` field.

Verified: round-trips `apply_chat_template` -> `parse_response` at the token level, and every truncated prefix of a response parses without error (streaming-safe). No other behavior changes.

Files changed (1) hide show
  1. tokenizer_config.json +12 -0
tokenizer_config.json CHANGED
@@ -325,6 +325,18 @@
325
  "pad_token": "<PAD>",
326
  "padding_side": "right",
327
  "processor_class": "CohereCompassProcessor",
 
 
 
 
 
 
 
 
 
 
 
 
328
  "sp_model_kwargs": {},
329
  "spaces_between_special_tokens": false,
330
  "tokenizer_class": "CohereTokenizer",
 
325
  "pad_token": "<PAD>",
326
  "padding_side": "right",
327
  "processor_class": "CohereCompassProcessor",
328
+ "response_template": {
329
+ "defaults": {
330
+ "role": "assistant"
331
+ },
332
+ "start_anchor": "<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>",
333
+ "fields": {
334
+ "content": {
335
+ "close_pattern": "<\\|END_OF_TURN_TOKEN\\|>\\s*",
336
+ "content": "text"
337
+ }
338
+ }
339
+ },
340
  "sp_model_kwargs": {},
341
  "spaces_between_special_tokens": false,
342
  "tokenizer_class": "CohereTokenizer",