update tokenizer to match latest release of command-nightly

.gitattributes

@@ -32,5 +32,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-*.md filter=lfs diff=lfs merge=lfs -text
 *.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,27 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b8d84577259b158f02ef147d8780ea9705088abf4edcdf8d020b919d83418e01
-size 701
+---
+license: apache-2.0
+---
+
+Cohere `command-nightly` tokenizer
+
+This is the tokenizer for the Cohere `command-nightly` [chat](https://docs.cohere.com/reference/chat) model.
+
+You can load it with the tokenizer library like this:
+
+```python
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("Cohere/command-nightly")
+text = "Hellö World, this is my input string!"
+enc = tokenizer.encode(text)
+print("Encoded input:")
+print(enc)
+
+inv_vocab = {v: k for k, v in tokenizer.get_vocab().items()}
+tokens = [inv_vocab[token_id] for token_id in enc.ids]
+print("Tokens:")
+print(tokens)
+
+number_of_tokens = len(enc.ids)
+print("Number of tokens:", number_of_tokens)
+```

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2a15f7b2a2f647bb4b288e1b2faeb7b7e7afe55acf014cb266ca44f3c29af9a
-size 3163464
+oid sha256:5e07ab012ff6144f924acb0c46e8462ade9119d6375a7712a170dc7620291493
+size 12777505