Upload folder using huggingface_hub

.gitattributes

@@ -1,3 +1,4 @@
+*.safetensors filter=lfs diff=lfs merge=lfs -text
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
@@ -22,8 +23,6 @@
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
 *.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
@@ -33,3 +32,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,111 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- lmms-lab/LLaVA-OneVision-Data
+- BAAI/Infinity-MM
+language:
+- en
+- zh
+base_model:
+- google/siglip2-so400m-patch16-512
+- Qwen/Qwen2-1.5B-Instruct
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+# Flash-VL-2B-Static
+[\[📜 Flash-VL Tech Report\]](https://www.arxiv.org/abs/2505.09498)
+
+![image/png](https://s3plus.meituan.net/automl-datasets/mlm/logo.jpg)
+
+## Introduction
+
+We are excited to introduce **Flash-VL**, a novel approach to optimizing Vision-Language Models (VLMs) for real-time applications, targeting ultra-low latency and high throughput without sacrificing accuracy. Leveraging advanced architectural enhancements and efficient computational strategies, Flash-VL 2B is designed to maximize throughput by reducing processing time while maintaining competitive performance across multiple vision-language benchmarks. Our approach includes tailored architectural choices, token compression mechanisms, data curation, training schemes, and a novel image processing technique called implicit semantic stitching that effectively balances computational load and model performance. Through extensive evaluations on 11 standard VLM benchmarks, we demonstrate that Flash-VL 2B achieves state-of-the-art results in both speed and accuracy, making it a promising solution for deployment in resource-constrained environments and large-scale real-time applications.
+
+
+### Environment Setup
+
+```bash
+pip install torch==2.1.2
+pip install transformers==4.50.0.dev0
+```
+
+
+### How to use it?
+
+```python
+import torch
+from PIL import Image
+import requests
+from io import BytesIO
+from transformers import AutoModel, AutoTokenizer, AutoProcessor
+
+model_path = "FlashVL/FlashVL-2B-Static"
+model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,trust_remote_code=True,device_map='cuda')
+model.tokenizer = AutoTokenizer.from_pretrained(model_path,device_map='cuda')
+model.im_trans = AutoProcessor.from_pretrained(model_path).image_processor
+
+# single-image single-round conversation (单图单轮对话)
+image_url ="https://s3plus.meituan.net/automl-datasets/mlm/0516.png"
+response = requests.get(image_url)
+image_data = BytesIO(response.content)
+pil_image = Image.open(image_data).convert('RGB')   
+messages = [{'role': 'user', 'content': "生成图中菜品的菜谱"}] # answer: EXTRA
+answer = model.chat(pil_image, messages, do_sample=False, max_new_tokens=256)
+print(answer)
+
+# single-image multi-round conversation (单图多轮对话)
+messages = [
+    {'role': 'user', 'content': '这是什么'},
+    {"role": "assistant", "content": '这是一道看起来像是银耳莲子汤的甜品。\
+     银耳是一种常见的食材，通常用于制作甜品和汤品，具有软糯的口感和清润的口感。莲 \
+     子是莲子的干燥部分，常用于中医和食疗中，具有补脾止泻的功效。图片中还可以看到 \
+     一些枸杞和核桃，枸杞富含维生素和抗氧化物质，核桃则提供丰富的蛋白质和健康脂肪。 \
+     整体来看，这道甜品不仅美味，还具有一定的营养价值。'},
+    {'role': 'user', 'content': '对图中菜品卡路里分析'}
+    ]
+answer = model.chat(pil_image, messages, do_sample=False, max_new_tokens=256)
+print(answer)
+
+# pure-text single-round conversation (纯文本对话）
+messages = [{'role': 'user', 'content': "who are you"}]
+answer = model.chat(None, messages, do_sample=False, max_new_tokens=256)
+print(answer)
+```
+
+### Evaluation
+
+| Benchmark | Qwen2-VL-2B | Aquila-VL-2B | InternVL2.5-2B | Flash-VL-2B<sub>s<sub> | Flash-VL-2B<sub>d<sub> | Flash-VL-2B<sub>d-ISS<sub> |
+| :-------------: | :-------------: | :-------------: | :-------------: |:-------------: |:-------------: |:-------------: |
+| MMMU<sub>val<sub>                | 41.9 | 44.4 | 41.8 | 43.6 | 42.9 | 42.9 |
+| MMBench<sup>en<sup>              | 74.9 | 78.6 | 74.7 | 78.4 | 78.4 | 79.1 |
+| MMBench<sup>cn<sup>              | 73.5 | 76.3 | 71.6 | 74.7 | 74.9 | 76.7 |
+| MMStar                           | 48.0 | 54.9 | 54.1 | 53.8 | 54.4 | 54.1 |
+| MathVista<sub>testmini<sub>      | 43.0 | 59.4 | 50.9 | 59.3 | 58.1 | 61.5 |
+| AI2D<sub>test<sub>               | 74.1 | 75.0 | 75.1 | 74.2 | 74.1 | 74.4 |
+| MMVet                            | 49.5 | 40.9 | 61.7 | 47.3 | 52.7 | 50.7 |
+| HallusionBench                   | 39.2 | 38.5 | 42.7 | 43.5 | 45.5 | 49.0 |
+| OCRBench                         |  794 |  773 | 800  | 764  | 831  | 843  |
+| MME                              | 1872 | 1813 | 2091 | 1715 | 1866 | 1850 |
+| SEEDBench                        | 71.5 | 78.9 | 73.2 | 73.6 | 73.6 | 74.5 |
+| Average                          | 60.2 | 62.6 | 63.6 | 62.4 | 64.0 | 64.8 |
+
+
+We use [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) to evaluate FlashVL-2B-Static. 
+
+
+
+## Citation
+If you find this project useful in your research, please consider citing:
+
+```BibTeX
+@misc{zhang2025flashvl2boptimizingvisionlanguage,
+      title={Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput}, 
+      author={Bo Zhang and Shuo Li and Runhe Tian and Yang Yang and Jixin Tang and Jinhao Zhou and Lin Ma},
+      year={2025},
+      eprint={2505.09498},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2505.09498}, 
+}
+```

adapters.py

@@ -0,0 +1,59 @@
+import os
+import math
+import torch
+from torch import nn
+from functools import partial
+import torch.nn.functional as F
+
+class Adapter_Template(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.gradient_checkpointing = False
+    
+    def freeze_module(self, module):
+        for p in module.parameters():
+            p.requires_grad = False
+
+    def forward(self, inputs, add_start_end=True):
+        input_ids, hidden_states, targets, attn_mask, loss_mask = inputs
+        image_features = self.forward_adapter_modules(hidden_states)
+        return (input_ids, image_features, targets, attn_mask, loss_mask)
+
+class AdapterSigLIP(Adapter_Template):
+    
+    def __init__(self, config):
+        super().__init__(config)
+        self.p0 = nn.Sequential(
+            nn.LayerNorm(config.vision_config.hidden_size*4),
+            nn.Linear(config.vision_config.hidden_size*4, config.intermediate_size),
+            nn.GELU(),
+            nn.Linear(config.intermediate_size, config.intermediate_size),
+            nn.GELU(),
+        )
+        self.proj = nn.Linear(config.intermediate_size, config.vision_config.proj_output_dim)
+
+    def freeze(self):
+        self.freeze_module(self.p0)
+        self.freeze_module(self.proj)
+
+    def pixel_shuffle(self, x, scale_factor=0.5):
+        n, w, h, c = x.size()
+        if w % 2 == 0 and h % 2 == 0:
+            # N, W, H, C --> N, W, H * scale, C // scale
+            x = x.reshape(n, w, int(h * scale_factor), int(c / scale_factor))
+            # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
+            x = x.permute(0, 2, 1, 3).contiguous()
+            # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
+            x = x.view(n, int(h * scale_factor), int(w * scale_factor),
+                    int(c / (scale_factor * scale_factor)))
+        return x
+
+    def forward_adapter_modules(self, hidden_states):
+        h = w = int(hidden_states.shape[1] ** 0.5)
+        hidden_states = hidden_states.reshape(hidden_states.shape[0], h, w, -1)
+        hidden_states = self.pixel_shuffle(hidden_states, scale_factor=0.5)
+        hidden_states = hidden_states.reshape(hidden_states.shape[0], -1, hidden_states.shape[-1])
+
+        hidden_states = self.proj(self.p0(hidden_states))
+
+        return hidden_states

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

config.json

@@ -0,0 +1,112 @@
+{
+  "_name_or_path": "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/lishuo/repo/fine_tuning_package/model",
+  "architectures": [
+    "FlashVLStatic"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_FlashVLStatic.FlashVLStaticConfig",
+    "AutoModel": "modeling_FlashVLStatic.FlashVLStatic"
+  },
+  "intermediate_size": 7168,
+  "image_token_num": 256,
+  "llm_config":{
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "eos_token_id": 151645,
+    "hidden_act": "silu",
+    "hidden_size": 1536,
+    "initializer_range": 0.02,
+    "intermediate_size": 8960,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 21,
+    "model_type": "qwen2",
+    "num_attention_heads": 12,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000.0,
+    "sliding_window": 32768,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.50.0.dev0",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 151936
+  },
+  "vision_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 512,
+    "intermediate_size": 4304,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "siglip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 16,
+    "prefix": null,
+    "problem_type": null,
+    "proj_output_dim": 1536,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false
+  }
+}

configuration_FlashVLStatic.py

@@ -0,0 +1,27 @@
+import transformers
+from transformers import (Qwen2Config, Qwen2ForCausalLM, SiglipVisionConfig,
+                          SiglipVisionModel, PretrainedConfig)
+from transformers import AutoConfig
+import copy
+
+class FlashVLStaticConfig(PretrainedConfig):
+    model_type = 'FlashVLStaticConfig'
+    is_composition = True
+
+    def __init__(
+            self,
+            vision_config=dict(model_type='siglip_vision_model'),
+            llm_config=dict(architectures=['Qwen2ForCausalLM']),
+            **kwargs
+        ):
+        super().__init__(**kwargs)
+        self.vision_config = SiglipVisionConfig(**vision_config)
+        self.llm_config = Qwen2Config(**llm_config)
+
+    def to_dict(self):
+        
+        output = copy.deepcopy(self.__dict__)
+        output['vision_config'] = self.vision_config.to_dict()
+        output['llm_config'] = self.llm_config.to_dict()
+
+        return output

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.50.0.dev0"
+}

mm_constants.py

@@ -0,0 +1,4 @@
+# Model Constants
+IGNORE_INDEX = -100
+IMAGE_TOKEN_INDEX = -200
+IMAGE_PAD_TOKEN_INDEX = -201

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8cd0df6fba5b9a8de26ca7b9544e4bb437fad1a1219f140af70754a020b2a5a5
+size 4602705280

modeling_FlashVLStatic.py

@@ -0,0 +1,369 @@
+import os
+import math
+import copy
+
+import torch
+import torch.nn.functional as F
+from torch.nn import CrossEntropyLoss
+
+from PIL import Image
+from functools import partial
+from typing import List, Optional, Tuple, Union, Dict
+from dataclasses import dataclass
+
+import transformers
+from transformers.modeling_outputs import ModelOutput
+from transformers.modeling_utils import PreTrainedModel
+from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, PretrainedConfig, Qwen2Config, SiglipVisionModel
+
+from .adapters import AdapterSigLIP
+from .mm_constants import IGNORE_INDEX, IMAGE_TOKEN_INDEX
+from .processing_FlashVL import tokenizer_image_token_qwen
+from .configuration_FlashVLStatic import FlashVLStaticConfig
+
+@dataclass
+class FlashVLStaticOutputWithPast(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+
+
+class FlashVLStatic(PreTrainedModel):
+    config_class = FlashVLStaticConfig
+    
+    def __init__(self, config):
+        super().__init__(config)
+        self.llm = AutoModelForCausalLM.from_config(config.llm_config, trust_remote_code=True)
+        self.vit = SiglipVisionModel(config.vision_config).vision_model
+        self.adp = AdapterSigLIP(config)
+        self.image_token_num = config.image_token_num
+        self.image_size = config.vision_config.image_size
+
+    def merge_text_image_tokens(self, inputs):
+        input_ids, image_features, targets, attn_mask, loss_mask = inputs
+        micro_batch_size, tokens_len = input_ids.shape
+        device = input_ids.device
+        
+        img_rows, img_cols = torch.where(input_ids == IMAGE_TOKEN_INDEX)
+        image_idxs = {i: [] for i in range(micro_batch_size)}
+        for row, col in zip(img_rows.tolist(), img_cols.tolist()):
+            image_idxs[row].append(col)
+        for row in range(micro_batch_size):
+            image_idxs[row] = sorted(image_idxs[row])
+
+        split_sizes = []
+        for row in range(micro_batch_size):
+            image_num = len(image_idxs[row])
+            if image_num == 0:
+                split_sizes.append(tokens_len)
+                continue
+
+            if image_idxs[row][0] != 0:
+                split_sizes.append(image_idxs[row][0])
+
+            for idx in range(image_num - 1):
+                split_sizes.append(self.image_token_num)
+                if image_idxs[row][idx + 1] > image_idxs[row][idx] + self.image_token_num:
+                    split_sizes.append(image_idxs[row][idx + 1] - (image_idxs[row][idx] + self.image_token_num))
+
+            if image_idxs[row][image_num - 1] + self.image_token_num >= tokens_len:
+                split_sizes.append(tokens_len - image_idxs[row][image_num - 1])
+            else:
+                split_sizes.append(self.image_token_num)
+                split_sizes.append(tokens_len - (image_idxs[row][image_num - 1] + self.image_token_num))
+
+        input_ids_noim = torch.where(input_ids < 0, 151643, input_ids)
+        input_ids_noim = input_ids_noim.view(-1)
+        input_embeds = self.llm.model.embed_tokens(input_ids_noim)
+        input_embeds_split = torch.split(input_embeds, split_sizes, dim=0)
+
+        vl_embeds_list = []
+        cur_language_idx = 0
+        cur_image_idx = 0
+        for row in range(micro_batch_size):
+            image_num = len(image_idxs[row])
+            if image_num == 0:
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+                vl_embeds_list.append(image_features[cur_image_idx][0:0])
+                cur_image_idx += 1
+                continue
+
+            if image_idxs[row][0] != 0:
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+
+            for idx in range(image_num - 1):
+                vl_embeds_list.append(image_features[cur_image_idx])
+                cur_language_idx += 1
+                cur_image_idx += 1
+
+                if image_idxs[row][idx + 1] > image_idxs[row][idx] + self.image_token_num:
+                    vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                    cur_language_idx += 1
+
+            if image_idxs[row][image_num - 1] + self.image_token_num >= tokens_len:
+                vl_embeds_list.append(image_features[cur_image_idx][0 : tokens_len - image_idxs[row][image_num - 1]])
+                cur_language_idx += 1
+                cur_image_idx += 1
+            else:
+                vl_embeds_list.append(image_features[cur_image_idx])
+                cur_language_idx += 1
+                cur_image_idx += 1
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+
+        vl_embeds = torch.cat(vl_embeds_list)
+        vl_embeds = vl_embeds.view(micro_batch_size, tokens_len, vl_embeds.shape[-1])
+        return (input_ids, vl_embeds, targets, attn_mask, loss_mask)
+    
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        pixel_values: torch.FloatTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        local_pos_batch: Optional[torch.LongTensor] = None,
+        image_idx_batch: Optional[torch.Tensor] = None,
+        loss_mask: Optional[torch.Tensor] = None,
+        use_cache: Optional[bool] = None,
+    ):
+        inputs = [input_ids, pixel_values, labels, attention_mask, loss_mask]
+        
+        if isinstance(inputs[1], list):
+            pixel_values = [p.bfloat16() for p in inputs[1]]
+        else:
+            pixel_values = inputs[1].bfloat16()
+        img_token = self.vit.forward(pixel_values)
+        
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+        
+        inputs = self.adp(inputs[:1]+[img_token]+inputs[2:])
+
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        outputs = self.llm.forward(
+            inputs_embeds = hidden_states,
+            attention_mask = attn_mask,
+            use_cache = use_cache)
+        
+        lm_logits = outputs.logits
+        
+        loss = None
+        if targets is not None:
+            labels = targets.to(lm_logits.device)
+            shift_logits = lm_logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+
+            loss_fct = CrossEntropyLoss(reduction='none')
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)
+            )
+
+            batch_size = labels.size(0)
+            loss_mask = loss_mask[:, 1:].to(loss.dtype)
+            loss = (loss.view(batch_size, -1) * loss_mask).sum() / loss_mask.sum()
+
+        return FlashVLStaticOutputWithPast(
+            loss=loss,
+            logits=lm_logits
+        )
+    
+    def get_input_embeddings(self):
+        return self.llm.get_input_embeddings()
+    
+    def generate(
+        self,
+        input_ids=None,
+        pixel_values=None,
+        attention_mask=None,
+        **kwargs
+        ):  
+        image = pixel_values
+        img_token = self.vit.forward(image.bfloat16())
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+        inputs = self.adp((
+                    input_ids.to(self.device),
+                    img_token,
+                    None, None, None))
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        keys_to_pop = ['loss_mask', 'labels','attention_mask']
+        kwargs = {k: v for k, v in kwargs.items() if k not in keys_to_pop}
+            
+        outputs = self.llm.generate(
+            inputs_embeds=hidden_states.bfloat16(),
+            max_new_tokens=2048,
+            do_sample=False,
+            **kwargs
+        )
+        
+        return outputs
+    
+    def chat(self, pil_image, messages, answer_prompt=None, do_sample=True, max_new_tokens=256):
+        data={}
+        data['img'] = pil_image
+        data['text_only'] = (pil_image is None)
+        data['messages'] = messages
+
+        sources = self.to_llava_format(data)
+        sources = [sources]
+        has_image = not sources[0]['text_only']
+
+        if has_image:
+            img_list = sources[0]['image']
+            if not isinstance(img_list, list):
+                img_list = [img_list]
+            image = torch.stack([torch.from_numpy(self.im_trans(i)['pixel_values'][0]) for i in img_list], dim=0)
+            
+        sources = copy.deepcopy([e["conversations"] for e in sources])
+        
+        data_dict = self.preprocess_qwen(
+                sources,
+                self.tokenizer,
+                has_image=has_image,
+                )
+        
+        input_ids_data = data_dict["input_ids"][0]
+        data_dict["input_ids"] = [ input_ids_data, ]
+        
+        if not has_image:
+            image = torch.zeros(1, 3, self.image_size, self.image_size)
+        data_dict = dict(tokens=data_dict["input_ids"][0],)
+
+        img_token = self.vit.forward(image.cuda().bfloat16())
+        
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+
+        inputs = self.adp((
+                    data_dict['tokens'].unsqueeze(0).to(self.device),
+                    img_token, 
+                    None, None, None))
+        
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        outputs = self.llm.generate(
+            inputs_embeds=hidden_states.bfloat16(),
+            return_dict_in_generate=False,
+            max_new_tokens=max_new_tokens,
+            do_sample=do_sample,
+            pad_token_id=False,
+        )
+        decoded = self.tokenizer.decode(outputs[0])
+
+        stop_words_ids = [self.llm.generation_config.bos_token_id,
+                          self.llm.generation_config.eos_token_id,
+                          self.tokenizer.convert_tokens_to_ids('<|im_start|>')]
+        stop_words = [self.tokenizer.decode(w) for w in stop_words_ids]
+
+        for stop_word in stop_words:
+            decoded = decoded.replace(stop_word, "").strip()
+
+        return decoded
+    
+    def preprocess_qwen(
+        self,
+        sources, 
+        tokenizer: transformers.PreTrainedTokenizer, 
+        has_image: bool = False, 
+        max_len=2048, 
+        system_message: str = "You are a helpful assistant.",) -> Dict:
+
+        roles = {"human": "user", "gpt": "assistant"}
+        tokenizer = copy.deepcopy(tokenizer)
+        
+        tokenizer.add_tokens(["<image>"], special_tokens=True)
+        image_token_index = tokenizer.convert_tokens_to_ids("<image>")
+        im_start, im_end = tokenizer.additional_special_tokens_ids[:2]
+        
+        unmask_tokens_idx =  [198, im_start, im_end]
+        nl_tokens = tokenizer("\n").input_ids
+
+        chat_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+        tokenizer.chat_template = chat_template
+        
+        input_ids, targets = [], []
+        for i, source in enumerate(sources):
+            if roles[source[0]["from"]] != roles["human"]:
+                source = source[1:]
+            input_id, target = [], []
+        
+            input_id += tokenizer.apply_chat_template([{"role" : "system", "content" : system_message}])
+            target += [IGNORE_INDEX] * len(input_id)
+            i=0
+            for conv in source:
+                try:
+                    role = conv["role"]
+                    content = conv["content"]
+                except:
+                    role = conv["from"]
+                    content = conv["value"]
+                role =  roles.get(role, role)
+                if i==len(source)-1:
+                    conv = [{"role" : role, "content" : content}]
+                    encode_id = tokenizer.apply_chat_template(conv,add_generation_prompt=True)
+                else:     
+                    conv = [{"role" : role, "content" : content}]
+                    encode_id = tokenizer.apply_chat_template(conv)
+                i=i+1
+
+                if image_token_index in encode_id:
+                    encode_id = tokenizer_image_token_qwen(encode_id, tokenizer, image_token_index, image_token_num=self.image_token_num)
+                input_id += encode_id 
+                if role in ["user", "system"]:
+                    target += [IGNORE_INDEX] * len(encode_id)
+                else:
+                    target += encode_id
+
+            assert len(input_id) == len(target), f"{len(input_id)} != {len(target)}"
+            for idx, encode_id in enumerate(input_id):
+                if encode_id in unmask_tokens_idx:
+                    target[idx] = encode_id
+                if encode_id == image_token_index:
+                    input_id[idx] = IMAGE_TOKEN_INDEX
+            input_ids.append(input_id)
+            targets.append(target)
+            
+        input_ids = torch.tensor(input_ids, dtype=torch.long)
+        targets = torch.tensor(targets, dtype=torch.long)
+
+        return dict(
+            input_ids=input_ids,
+            labels=targets,
+        )
+    
+    def to_llava_format(self, data):
+        img_pil = data['img']
+        messages = data['messages']
+        text_only = data['text_only']
+        is_video=False
+        if 'is_video' in data:
+            is_video=data['is_video']
+            
+        messages.append({'role': 'assistant', 'content': ''})
+        conversations = []
+        for i,m in enumerate(messages):
+            if m['role'] == 'user':
+                value = str(m['content']).replace('<image>', '')
+                if i == 0 and not text_only:
+                    value = '<image>\n' + value
+                
+                conversations.append({'from': 'human', 'value': value})
+            elif m['role'] == 'assistant':
+                conversations.append({'from': 'gpt', 'value': str(m['content']).replace('<image>', '')})
+            else:
+                raise ValueError(f"Wrong role in conversation. {m['role']}")
+
+        return {'image': img_pil,
+                'text_only': text_only,
+                'is_video':is_video,
+                'conversations': conversations}

preprocessor_config.json

@@ -0,0 +1,24 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "SiglipProcessor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 512,
+    "width": 512
+  }
+}

processing_FlashVL.py

@@ -0,0 +1,19 @@
+from .mm_constants import IMAGE_TOKEN_INDEX, IMAGE_PAD_TOKEN_INDEX
+
+def tokenizer_image_token_qwen(prompt, tokenizer, image_token_index, image_token_num=256):
+    prompt_chunks, tmp = [], []
+    for n in prompt:
+        if n == image_token_index:
+            prompt_chunks.append(tmp)
+            tmp = []
+        else:
+            tmp.append(n)
+    if tmp: prompt_chunks.append(tmp)
+
+    input_ids = []
+    for i, chunk in enumerate(prompt_chunks):
+        if i > 0:
+            input_ids.extend([IMAGE_TOKEN_INDEX] + [IMAGE_PAD_TOKEN_INDEX] * (image_token_num - 1))
+        input_ids.extend(chunk)
+
+    return input_ids

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}