init commit

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,113 @@
+
+---
+pipeline_tag: text-classification
+tags:
+- vidore
+- reranker
+- qwen2_vl
+language:
+- multilingual
+base_model:
+- Qwen/Qwen2-VL-2B-Instruct
+inference: false
+license: cc-by-nc-4.0
+library_name: transformers
+---
+
+<br><br>
+
+<p align="center">
+<img src="https://huggingface.co/datasets/jinaai/documentation-images/resolve/main/logo.webp" alt="Jina AI: Your Search Foundation, Supercharged!" width="150px">
+</p>
+
+<p align="center">
+<b>Trained by <a href="https://jina.ai/"><b>Jina AI</b></a>.</b>
+</p>
+
+# jina-reranker-v3
+
+## Intended Usage & Model Info
+
+The **Jina Reranker v3** (`jina-reranker-v3`) is multi-lingual, and multi-modal model that has been fine-tuned for text and visual document reranking task, which is a crucial component in many information retrieval systems. It takes a query and a document pair as input and outputs a score indicating the relevance of the document to the query. The model is trained on a large dataset of query-document pairs and is capable of reranking documents in multiple languages with high accuracy.
+
+
+# Usage
+
+_This model repository is licenced for research and evaluation purposes under CC-BY-NC-4.0. For commercial usage, please refer to Jina AI's APIs, AWS Sagemaker or Azure Marketplace offerings. Please [contact us](https://jina.ai/contact-sales) for any further clarifications._
+1. The easiest way to use `jina-reranker-v3` is to call Jina AI's [Reranker API](https://jina.ai/reranker/).
+
+```bash
+curl https://api.jina.ai/v1/rerank \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer YOUR_API_KEY" \
+  -d '{
+  "model": "jina-reranker-v3",
+  "query": "Organic skincare products for sensitive skin",
+  "documents": [
+    {"text": "Organic skincare for sensitive skin with aloe vera and chamomile."},
+    {"text": "New makeup trends focus on bold colors and innovative techniques"},
+    {"text": "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille"},
+    {"text": "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken"},
+    {"text": "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla"},
+    {"text": "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras"},
+    {"text": "针对敏感肌专门设计的天然有机护肤产品"},
+    {"text": "新的化妆趋势注重鲜艳的颜色和创新的技巧"},
+    {"text": "敏感肌のために特別に設計された天然有機スキンケア製品"},
+    {"text": "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています"}
+
+  ],
+  "top_n": 3
+}'
+```
+
+2. You can also use the `transformers` library to interact with the model programmatically.
+
+Before you start, install the `transformers` libraries:
+
+```bash
+pip install transformers >= 4.47.3
+```
+
+And then:
+```python
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained(
+    'jinaai/jina-reranker-v3',
+    torch_dtype="auto",
+    trust_remote_code=True,
+)
+
+model.to('cuda') # or 'cpu' if no GPU is available
+model.eval()
+
+# Example query and documents
+query = "Organic skincare products for sensitive skin"
+documents = [
+    "Organic skincare for sensitive skin with aloe vera and chamomile.",
+    "New makeup trends focus on bold colors and innovative techniques",
+    "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille",
+    "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken",
+    "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla",
+    "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras",
+    "针对敏感肌专门设计的天然有机护肤产品",
+    "新的化妆趋势注重鲜艳的颜色和创新的技巧",
+    "敏感肌のために特別に設計された天然有機スキンケア製品",
+    "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています",
+]
+
+# construct sentence pairs
+sentence_pairs = [[query, doc] for doc in documents]
+
+scores = model.compute_score(sentence_pairs, max_length=1024)
+```
+
+The scores will be a list of floats, where each float represents the relevance score of the corresponding document to the query. Higher scores indicate higher relevance.
+For instance the returning scores in this case will be:
+```bash
+[0.8311430811882019, 0.09401018172502518,
+ 0.6334102749824524, 0.08269733935594559,
+ 0.7620701193809509, 0.09947021305561066,
+ 0.9263036847114563, 0.05834583938121796,
+ 0.8418256044387817, 0.11124119907617569]
+```

added_tokens.json

@@ -0,0 +1,16 @@
+{
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "jinaai/jina-reranker-v3",
+  "architectures": ["JinaVLForRanking"],
+  "auto_map": {
+    "AutoModel": "modeling.JinaVLForRanking"
+  },
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "image_token_id": 151655,
+  "initializer_range": 0.02,
+  "intermediate_size": 8960,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2_vl",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 2,
+  "rms_norm_eps": 1e-6,
+  "rope_scaling": {
+    "mrope_section": [16, 24, 24],
+    "rope_type": "default",
+    "type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.47.1",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "hidden_size": 1536,
+    "in_chans": 3,
+    "model_type": "qwen2_vl",
+    "spatial_patch_size": 14
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "vision_token_id": 151654,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "attn_implementation": "flash_attention_2",
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.01,
+  "top_k": 1,
+  "top_p": 0.001,
+  "transformers_version": "4.47.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d0a7b5fd0966512850481633159f357450dc738665870c9ac4f2b2da252f5e2
+size 4889523546

modeling.py

@@ -0,0 +1,221 @@
+import torch
+from torch import nn
+from typing import Optional, Tuple, List, Union, Any
+from transformers import Qwen2VLForConditionalGeneration
+import logging
+import warnings
+from PIL import Image
+from transformers.image_utils import load_image
+
+logger = logging.getLogger(__name__)
+
+
+def load_images(images, lazy_load: bool = True):
+    # Disable PIL DecompositionBomb threshold for reading large images.
+    pil_max_px = Image.MAX_IMAGE_PIXELS
+    Image.MAX_IMAGE_PIXELS = None
+
+    images_batch = []
+    for image in images:
+        if isinstance(image, Image.Image):
+            images_batch.append(image)
+        else:
+            pil_image = load_image(image)
+            if lazy_load:
+                images_batch.append(pil_image)
+            else:
+                # avoid Too many open files error
+                images_batch.append(pil_image.copy())
+                pil_image.close()
+    Image.MAX_IMAGE_PIXELS = pil_max_px
+
+    return images_batch
+
+
+def formatting_prompts_func(
+    query: str,
+    doc: str,
+    query_type: str = 'text',
+    doc_type: str = 'text',
+    prefix_str: str = '',
+) -> str:
+    """
+    Format prompts for different combinations of query and content types.
+
+    Args:
+        query: Query text or image path
+        doc: Content text or image path
+        query_type: Whether query is an image
+        doc_type: Whether content is an image
+        prefix_str: Optional prefix string to add
+    """
+    # Format query part
+    if query_type == 'image':
+        query_part = "**Query**:\n<|vision_start|><|image_pad|><|vision_end|>"
+    else:
+        query_part = f"**Query**:\n{query}"
+
+    # Format content part
+    if doc_type == 'image':
+        doc_part = "**Document**:\n<|vision_start|><|image_pad|><|vision_end|>"
+    else:
+        doc_part = f"**Document**:\n{doc}"
+
+    # Combine parts
+    prompt = doc_part + '\n' + query_part
+
+    # Add prefix if provided
+    if prefix_str:
+        prompt = prefix_str + '\n' + prompt
+
+    return prompt
+
+class JinaVLForRanking(Qwen2VLForConditionalGeneration):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.padding_side = "left"
+        self.num_labels = 1  # config.num_labels
+
+        # hack the lm_head to do nothing, since we only want the hidden states
+        self.lm_head = nn.Identity()
+
+        # copy the idea from `Qwen2ForRewardModel` to have a MLP layer to get the final score
+        self.score = nn.Sequential(
+            nn.Linear(config.hidden_size, config.hidden_size),
+            nn.ReLU(),
+            nn.Linear(config.hidden_size, self.num_labels),
+        )
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+        self.score_token_id = 100
+
+    def forward(self, *args, **kwargs) -> torch.Tensor:
+        # Delete output_hidden_states from kwargs
+        kwargs.pop("output_hidden_states", None)
+        kwargs.pop("use_cache", None)
+        assert kwargs.pop("labels", None) is None, "labels should not be passed to forward()"
+
+        outputs = super().forward(
+            *args,
+            use_cache=False,
+            output_hidden_states=True,
+            **kwargs,
+        )
+
+        # get the hidden states of the last layer
+        hidden_states = outputs.hidden_states[-1]
+
+        # IMPORTANT: the padding token must be on the left side
+        # get the hidden states of the last token and apply the linear layer
+        pooled_logits = self.score(hidden_states[:, -1])
+
+        return pooled_logits.squeeze(-1)
+
+    @torch.no_grad()
+    def compute_score(
+        self,
+        pairs: Union[List[Tuple[str, str]], Tuple[str, str]],
+        batch_size: int = 8,
+        max_length: int = 8192,
+        max_query_length: int = 512,
+        max_doc_length: Optional[int] = None,
+        query_type: str = 'text',
+        doc_type: str = 'text',
+        show_progress: bool = False,
+    ) -> List[float]:
+
+        if not hasattr(self, "_processor"):
+            from transformers import AutoProcessor
+            self._processor = AutoProcessor.from_pretrained(self.name_or_path, trust_remote_code=True)
+
+        assert isinstance(pairs, list)
+
+        if isinstance(pairs[0], str):
+            pairs = [pairs]
+
+        max_length = max_length or self.config.max_length
+
+        if max_doc_length is None:
+            max_doc_length = max(max_length - max_query_length, max_query_length)
+
+        if max_doc_length < max_query_length:
+            warnings.warn(
+                f"max_doc_length={max_doc_length} should be greater than max_query_length={max_query_length}"
+            )
+
+        assert (
+            max_doc_length + max_query_length <= max_length
+        ), f"max_doc_length ({max_doc_length}) + max_query_length ({max_query_length}) should be less than max_length ({max_length})"
+
+        max_length = max_length - 1
+
+        all_scores = []
+
+        device = next(self.parameters()).device
+
+        batch_iter = range(0, len(pairs), batch_size)
+        if show_progress:
+            from tqdm import trange
+
+            batch_iter = trange(0, len(pairs), batch_size, desc="Computing scores")
+
+        for start_index in batch_iter:
+            mini_batch = pairs[start_index : start_index + batch_size]
+
+            batch_inputs = []
+            for q, d in mini_batch:
+                # TEMP FIX: Truncate long documents
+                if doc_type == 'text':
+                    tokens = self._processor.tokenizer(d, truncation=True, max_length=max_doc_length)
+                    if len(tokens['input_ids']) >= max_doc_length:
+                        d = self._processor.tokenizer.decode(tokens['input_ids'])
+
+                batch_inputs.append(
+                    formatting_prompts_func(
+                        q, d, query_type=query_type, doc_type=doc_type
+                    )
+                )
+
+            batch_images = None
+            if doc_type == 'image':
+                batch_images = load_images([d for (q, d) in mini_batch])
+            elif query_type == 'image':
+                batch_images = load_images([q for (q, d) in mini_batch])
+
+            batch = self._processor(
+                text=batch_inputs,
+                images=batch_images,
+                return_tensors="pt",
+                padding=True,
+                truncation=True,
+                max_length=max_length,
+            )
+
+            # append the reward token to the input_ids and attention_mask
+            batch_size = batch["input_ids"].size(0)
+            batch["input_ids"] = torch.cat(
+                [
+                    batch["input_ids"],
+                    torch.full((batch_size, 1), self.score_token_id, device=batch["input_ids"].device),
+                ],
+                dim=1,
+            )
+            batch["attention_mask"] = torch.cat(
+                [
+                    batch["attention_mask"],
+                    torch.ones((batch_size, 1), device=batch["attention_mask"].device),
+                ],
+                dim=1,
+            )
+            # move the batch to the correct device
+            batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}
+
+            scores = self.forward(**batch).view(-1).cpu().float().numpy().tolist()
+            all_scores.extend(scores)
+
+        if len(all_scores) == 1:
+            return all_scores[0]
+        return all_scores

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Qwen2VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "max_pixels": 12845056,
+    "min_pixels": 3136
+  },
+  "temporal_patch_size": 2
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:091aa7594dc2fcfbfa06b9e3c22a5f0562ac14f30375c13af7309407a0e67b8a
+size 11420371

tokenizer_config.json

@@ -0,0 +1,145 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "processor_class": "Qwen2VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}