diff --git a/docs.json b/docs.json index f3671828..55334535 100644 --- a/docs.json +++ b/docs.json @@ -1,7 +1,7 @@ { "$schema": "https://mintlify.com/docs.json", "banner": { - "content": "πŸš€ New: LFM2.5-2.6B β€” our on-device agentic model is now available! [Learn more β†’](/lfm/models/lfm25-2.6b)", + "content": "πŸš€ New: LFM2.5-VL-3B β€” our most capable vision-language model is now available! [Learn more β†’](/lfm/models/lfm25-vl-3b)", "dismissible": true }, "theme": "mint", @@ -73,7 +73,8 @@ }, "lfm/key-concepts/chat-template", "lfm/key-concepts/text-generation-and-prompting", - "lfm/key-concepts/tool-use" + "lfm/key-concepts/tool-use", + "lfm/key-concepts/vision-capabilities" ] }, { diff --git a/images/lfm/key-concepts/vision-capabilities/cell-13-output-5.png b/images/lfm/key-concepts/vision-capabilities/cell-13-output-5.png new file mode 100644 index 00000000..b47f79b6 Binary files /dev/null and b/images/lfm/key-concepts/vision-capabilities/cell-13-output-5.png differ diff --git a/images/lfm/key-concepts/vision-capabilities/cell-15-output-6.png b/images/lfm/key-concepts/vision-capabilities/cell-15-output-6.png new file mode 100644 index 00000000..ae662934 Binary files /dev/null and b/images/lfm/key-concepts/vision-capabilities/cell-15-output-6.png differ diff --git a/lfm/help/deprecations.mdx b/lfm/help/deprecations.mdx index 7d7dff26..cb6140b1 100644 --- a/lfm/help/deprecations.mdx +++ b/lfm/help/deprecations.mdx @@ -47,6 +47,14 @@ Deprecated models remain available for download on Hugging Face, but they are no βœ“ Vision Language Models + + LFM2-VL-3B + LFM2.5-VL-3B + βœ“ + βœ“ + βœ“ + βœ“ + LFM2-VL-1.6B LFM2.5-VL-1.6B diff --git a/lfm/key-concepts/vision-capabilities.mdx b/lfm/key-concepts/vision-capabilities.mdx new file mode 100644 index 00000000..15698347 --- /dev/null +++ b/lfm/key-concepts/vision-capabilities.mdx @@ -0,0 +1,556 @@ +--- +title: "Vision Capabilities" +description: "Use LFM2.5-VL models for image understanding, multi-image prompting, OCR, layout annotations, grounding, and vision-guided tool calling." +--- + + +LFM2.5-VL models' vision capabilities enable the model to analyze and understand images. +These models support common vision-language tasks such as describing images, answering questions about visual content, comparing multiple images, reading text, and localizing objects. LFM2.5-VL also supports tool calling, including examples where an image helps determine the tool arguments. + +The examples below show how to send images to LFM2.5-VL models. + +Install PyTorch, Transformers, and the image-processing packages used by the examples. `torchvision` and `Pillow` are required by the LFM2.5-VL image processor, and `opencv-python` is used to draw grounding and layout boxes. + +```python +# Install PyTorch and acceleration libraries +%pip install -q torch torchvision accelerate + +# Install image helper libraries +%pip install -q pillow opencv-python + +# Install Transformers +%pip install -q "transformers>=5.10.1" +``` + +Load the model with `AutoProcessor` and `AutoModelForImageTextToText`, matching the standard LFM2.5-VL documentation examples. Start with `LiquidAI/LFM2.5-VL-450M` or `LiquidAI/LFM2.5-VL-1.6B` for fast iteration. Use LFM2.5-VL-3B for the strongest grounding, layout parsing, and tool-calling examples. + +```python +from transformers import AutoProcessor, AutoModelForImageTextToText + +MODEL_ID = "LiquidAI/LFM2.5-VL-3B" # "LiquidAI/LFM2.5-VL-450M", "LiquidAI/LFM2.5-VL-1.6B", "LiquidAI/LFM2.5-VL-3B" + +processor = AutoProcessor.from_pretrained(MODEL_ID) +model = AutoModelForImageTextToText.from_pretrained( + MODEL_ID, + device_map="auto", + dtype="bfloat16", +) +``` + +## Single-image prompt + +Provide one image and a text question in the same user message. This pattern is useful for captioning, visual question answering, and scene understanding. + +```python +import torch +from IPython.display import display +from transformers.image_utils import load_image + +img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png" +input_image = load_image(img_url) +display(input_image) + +messages = [ + { + "role": "user", + "content": [ + {"type": "image", "image": input_image}, + {"type": "text", "text": "Describe this image in two concise sentences."}, + ], + } +] + +inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", +).to(model.device) + +with torch.inference_mode(): + outputs = model.generate( + **inputs, + do_sample=True, + temperature=0.2, + top_k=50, + repetition_penalty=1.0, + max_new_tokens=256, + ) + +output = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0] +print(output) +``` + +**Output** + +![Cell 7 output](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png) + +```text +Two cats are sleeping on a pink couch with two remote controls. +``` + +## Multi-image prompt +**Model support:** This capability is best supported by LFM2.5-VL-3B. + +You can include multiple images in a single prompt. Label each image in the prompt, such as `Media-1` and `Media-2`, to make cross-image references clearer. + +```python +import torch +from IPython.display import display +from transformers.image_utils import load_image + +img_urls = [ + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG", + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png", +] +input_images = [load_image(img_url) for img_url in img_urls] +for input_image in input_images: + display(input_image) + +messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Media-1\n"}, + {"type": "image", "image": input_images[0]}, + {"type": "text", "text": "\nMedia-2\n"}, + {"type": "image", "image": input_images[1]}, + {"type": "text", "text": "\nCaption Media-1 and Media-2 separately. Keep each caption to one sentence."}, + ], + } +] + +inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", +).to(model.device) + +with torch.inference_mode(): + outputs = model.generate( + **inputs, + do_sample=True, + temperature=0.2, + top_k=50, + repetition_penalty=1.0, + max_new_tokens=256, + ) + +output = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0] +print(output) +``` + +**Output** + +
+ Media-1 input + Media-2 input +
+ +```text +Media-1: A person is holding a collection of colorful M&M's with various designs on them. + +Media-2: Two cats are peacefully sleeping on a pink couch, each with a remote control nearby. +``` + +## OCR + +**Model support:** This capability is best supported by LFM2.5-VL-3B. + +LFM2.5-VL can read text in document images and, with LFM2.5-VL-3B, return structured layout annotations, which can be used to visualize the parsed layout regions. + +> **Image quality:** OCR and layout parsing work best when text is legible and the image is not blurry, rotated, or heavily compressed. For dense documents, crop to the relevant page or region when possible. + +```python +import torch +from IPython.display import display +from transformers.image_utils import load_image + +img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/enterprise/audit-logs.png" +input_image = load_image(img_url) +display(input_image) + +messages = [ + { + "role": "user", + "content": [ + {"type": "image", "image": input_image}, + { + "type": "text", + "text": "Read this audit log screenshot. Identify the page heading, table columns, visible rows, users, actions, timestamps, and other structured fields. Transcribe the visible text in reading order.", + }, + ], + } +] + +inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", +).to(model.device) + +with torch.inference_mode(): + outputs = model.generate( + **inputs, + do_sample=True, + temperature=0.1, + top_k=50, + repetition_penalty=1.05, + max_new_tokens=512, + ) + +generated_ids = outputs[:, inputs["input_ids"].shape[1] :] +output = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] +print(output) +``` + +**Output** + +![Cell 11 output](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/enterprise/audit-logs.png) + +```text +Audit log + +| Events log | +| --- | +| clboetticher org.update_join_settings | +| clboetticher enabled join requests with manual approval and set default role to write | +| United States, Austin Β· 136.62.181.0 Β· On Oct 15 | +| clboetticher repo.delete | +| clboetticher deleted a model: enterprise-explorers/test_model | +| United States, Austin Β· 136.62.181.0 Β· On Oct 10 | +| clboetticher repo.create | +| clboetticher created a public model: enterprise-explorers/test_model | +| United States, Austin Β· 136.62.181.0 Β· On Oct 10 | +| System billing.renew_subscription | +| System Automatic subscription renewal for hf/enterprise-free | +| Unknown Location Β· On Oct 7 | +| derek-thomas resource_group.add_users | +| derek-thomas added users to resource group Read access members: derek-thomas (read) | +| United Arab Emirates, Abu Dhabi Β· 217.164.173.0 Β· On Sep 17 | +| erikrigner resource_group.change_role | +| erikrigner changed role for user erikrigner in resource group Read access members from write to admin | +| Sweden, NorrkΓΆping Β· 2a02:1406:59:5d60:1dfd:0:0 Β· On Sep 16 | +``` + +## Document layout annotations + +For document understanding tasks, LFM2.5-VL-3B can return OCR with layout annotations. Each region contains a label, normalized bounding box, and content: + +```text +image_index=