Skip to content
BranchesTags
 
 

Repository files navigation

Contributors Forks Stargazers Issues MIT License

boofinity

boofinity is a community fork of michaelfeil/infinity (MIT): a high-throughput, low-latency REST API for serving text embeddings, rerankers, image-text retrieval models, audio-text retrieval models, and late-interaction vision-language retrieval models such as ColPali.

The Python package is renamed boofinity (version 0.3.0) and lives in libs/boofinity/. The fork targets serving BAAI/bge-m3 plus a reranker on CPU boxes (and optional CUDA), and adds CausalLM rerankers and multimodal/VLM backends on top of the upstream torch path. CUDA is supported but never required: the CPU path stays fully functional with no CUDA present.

  • Deploying? Read DEPLOY.md (validated CPU/Pascal/Blackwell install + serve + smoke test).
  • Working on the fork? Read CLAUDE.md (commands, architecture, known traps; local-only dev guide).
  • The fork is not published to PyPI and does not build docker images; install from source below.
  • Upstream marketing copy, news, docker images, and the upstream RestAPI client are kept as reference material in docs/upstream-readme.md, not repeated here.

Getting started

Install from source (this fork is not on PyPI)

Install torch first from the wheel index that matches the box, then the package with the [torch,server,logging] extras. See DEPLOY.md for CUDA/Pascal/Blackwell variants.

python3 -m venv .venv && .venv/bin/pip install -U pip
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install -e "libs/boofinity[torch,server,logging]"

With the venv active, you can run the CLI directly.

boofinity v2 --model-id BAAI/bge-small-en-v1.5

Check the v2 --help command to get a description for all parameters.

boofinity v2 --help

Advanced CLI usage

Launching multiple models at once

Since boofinity>=0.0.34, you can use cli v2 method to launch multiple models at the same time. Checkout boofinity v2 --help for all args and validation.

Multiple Model CLI Playbook:

    1. cli options can be repeated e.g. v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4. This will create two models model/id1 and model/id2
    1. or adapt the defaults by setting ENV Variables separated by ;: BOOFINITY_MODEL_ID="model/id1;model/id2;" && BOOFINITY_BATCH_SIZE="8;4;"
    1. single items are broadcasted to --model-id length, v2 --model-id model/id1 --model-id/id2 --batch-size 8 making both models have batch-size 8.
    1. Everything is broadcasted to the number of --model-id + API requests are routed to the --served-model-name/--model-id
Using environment variables instead of the cli All CLI arguments are also launchable via environment variables.

Environment variables start with BOOFINITY_{UPPER_CASE_SNAKE_CASE} and often match the --{lower-case-kebab-case} cli arguments.

The following two are equivalent:

  • CLI boofinity v2 --model-id BAAI/bge-base-en-v1.5
  • ENV-CLI: export BOOFINITY_MODEL_ID="BAAI/bge-base-en-v1.5" && boofinity v2

Multiple arguments can be used via ; syntax: BOOFINITY_MODEL_ID="model/id1;model/id2;"

API Key Supply an `--api-key secret123` via CLI or ENV BOOFINITY_API_KEY="secret123".
Chosing the fastest engine

With the command --engine torch the model must be compatible with https://github.com/UKPLab/sentence-transformers/ and AutoModel

With the command --engine optimum, there must be an onnx file. Models from https://huggingface.co/Xenova are recommended.

Telemetry opt-out

See which telemetry is collected: https://michaelfeil.eu/infinity/main/telemetry/

# Disable
export BOOFINITY_ANONYMOUS_USAGE_STATS="0"

What "ColPali" means

ColPali is a family of late-interaction vision-language retrieval models for searching document images with text queries. Instead of producing one pooled embedding per item, these models keep richer token or patch-level representations and score them later with MaxSim-style retrieval.

If you only need plain text embeddings or reranking, you can ignore ColPali entirely. In this fork it just means "document-image retrieval models such as ColPali and ColQwen are supported when the vision extras are installed."

Why Infinity

  • Deploy any model from HuggingFace: deploy any embedding, reranking, clip and sentence-transformer model from HuggingFace
  • Fast inference backends: The inference server is built on top of PyTorch and optimum (ONNX/TensorRT), using FlashAttention to get the most out of your NVIDIA CUDA, AMD ROCM, CPU, AWS INF2 or APPLE MPS accelerator. Infinity uses dynamic batching and tokenization dedicated in worker threads.
  • Multi-modal and multi-model: Mix-and-match multiple models. Infinity orchestrates them.
  • Tested implementation: Unit and end-to-end tested. Embeddings via infinity are correctly embedded. Lets API users create embeddings till infinity and beyond.
  • Easy to use: Built on FastAPI. Infinity CLI v2 allows launching of all arguments via Environment variable or argument. OpenAPI aligned to OpenAI's API specs. View the docs at https://indifferentketchup.github.io/boofinity/ on how to get started.

Supported Tasks and Models by Infinity

Infinity aims to be the inference server supporting most functionality for embeddings, reranking and related RAG tasks. The following Infinity tests 15+ architectures and all of the below cases in the Github CI. Click on the sections below to find tasks and validated example models.

Text Embeddings

Text embeddings measure the relatedness of text strings. Embeddings are used for search, clustering, recommendations. Think about a private deployed version of openai's text embeddings. https://platform.openai.com/docs/guides/embeddings

Tested embedding models:

Other models:

Reranking Given a query and a list of documents, Reranking indexes the documents from most to least semantically relevant to the query. Think like a locally deployed version of https://docs.cohere.com/reference/rerank

Tested reranking models:

Other reranking models:

Multi-modal and cross-modal - image and audio embeddings Specialized embedding models that allow for image<->text or image<->audio search. Typically, these models allow for text<->text, text<->other and other<->other search, with accuracy tradeoffs when going cross-modal.

Image<->text models can be used for e.g. photo-gallery search, where users can type in keywords to find photos, or use a photo to find related images. Audio<->text models are less popular, and can be e.g. used to find music songs based on a text description or related music songs.

Tested image<->text models:

Tested audio<->text models:

  • Clap Models from LAION
  • limited number open source organizations training these models
    • Note: The sampling rate of the audio data needs to match the model *

Not supported:

  • Plain vision models e.g. nomic-ai/nomic-embed-vision-v1.5
ColBert-style late-interaction Embeddings ColBert Embeddings don't perform any special Pooling methods, but return the raw **token embeddings**. The **token embeddings** are then to be scored with the MaxSim Metric in a VectorDB (Qdrant / Vespa)

For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding. Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing

Tested colbert models:

ColPali-style late-interaction Image<->Text Embeddings Similar to ColBERT, but for document images or screenshots matched against text queries instead of text-only retrieval. These models are useful for page-level PDF search, screenshot search, and OCR-free document retrieval.

For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding. Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing

Tested ColPali/ColQwen models:

Text classification A bert-style multi-label text classification. Classifies it into distinct categories.

Tested models:

Infinity usage via the Python API

Instead of the cli & RestAPI use infinity's interface via the Python API. This gives you most flexibility. The Python API builds on asyncio with its await/async features, to allow concurrent processing of requests. Arguments of the CLI are also available via Python.

Embeddings

import asyncio
from boofinity import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["Embed this is sentence via Infinity.", "Paris is in France."]
array = AsyncEngineArray.from_args([
  EngineArgs(model_name_or_path = "BAAI/bge-small-en-v1.5", engine="torch", embedding_dtype="float32", dtype="auto")
])

async def embed_text(engine: AsyncEmbeddingEngine): 
    async with engine: 
        embeddings, usage = await engine.embed(sentences=sentences)
    # or handle the async start / stop yourself.
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    await engine.astop()
asyncio.run(embed_text(array[0]))

Reranking

Reranking gives you a score for similarity between a query and multiple documents. Use it in conjunction with a VectorDB+Embeddings, or as standalone for small amount of documents. Please select a model from huggingface that is a AutoModelForSequenceClassification compatible model with one class classification.

import asyncio
from boofinity import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
query = "What is the python package boofinity?"
docs = ["This is a document not related to the python package boofinity, hence...", 
    "Paris is in France!",
    "boofinity is a package for sentence embeddings and rerankings using transformer models in Python!"]
array = AsyncEmbeddingEngine.from_args(
  [EngineArgs(model_name_or_path = "mixedbread-ai/mxbai-rerank-xsmall-v1", engine="torch")]
)

async def rerank(engine: AsyncEmbeddingEngine): 
    async with engine:
        ranking, usage = await engine.rerank(query=query, docs=docs)
        print(list(zip(ranking, docs)))
    # or handle the async start / stop yourself.
    await engine.astart()
    ranking, usage = await engine.rerank(query=query, docs=docs)
    await engine.astop()

asyncio.run(rerank(array[0]))

When using the CLI, use this command to launch rerankers:

boofinity v2 --model-id mixedbread-ai/mxbai-rerank-xsmall-v1

Image-Embeddings: CLIP models

CLIP models are able to encode images and text at the same time.

import asyncio
from boofinity import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["This is awesome.", "I am bored."]
images = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
engine_args = EngineArgs(
    model_name_or_path = "wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M", 
    engine="torch"
)
array = AsyncEngineArray.from_args([engine_args])

async def embed(engine: AsyncEmbeddingEngine): 
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    embeddings_image, _ = await engine.image_embed(images=images)
    await engine.astop()

asyncio.run(embed(array["wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M"]))

Audio-Embeddings: CLAP models

CLAP models are able to encode audio and text at the same time.

import asyncio
from boofinity import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
import requests
import soundfile as sf
import io

sentences = ["This is awesome.", "I am bored."]

url = "https://bigsoundbank.com/UPLOAD/wav/2380.wav"
raw_bytes = requests.get(url, stream=True).content

audios = [raw_bytes]
engine_args = EngineArgs(
    model_name_or_path = "laion/clap-htsat-unfused",
    dtype="float32", 
    engine="torch"

)
array = AsyncEngineArray.from_args([engine_args])

async def embed(engine: AsyncEmbeddingEngine): 
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    embedding_audios = await engine.audio_embed(audios=audios)
    await engine.astop()

asyncio.run(embed(array["laion/clap-htsat-unfused"]))

Text Classification

Use text classification with Infinity's classify feature, which allows for sentiment analysis, emotion detection, and more classification tasks.

import asyncio
from boofinity import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["This is awesome.", "I am bored."]
engine_args = EngineArgs(
    model_name_or_path = "SamLowe/roberta-base-go_emotions", 
    engine="torch", model_warmup=True)
array = AsyncEngineArray.from_args([engine_args])

async def classifier(engine: AsyncEmbeddingEngine): 
    async with engine:
        predictions, usage = await engine.classify(sentences=sentences)
    # or handle the async start / stop yourself.
    await engine.astart()
    predictions, usage = await engine.classify(sentences=sentences)
    await engine.astop()
asyncio.run(classifier(array["SamLowe/roberta-base-go_emotions"]))

Integrations:

Documentation

View the docs at https://indifferentketchup.github.io/boofinity on how to get started. After startup, the Swagger Ui will be available under {url}:{port}/docs, in this case http://localhost:7997/docs.

Contribute and Develop

This fork develops with plain pip venvs, not poetry (the upstream [all] extra and Makefile assume poetry). Build a venv with torch from the CPU wheel index first, then the package with test tooling:

python3 -m venv .venv-dev && .venv-dev/bin/pip install -U pip
.venv-dev/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv-dev/bin/pip install -e "libs/boofinity[torch,server,logging]"
.venv-dev/bin/pip install pytest pytest-mock httpx asgi_lifespan anyio trio

Run the tests, lint, and the embedding parity gate (required after any dependency or model-path change):

cd libs/boofinity && ../../.venv-dev/bin/python -m pytest tests/unit_test -x -q
ruff check libs/boofinity/boofinity
cd tests/parity && ../../.venv-dev/bin/python check_parity.py

See CLAUDE.md for the full command reference and architecture notes.

All contributions must be made in a way to be compatible with the MIT License of this repo.

Citation

@software{feil_2023_11630143,
  author       = {Feil, Michael},
  title        = {Infinity - To Embeddings and Beyond},
  month        = oct,
  year         = 2023,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.11630143},
  url          = {https://doi.org/10.5281/zenodo.11630143}
}

💚 Current contributors

About

Fork of Infinity for serving embeddings, rerankers, and multimodal retrieval, including ColPali document-image search.

michaelfeil.github.io/infinity/

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages