Backblaze Labs
All projects
SampleSample · unsupportedUnsupported · no SLA

OpenCLIP Batch Embeddings

Batch-embed large image corpora with on-device OpenCLIP into a per-job FAISS vector index for text-to-image semantic search. Source images, .npy embedding shards, and the index all live on Backblaze B2 over the S3-compatible API, with no database and no external inference key.

Built for: Developers and ML engineers building semantic image search, near-duplicate removal, or dataset clustering who want Backblaze B2 as the durable store for the media, the embedding shards, and the vector index, with local key-free inference.

Get the codeCheck project fit

MIT licensed

Dashboard with corpus, vector, shard, and index metrics, a write-amplification table, a throughput chart, and a recent-jobs table.
The dashboard shows corpus, vector, shard, and index metrics, a write-amplification projection at scale, an embedding-throughput chart, and recent jobs.

What it's built with

Each piece of the stack, and the job it does in this project.

Next.js 16 + React 19
Dashboard, jobs, search, corpus, files, and upload UI
Tailwind CSS + shadcn/ui
Design tokens and reusable interface primitives
TanStack Query
Client-side data fetching and cache management
FastAPI + Pydantic
Typed, layered Python API and configuration validation
OpenCLIP (open_clip_torch + torch)
On-device image and text embedding, no API key
FAISS (faiss-cpu)
Per-job vector index build and top-k search
boto3 + Backblaze B2
S3-compatible object storage for images, shards, and indexes
pnpm workspaces
TypeScript and Python monorepo workflow

Core capabilities

What the project does out of the box, before you write any code of your own.

  • Embedding jobs you can run

    Create, edit, delete, and run batch embedding jobs, each with its own OpenCLIP model and float32 or float16 precision, locked once the job has run.

  • On-device batch CLIP pipeline

    Stream images from B2, encode them locally with OpenCLIP (CUDA then Apple MPS then CPU autodetect, no API key), and write .npy embedding shards back to the bucket.

  • Per-job FAISS vector index

    Build a FAISS index per job and upload it with its id map to indexes/<job>/ so any worker can reload the exact same vector space.

  • Semantic text-to-image search

    Embed a text query into the same OpenCLIP space, rank it against a job's index, and stream the top-k matching images straight from B2.

  • B2 as the sole store, no database

    Source images, embedding shards, the FAISS index, and job manifests all live in one B2 bucket over the S3-compatible API; there is no separate database.

  • Reusable file browser and upload

    A full-bucket file browser and a bulk upload flow land images under corpus/ so they are immediately ready to embed.

A closer look

More screens from the running project.

Embedding Jobs list showing each job's status, model, vector count, and run duration.
The Jobs page lists every embedding job with its model, precision, vector count, and run duration.
Job detail with configuration, run artifacts, and the B2 shard and index keys.
Job detail pairs one job's configuration with the run artifacts it wrote to B2: the .npy shards and the FAISS index key.
Semantic search results for a text query, each card scored by cosine similarity.
Semantic search embeds a text query into the same OpenCLIP space and ranks it against a job's FAISS index, streaming every image from B2.
Corpus gallery of the source images stored under the corpus prefix in B2.
The Corpus page is a gallery of the source images the pipeline embeds, under the corpus/ prefix in B2.

Choose it for the right job

OpenCLIP Batch Embeddings is a head start for a specific shape of project, not a supported, general-purpose product.

Use it when

  • You want to batch-embed an image corpus with on-device OpenCLIP and search it by text, with the create-job to run to search flow working end to end.
  • You need source images, .npy embedding shards, and the FAISS index all stored durably in a single B2 bucket with no database to operate.
  • You want to bulk-upload images and turn them straight into an embedding job from the UI.
  • You want a reproducible, key-free reference you can run from a clean clone, since the default LAION OpenCLIP weights are ungated.

Choose another path when

  • You need a managed vector database or hosted SaaS; this has no user accounts, authentication, tenant isolation, or billing.
  • You are indexing billions of vectors and need a tuned approximate index; this favors an exact FAISS IndexFlatIP at demo scale.
  • You want a one-click serverless deploy; torch and FAISS do not run on Vercel serverless, so you self-host the FastAPI service and own its operations, capacity, and index scaling.
  • You want interactive single-image product search rather than batch corpus embedding; the clip-visual-product-search sample fits that better.
Sample · unsupported

Project status and support

Report defects on the repository's GitHub Issues; this sample is maintained without a service-level agreement or guaranteed support.