Text-to-Image Search for Clothing with FashionCLIP and CLIP

2026-10-05

Python · PyTorch · Hugging Face Transformers · FAISS · Pandas · NumPy

What I did

I built a search engine for clothing photos. You type something like striped shirt or oversized blazer and it pulls the closest matches from 45,622 images in Fashionpedia, a research dataset where every garment is labeled with its category and a set of fine-grained attributes. The query text and the images get turned into vectors in the same space, and whichever images sit closest to the query come back.

I ran every search through two models: FashionCLIP, which was trained on fashion products, and OpenAI's general-purpose CLIP. Then I measured how often each one returned the right clothes. Single garments are the easy case, so I also wanted to see what happened with attributes like "striped" or "baggy," with outfits made of more than one piece, and with style.

Before I embedded a single image, I wrote down 20 test queries, what would count as a relevant result for each, the metrics, the pass thresholds, and six ways I expected the models to fail. That way I couldn't quietly adjust the test once I saw which model was ahead.

The queries fall into four tiers:

• Tier A, 5 queries: a single garment, like cardigan or shorts.
• Tier B, 7 queries: a garment plus one attribute, like striped shirt or baggy jeans.
• Tier C, 4 queries: combinations, like blazer over a white t-shirt or long coat with a scarf.
• Tier D, 4 queries: color, occasion, and style, like monochrome black outfit and goth outfit. Fashionpedia has no labels for any of these, so I predicted this tier would fail.

The methodology

Both models went through the same pipeline. FashionCLIP and OpenAI's CLIP ViT-B/32 were used zero-shot with identical preprocessing and embedding code, so the weights are the only difference between them. Queries were embedded as plain text with no "a photo of..." template. I decided that up front so I couldn't tune a template against the test set.

Search is exact. Each image is a 512-dimensional vector in a FAISS index that computes exact cosine similarity. At about 45,000 images there's no reason to use an approximate index, and doing so would mix its error in with the model quality I was trying to measure.

For Tiers A to C, relevance comes from the dataset's labels. Each query has a rule written over Fashionpedia's annotations. For striped shirt, a result counts if the image has a shirt with the stripe attribute. The main metric is capped recall@10: relevant images in the top 10, divided by 10, or by the total number of relevant images if the dataset has fewer than 10. I also report mean reciprocal rank (MRR), which tracks how high the first relevant result shows up.

Labels miss things, so I also judged the top 10 for every query and model by hand. That came to 391 image and query pairs, each scored 2 (clearly relevant), 1 (arguably relevant), or 0 (not relevant). I pooled the pairs across both models and shuffled them, so I never knew which model returned an image or at what rank. Strict precision@10 only counts the 2s.

The pass thresholds were set in advance on capped recall@10: 0.80 for Tier A, 0.50 for Tier B, and 0.30 for Tier C.

What I found

FashionCLIP beat CLIP on 8 of the 16 graded queries, lost on 3, and tied on 5. Sixteen queries is too few for a significance test, so I'm treating this as a description and not a statistical claim.

My hand judging put the models in the same order on every tier. Strict precision@10 for FashionCLIP and CLIP was 0.92 and 0.76 on Tier A, 0.83 and 0.51 on Tier B, 0.33 and 0.28 on Tier C, and 0.85 and 0.57 on Tier D.

The queries with the most misses:

• striped shirt: seven of CLIP's top 10 were striped t-shirts, which left it at 0.1 strict precision. FashionCLIP got 0.9.
• baggy jeans: CLIP mostly returned straight-leg and slim jeans (0.1). FashionCLIP did better with "baggy" at 0.7.
• oversized blazer: 0.0 for both. I got fitted blazers, long coats, and cardigans, and not one clearly oversized blazer.
• long coat with a scarf: both models found the scarves, but the coats were too short. Only 16 images in the whole dataset match this query.

The hand judging also showed how much Fashionpedia's labels miss. Of the judged images the labels called irrelevant, 28% (40 of 144) were clearly relevant. The worst case was sleeveless dress. Capped recall was 0.0 for both models, while strict precision was 1.0 and 0.9. The reason is that Fashionpedia attaches the "sleeveless" attribute to sleeves, and a sleeveless dress doesn't have one to attach it to. Of the 19 judged images the labels missed for this query, 18 were clearly sleeveless dresses.

I predicted Tier D would fail, and it mostly didn't. FashionCLIP scored 1.0 on monochrome black outfit, 1.0 on business casual office outfit, and 0.9 on the beach query. CLIP came in lower on all three at 0.8, 0.7, and 0.3. The one pure style query, goth outfit, landed at 0.5 for both.

A few caveats. I was the only judge, there are only 20 queries, and each one is graded on 10 images, so a 0.1 difference on a single query means one image.

Full analysis

The full pipeline, the pre-registered protocol, all 391 hand judgments with my reasoning, and the complete error analysis are on GitHub: Garment Search.