Polo Sologub

Sketchy Collections is an AI-driven image search tool that makes it possible to explore three museum collections – The Met, Cooper Hewitt and Science Museum – through sketching and uploading pictures. My goal was to make a prototype for a web app that makes both cultural heritage and AI technology more visible to non-experts.

The AI system is based on OpenAI's CLIP model, which makes it possible to predict text descriptions for images with custom vocabulary. I used this to make an image retrieval system that is based on the official tags used by the museums (or taxonomy terms in the case of Science Museum). This means that instead of directly comparing the user's sketch against the museum object images, CLIP is used first to turn the images into probability distributions of the most likely tags. When the user uploads an image or a sketch, the top matches from each museum are shown well as the top 5 predicted tags. This makes it possible to compare the images and tags to make sense of what the AI recognises in the images and to understand how much the results of the tool are influenced by the structure of museums as reflected in the museum objects and terminology.

Visit the project repository: https://github.com/polosologub/sketchy-collections