Generating clothing graphic captions for visually impaired users

Purchasing clothing online requires the identification of many different visual features such as colour, style, material and size. While this is simple task for most people, this is no so for Visually Impaired (VI) people, who are limited to relying on textual information, which is often lacking. Although current research has proposed image captioning models to enrich clothing descriptions, the models used thus far have been unable to generate descriptions of essential fine-grained elements such as graphics. We conducted a manual analysis of Ground Truth (GT) clothing image-text datasets and determined that several limitations disqualify them from consideration as gold standard captions. We collected clothing graphic images creating a new Clothing Graphic Dataset (CGD) and developed a novel zero-shot VI-friendly clothing graphic captioning model which adopts a Region Of Interest-based approach to generate a focused, detailed description of graphic elements, leveraging the power of Large Language Models to consider VI people's needs. The results of our evaluation revealed that, compared with BLIP-2 and GT captions, our model generates the most informative captions for VI people. The model's ability to produce detailed, relevant descriptions demonstrates its potential to improve online clothing accessibility for VI users without the need for costly and time-consuming pre-training data.