OV-SGT: Open Vocabulary Semantic Graph Transformer for Scene Graph Generation

Scene graph generation bridges visual perception and semantic understanding, but existing approaches face two challenges: closed vocabularies that limit real-world applicability and long-tail predicate distributions where common relationships dominate training data. We introduce the Open Vocabulary Semantic Graph Transformer (OV-SGT), which addresses both challenges through CLIP-aligned representation learning. Our method learns relationship embeddings within CLIP’s semantic space, enabling zero-shot generalization to unseen predicates. Key contributions include: (1) a node-edge fusion strategy preserving relationship directionality; (2) graph Laplacian eigenvector-based positional encoding capturing structural context; and (3) a multi-component loss combining contrastive, semantic, triplet, and focal objectives for zero-shot transfer while handling class imbalance. Experiments on Visual Genome demonstrate state-of-the-art performance, with significant gains on mean Recall@K metrics reflecting improved rare predicate recognition.