Article: Visual relationship extraction in images and a semantic interpretation with ontologies Journal: International Journal of Intelligent Information and Database Systems (IJIIDS) 2022 Vol.15 No.2 pp.223 - 247 Abstract: Nowadays, three challenges should be considered in order to build a strong model that is used to extract and semantically interpret the relationship between objects in images namely: long-tail problem, large intra-class divergence, and the semantic dependency or semantic gap. In order to overcome those challenges, we propose three main contributions: 1) an ontological semantic model to filter false negatives/positives using a statistical ranking module; 2) a combination of semantic ontological module and visual relationship module that both takes as input the results of the statistical ranking module and produces as output classification of < human - predicate - object >; 3) a semantic model for the visual relationship module that ranks each prediction of relation classes by transferring the spatial relationship onto a high dimension spatial feature. We use HCVRD that highlights two important practical problems, the long-tail distribution issue, and the zero-shot problem. The experimental results on the HCVRD dataset demonstrate the superior performance of the proposed approach. Inderscience Publishers - linking academia, business and industry through research

Title: Visual relationship extraction in images and a semantic interpretation with ontologies

Authors: Adel Zga; Brahim Nini

Addresses: Department of Mathematics and Computer Science, University of Larbi Ben M'hidi Oum El Bouaghi, Algeria; Department Computer Science and Information Technologies, University Kasdi Merbah Ouargla (UKMO), Ghardaia Road, BP. 511, 30000, Algeria ' Research Laboratory on Computer Science's, Department of Mathematics and Computer Science, Complex System (RELA(CS)² ), Oum El-Bouaghi University, Algeria

Abstract: Nowadays, three challenges should be considered in order to build a strong model that is used to extract and semantically interpret the relationship between objects in images namely: long-tail problem, large intra-class divergence, and the semantic dependency or semantic gap. In order to overcome those challenges, we propose three main contributions: 1) an ontological semantic model to filter false negatives/positives using a statistical ranking module; 2) a combination of semantic ontological module and visual relationship module that both takes as input the results of the statistical ranking module and produces as output classification of < human - predicate - object >; 3) a semantic model for the visual relationship module that ranks each prediction of relation classes by transferring the spatial relationship onto a high dimension spatial feature. We use HCVRD that highlights two important practical problems, the long-tail distribution issue, and the zero-shot problem. The experimental results on the HCVRD dataset demonstrate the superior performance of the proposed approach.

Keywords: deep learning; semantic gap; ontologies; human-object interaction; large intra-class divergence.

DOI: 10.1504/IJIIDS.2022.121931

International Journal of Intelligent Information and Database Systems, 2022 Vol.15 No.2, pp.223 - 247

Received: 28 Jan 2021
Accepted: 26 May 2021
Published online: 07 Apr 2022 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article

Title: Visual relationship extraction in images and a semantic interpretation with ontologies

Keep up-to-date