19th International CODATA Conference
Category: Data Visualization
Caliph & Emir: Semantics in Multimedia Retrieval and Annotation
Mathias Lux (mlux@know-center at), Werner Klieber (wklieber@know-center at), Michael Granitzer (mgrani@know-center at)
Know-Center Graz, Austria
http://www.know-center.at
Applying Information Retrieval methods directly to multimedia objects like images,
video audio documents or interactive presentations, is often not possible without
adaptations of existing algorithms. Aiming for cross media retrieval, which
allows retrieval across different formats and media types, a common denominator
for different media types and formats has to be found. A common approach is
the usage of metadata in a way that does not differentiate between single media
types and formats. For multimedia objects one of these already specified metadata
schemes is MPEG-7. On the one hand MPEG-7 allows low level and technical descriptions
like color distributions or blockiness
of an image. On the other hand manual annotation of multimedia objects in different
levels, ranging from free text annotations to complex computer-readable structured
semantic descriptions, is supported. The paper we present is twofold. In the
first half a current research project prototype called "Caliph
& Emir" is presented. It allows annotation and retrieval of
semantic descriptions, created and stored as directed graphs, in the context
of digital photos. The annotation and retrieval capabilities are based on MPEG-7.
In addition to semantic descriptions common content based image retrieval mechanisms
were implemented and widely used metadata formats for digital photos like EXIF,
Adobe's XMP and IPTC IIM were integrated. Caliph, which stands for "Common
And Light weight PHoto Annotation",
is the annotation part supporting graphical annotation and graph visualization.
Emir, which is short for "Experimental Metadata based Image Retrieval",
is the retrieval engine. Besides the prototype description the value added by
semantic descriptions is discussed. The second part gives an outlook on future
work. The next step in development is merging the retrieval capabilities of
"Caliph & Emir" with another research
project, a cross media retrieval and visualization application. We also outline
further developments towards cross media applications in general and we discuss
the integration of existing ontologies for semantic
annotation and retrieval in combination based on MPEG-7.