Multilevel Representation and Query Processing in Multimedia Database Systems

Arif Ghafoor (PI), Rangasami L. Kashyap (Co-PI), Shankar Moni (Co-PI)
Purdue University, West Lafayette, IN, 47907

Contact Information

Arif Ghafoor
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, 47907
Phone: (765) 494-0638    Fax: (765) 494-3371    Email: ghafoor@ecn.purdue.edu

WWW PAGE

Distributed Multimedia Systems Laboratory (URL: http://shay.ecn.purdue.edu/~dmultlab/nsf_proj/project.html)

Supported Students

Keywords

multimedia data representation, fuzzy queries, spatio-temporal modeling, content-based retrieval, image processing

Project Award Information

Project Summary

The goal of this project is to develop a multimedia database system with capabilities to handle heterogeneous media queries. This system caters to the computational and storage requirements while accommodating and exploiting the inevitable semantic and representational imprecisions. The design of this system is based on multilevel data models and search mechanisms. These methodologies facilitate the users for posing various types of queries, including: (i) low-level, such as finding objects in a multimedia database, (ii) mid-level, based on spatio-temporal semantics, such as locating events associated with multimedia data, and (iii) high-level, targeted towards searching pre-composed multimedia documents, based on constituent mono-media data, their spatio-temporal dimensions, and logical structure. The multi-level search mechanisms are tightly interlinked. Imprecision in the search results is modeled using a set of fuzzy parameters. The results of this research are helping develop a comprehensive framework for building wide variety of multimedia applications in commercial, educational, governmental and military sectors.

Goals, Objectives, and Targeted Activities

Our long term goals include the development of a generic multi-level representation of multimedia data that can overcome several challenges faced by the database community. In particular, for low-level data modeling our research will focus on evaluating the unsupervised classification approach for image segmentation using a log-likelihood function. Two image segmentation models, namely the facet model and the texture model, are being evaluated in terms of computation and exactness of match. We are also planning to develop and evaluate other models. For mid-level multimedia data representation, we are planning to develop indexing, Petri-net and neighborhood graph models for semantic modeling and managing fuzziness in query formulation.

Indication of Success

So far the stated objectives for this project have been met quite well, as demonstrated by the concrete research results produced to date. Success of this project in terms of quality research results has been aided by the collaboration between two researchers with backgrounds in image processing and multimedia databases. This collaboration has been instrumental in providing a sharp understanding of research challenges astride these areas. As a result of this unique collaboration, several research ideas and publications are being produced. The following research results has been obtained:
  1. We have developed a conceptual model for multimedia documents that incorporates space, time, and content information. In order to model the spatio-temporal relationships among multimedia objects in a multimedia document, the OCPN modeling scheme proposed by one of the PI's has been enhanced and utilized as a multimedia document model.
  2. Based on the enhanced OCPN-based model, we have developed a database architecture and are currently developing a prototype system. The architecture of the system has the following components:
    1. An authoring tool for multimedia document creation
    2. An OCPN script generator, translator, and an automatic deadline generator from OCPN scripts.
    3. A query processor allowing the user to store, query and retrieve multimedia documents from the underlying multimedia databases.
    4. A presentation manager responsible for scheduling the events and actions to play back and synchronize multimedia objects during document presentation.
    The prototype system is being implemented using an object oriented DBMS as an underlying database management system along with C++ and X/Motif Graphical User Interface.
  3. We have developed a semantic model that support content-based retrieval and modeling for video databases. The modeling approach uses multi-level representation of video data using hierarchical Petri-nets (HPNs). This model captures video data semantics, from very low level semantics such as scene change and object movements to higher level semantics involving high-level textual description of events.
  4. We have developed another semantic representation of video data using a coordinate valued neighborhood graph (CVNG) model. CVNG is an extension to the neighborhood graph model developed by Nabil et. al. to compute similarity between still images. In video, the spatial relationship between two objects over a sequence of frames is captured using a symbolic representation of a video sequence. We utilize the techniques reported in nsf-report year 2 in query processing which provide computationally efficient techniques to compare complex video scenes.
The project continues to provide viable solutions for developing a general framework for developing multimedia databases needed for a broad range of applications. The NSF funding has provided an opportunity for this collaboration which would have been difficult, otherwise.

Project Impact and Output

Two students are currently pursuing their doctoral studies as part of this project. One student has completed his doctoral thesis in this area. The research results of this project have been incorporated in a graduate level course on multimedia systems (EE 624), which is taught by the PI. The project and its planned implementation is a cornerstone of the cutting-edge research being carried out in our lab. We anticipate interest from industrial organizations as the project matures and implementation-worthy results are produced.

Project References


[1] Shu-Ching Chen and R. L. Kashyap, "A Spatial-Temporal Semantic Model for Multimedia Presentations and Multimedia Database Systems" to appear in IEEE Trans. on Knowledge & Data Engineering.
[2] S. Dagtas, W. Al-Khatib, A. Ghafoor, and R. L. Kashyap, ``Models for Motion-based Video Indexing and Retrieval'', IEEE Trans. on Image Processing, 9(1), pp. 88-101, January 2000
[3] W. Al-Khatib and A. Ghafoor, ``An Approach for Video Meta-Data Modeling and Query Processing", in the proceedings of the seventh ACM Multimedia International Conference, pp. 215-224, Orlando, Florida, October 30-November 5, 1999.
[4] S. Dagtas and A. Ghafoor, ``Indexing and Retrieval of Video based on Spatial Relation Sequences", in the proceedings of the seventh ACM Multimedia International Conference, Part 2, pp. 119-122, Orlando, Florida, October 30-November 5, 1999.
[5] Y. F. Day, A. Khokhar, S. Dagtas, and A. Ghafoor, ``A Multi-level Abstraction and Modeling in Video Databases'', Multimedia Systems, 7(5), pp. 409-423, Septemeber 1999.
[6] S. Dagtas, W. Al-Khatib, A. Khokhar, and A. Ghafoor, ``Trail-Based Approach for Video Data Indexing and Retrieval'', In proceedings of the IEEE International Conference on Multimedia Computing and Systems (ICMCS), Vol II, pp. 235-239, Florence, Italy, June, 1999
[7] W. Al-Khatib, Y. F. Day, A. Ghafoor, and P. B. Berra, ``Semantic Modeling and Knowledge Representation in Multimedia Databases'', IEEE Trans. on Knowledge & Data Engineering, 11(1), pp. 64-80, January-February, 1999
[8] S. Sista and R. L. Kashyap, "Unsupervised Video Segmentation and Object Tracking," In proceedings of the IEEE International Conference on Image Processing, Kobe, Japan, October 1999.
[9] Shu-Ching Chen and R. L. Kashyap, "Empirical Studies of Multimedia Semantic Models for Multimedia Presentations," 13th International Conference on Computer and Their Applications, Honolulu, Hawaii, USA, March 25-27, 1998.
[10] Shu-Ching Chen and R. L. Kashyap, "Temporal and Spatial Semantic Models for Multimedia Presentations," 1997 International Symposium on Multimedia Information Processing, Dec. 11-13, 1997.

Area Background

Our project is concerned with data management and information retrieval technologies essential for developing future multimedia systems. The emphasis of our research is on developing an automated system to allow multi-level data representation to assist query processing at different levels of abstraction. It will allow users to access images and video data based on appearance of objects as well as events surrounding these objects. The key tradeoff for users is between the accuracy of matching and the computational cost of the query. The framework developed for this project will provide solutions for challenging problems in multimedia data organization and integration, indexing and retrieval mechanisms, intelligent searching techniques, information browsing, content-based query processing and so forth. A large variety of potential applications will benefit from this framework.

Area References

There are several journals and conferences which have excellent coverage of issues in this area. They include: IEEE Multimedia; ACM Journal on Multimedia Systems; IEEE Trans. on Knowledge and Data Engineering; ACM Multimedia Conference; IEEE Int. Conf. on Multimedia Computing and Systems. Several special issues from IEEE Computer and ACM Journal on Multimedia Systems have been specifically devoted to this topic. Several industrial projects undertaken by IBM, Siemens, NEC, Oracle, Fuji Electric Co., etc., are focused on this topic.

Potential Related Projects

Within the NSF IDM program, several projects related to multimedia data modeling and management are being conducted in UCLA, Case Western Reserve University, University of Maryland, University of Nevada, University of Illinois at Chicago, University of California at Santa Barbara, University of Pittsburgh, University of Maine, and University of Washington.