Automatic recognition of actions and activities in multimedia content from sports domain (RAASS)
HRZZ Research Project (IP-06-2016)
Principal Investigator (PI)Assistant Professor MarinaIvasic-Kos, Ph.D.
Institution where the project will be implemented: Department of Informatics, University ofRijeka
Duration of project: 48 months
Abstract:
Multimedia content such as digital images and video have become an important storage and exchange media. A large amount of such content from competitions, games and training sessions is constantly being created in the sports domain, and there has been a great interest for multimedia content analysis among athletes and coaches, on all levels from professional to recreational.
Such analysis is indispensable in modern professional sports but is still done primarily by manual annotation of individual actions. Development of methods for automatic action detection and athlete’s activity interpretation would enable more efficient analysis and broaden the availability of analysis solutions to everyone involved with sports. Analysis of individual actions, such as swimming turns or steps in running, is important for athletes, coaches and physical therapists alike because it allows them to monitor the form of performing individual exercises, to improve the athlete’s performance and to keep track of the progress. For interpretation of multiple sequential actions into a meaningful activity, sports domain knowledge is necessary.
We propose a research in which methods for sports domain action detection and annotation in multimedia content would be developed. The starting point for action detection are methods based on machine learning and features that describe the spatio-temporal dimensions of multimedia content. For interpretation of sequential actions into more complex activities a model will be developed based on sports domain, which has well-defined rules, a known structure and context and clear roles exist for athletes and others involved. To model the fuzzy knowledge of the domain and temporal relations a knowledge representation scheme will be defined based on the theory of timed and fuzzy Petri nets or a similar formalism. Due to the specifics of different sports, actions from chosen sports will be considered in the research. The project outcomes will include models for automatic recognition of individual actions and activities as well as metrics for comparing a performance of an action with a reference performance. The developed prototype will be used to develop a system that will allow athletes, coaches, physiotherapists and others to track the performance of exercises, performance improvement, progress monitoring, collection of statistical data for analysis of a sports event and similar.
Team Members and Research Group:
- Assistant professor Marina Ivašić-Kos, PhD, Department of Informatics / University of Rijeka Croatia; (project leader)
- Senior assistant Miran Pobar, PhD, Department of Informatics / University of Rijeka Croatia;
- Associate professor Jordi Gonzalez Sabaté, PhD, Universitat Autonoma de Barcelona / School of Engineering Spain
- Matija Burić, PhD student at the Department of Informatics / University of Rijeka; employed at HEP d.o.o Croatia
- Professor Slobodan Ribarić, PhD, Faculty of electrical engineering and Computing / University of Zagreb Croatia; (consultant)
- Milan Kosanović, Kinesiologist, Croatia; (ekspert)
Related Publications:
Journal Articles:
- Ivašić-Kos, Marina; Pobar, Miran; Ribarić, Slobodan. Two-tier image annotation model based on a multi-label classifier and fuzzy-knowledge representation scheme. // Pattern recognition. 52 (2016); 287-305.
- Ivašić-Kos, Marina; Ipšić, Ivo; Ribarić, Slobodan. A knowledge-based multi-layered image annotation system. // Expert systems with applications. 42 (2015) , 2015; 9539-9553.
- Ivašić-Kos, Marina; Ipšić, Ivo; Ribarić, Slobodan. Multi-Level Image Annotation Using Bayes Classifier and Fuzzy Knowledge Representation Scheme. // WSEAS transactions on computers. 13 (2014); 635-644.
- Ivašić-Kos, Marina; Ribarić, Slobodan; Ipšić, Ivo. Low- and High-level Image Annotation Using Fuzzy Petri Net Knowledge Representation Scheme. // International Journal of Computer Information Systems and Industrial Management (IJCISIM). 4 (2012); 428-436.
Conference papers:
- Ivašić-Kos, Marina; Pobar, Miran; Ribarić, Slobodan. Automatic image annotation refinement using fuzzy inference algorithms // European Centre for Soft Computing, IFSA- EUSFLAT 2015 / Dr. Luis Magdalena Layos (ur.). Gijón, Asturias (Spain) : IFSA-EUSFLAT2015, 2015. rad 242.
- Ivašić-Kos, Marina; Pobar, Miran; Ribarić, Slobodan. Fuzzy Knowledge-Based Image Annotation Refinement // Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition IPCV'15 / Arabnia, Hamid R. ; Deligiannidis, Leonidas ; Tinetti, Fernando G. (ur.). Las Vegas, Nevada, USA : The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), 2015. 284-290
- Pobar, Miran; Ivašić-Kos, Marina. Multimodal Image Retrieval Based on Keywords and Low-Level Image Features // Semantic Keyword-based Search on Structured Data Sources: First COST Action IC1302 International KEYSTONE Conference, IKC 2015, Coimbra, Portugal, September 8-9, 2015. Revised Selected Papers / Cardoso, J., Guerra, F., Houben, G.-J., Pinto, A.M., Velegrakis, Y. (Eds.) (ur.). Coimbra : Springer, 2015. 133-140
- Ivašić-Kos, Marina; Iosifidis, Alexandros; Tefas, Anastasios; Pitas, Ioannis. Person De-Identification in Activity Videos // BiForD - BIOMETRICS & FORENSICS & DE- IDENTIFICATION AND PRIVACY PROTECTION / Slobodan Ribaric (ur.). Opatija, Hrvatska : Croatian Society for Information and Communication Technology, Electronics and Microelectronics - MIPR, 2014. 75-80
- Ivašić-Kos, Marina; Pobar, Miran; Ipšić, Ivo. Multi-layered Image Representation for Image Interpretation // Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014). Dublin, 2014. 115-117.
- Ivašić-Kos, Marina; Pobar, Miran; Ipšić, Ivo. Automatic Movie Posters Classification into Genres // Advances in Intelligent Systems and Computing 311 / Madevska Bogdanova, Ana ; Gjorgjevikj, Dejan (ur.). Springer, 2014. 319-328.
- Ivašić-Kos, Marina; Pobar, Miran; Ipšić, Ivo. Object Level vs. Scene Level Image Annotation // Recent Advances in Electrical and Electronic Engineering, Proceedings of the 3rd International Conference on Circuits, Systems, Communications, Computers and Applications (CSCCA '14) / Mastorakis, Nikos E. ; Nakamatsu, Kazumi ; Paspalakis, Emmanuel (ur.). Firenca : WSEAS Press, 2014. 162-168.
- Ivašić-Kos, Marina; Pobar, Miran; Mikec, Luka. Movie posters classification into genres based on low-level features // Proceedings of MIPRO 2014 / Biljanović, Petar (ur.). Rijeka : Croatian Society for Information and Communication Technology, Electronics and Microelectronics - MIPRO, 2014. 1448-1453.
- Ivašić-Kos, Marina; Ribarić, Slobodan; Ipšić, Ivo. Multi-level Image Classification Using Fuzzy Petri Net // Recent Advances in Neural Networks and Fuzzy Systems, Proceedings of the 2014 International Conference on Neural Networks - Fuzzy Systems (NN- FS '14) / Prof. Yingxu Wang, Prof. Pierre Borne (ur.). Venice, Italy : Europment, 2014. 39-45.
- Ivašić-Kos, Marina; Pavlić, Mile; Poščić, Patrizia. Descriptors’ Specification for Image Representation // Proceedings of International Conference on Image and Video Processing and Computer Vision. Orlando, Florida : PromoteResearch, 2010.
- Ivašić-Kos, Marina; Poščić, Patrizia; Pavlić, Mile. Definition of descriptors for semantic image interpretation // Proceedings of the IEEE 2nd European Workshop on Visual Information Processing - EUVIP 2010. Pariz, Francuska, 2010.
- Ivašić-Kos, Marina; Ribarić, Slobodan; Ipšić, Ivo. Image Annotation using Fuzzy Knowledge Representation Scheme // Proceedings of the 2010 International Conference of Soft Computing and Pattern Recognition. Pariz, Francuska : IEEE France, 2010. 218-223
- Ivašić-Kos, Marina; Pavlić, Mile; Pobar, Miran. Analyzing the semantic level of outdoor image annotation // Proceedings of MIPRO 2009 - 32nd International convention on information and communication technology, electronics and microelectronics / Bogunović, Nikola ; Ribarić, Slobodan (ur.). Opatija : Croatian Society for Information and Communication Technology, Electronics and Microelectronics - MIPRO, 2009. 293-296.