A framework for the analysis and evaluation of enterprise models

TL;DR: The purpose of this study is the development and validation of a comprehensive framework for the analysis and evaluation of enterprise models based on the distinction between the syntactic, semantic and pragmatic model aspects and populated with evaluation criteria drawn from an extensive literature survey.

Abstract: The purpose of this study is the development and validation of a comprehensive framework for the analysis and evaluation of enterprise models. The study starts with an extensive literature review of modelling concepts and an overview of the various reference disciplines concerned with enterprise modelling. This overview is more extensive than usual in order to accommodate readers from different backgrounds. The proposed framework is based on the distinction between the syntactic, semantic and pragmatic model aspects and populated with evaluation criteria drawn from an extensive literature survey. In order to operationalize and empirically validate the framework, an exhaustive survey of enterprise models was conducted. From this survey, an XML database of more than twenty relatively large, publicly available enterprise models was constructed. A strong emphasis was placed on the interdisciplinary nature of this database and models were drawn from ontology research, linguistics, analysis patterns as well as the traditional fields of data modelling, data warehousing and enterprise systems. The resultant database forms the test bed for the detailed framework-based analysis and its public availability should constitute a useful contribution to the modelling research community. The bulk of the research is dedicated to implementing and validating specific analysis techniques to quantify the various model evaluation criteria of the framework. The aim for each of the analysis techniques is that it can, where possible, be automated and generalised to other modelling domains. The syntactic measures and analysis techniques originate largely from the disciplines of systems engineering, graph theory and computer science. Various metrics to measure model hierarchy, architecture and complexity are tested and discussed. It is found that many are not particularly useful or valid for enterprise models. Hence some new measures are proposed to assist with model visualization and an original "model signature" consisting of three key metrics is proposed. Perhaps the most significant contribution ofthe research lies in the development and validation of a significant number of semantic analysis techniques, drawing heavily on current developments in lexicography, linguistics and ontology research. Some novel and interesting techniques are proposed to measure, inter alia, domain coverage, model genericity, quality of documentation, perspicuity and model similarity. Especially model similarity is explored in depth by means of various similarity and clustering algorithms as well as ways to visualize the similarity between models. Finally, a number of pragmatic analyses techniques are applied to the models. These include face validity, degree of use, authority of model author, availability, cost, flexibility, adaptability, model currency, maturity and degree of support. This analysis relies mostly on the searching for and ranking of certain specific information details, often involving a degree of subjective interpretation, although more specific quantitative procedures are suggested for some of the criteria. To aid future researchers, a separate chapter lists some promising analysis techniques that were investigated but found to be problematic from methodological perspective. More interestingly, this chapter also presents a very strong conceptual case on how the proposed framework and the analysis techniques associated vrith its various criteria can be applied to many other information systems research areas. The case is presented on the grounds of the underlying isomorphism between the various research areas and illustrated by suggesting the application of the framework to evaluate web sites, algorithms, software applications, programming languages, system development methodologies and user interfaces. ii Un ive rsi ty of Ca pe To wn Acknowledgements This work would not have been possible without the unwavering emotional, motivational and practical support from my dear wife Eva words will never be able to express my thanks and appreciation. I also thank my lovely children Anneke, Jonathan and Sylvia for their patience and understanding, especially in the last phases of the thesis where daddy was all work and no play. Only people who have been through the "Ph D" experience themselves, can appreciate the sacrifices that close and loved family members go through in order for the dissertation to be written. Without my family's steadfast support, this research would have been still-born and this dissertation represents almost as much the culmination of their hard work as mine. Of course, I must also thank my parents, Roger and Monica Van Belle-Deleu, who gave me my life, education and planted the seeds of all my inspirations. Thanks also to my supervisors, without whom this thesis would have been stillborn: Prof Jonathan Miller who fired the starting gun, Prof Paul Licker who counted the laps, and Prof Mike Hart who manned the finish line. I must also express my gratitude to Prof Dewald Roode who made numerous valuable suggestions. My colleagues from the information systems department here at UCT deserve a very special mention: Prof Derek Smith, Mike Eccles, Graham McLeod, and Jane Nash for their support, guidance and practical assistance. Many deans who saw me through and provided financial supported whenever I called on them: from my years at UWC Prof Andre Kritzinger and Prof Renfrew Christie and here at UCT: Prof Peter Kritzinger, Prof Brian Kantor and Prof Douglas Pitt. The following people shared their time, office, knowledge and often granted me hospitality at their homes: Tom Russ (Information Sciences Institute, University of Southern California), Henry Kim (Industrial Engineering, University of Toronto), Wouter Jansweijer (Social Science Informatics, University of Amsterdam), John Kingston (AIAI, University of Edinburgh), Elizabeth Post (Lincoln University), Prof Richard Weston and Paul Gilders (Manufacturing Systems Integration Research Institute, University of Loughborough). Prof Glen Lowry (Convenor, Information Systems Research Group, University of Tasmania), Prof Peter Bemus (Griffith University, vice-chairman of the IFIPIIF AC Task Force on Architectures for Enterprise Integration) and Prof Andy Bytheway (UWC) helped me scope this research, provided valuable high-level direction and shared their invaluable experience. Constructive criticism was also received from Prof. Trevor Wegner (UCT), Mike Uschold (AIAI, University of Edinburgh) and Jean-Louis Van Belle (University of Antwerp). I also wish to thank the following organisations who helped fund my research travel expenses and conference attendance: VCT, UWC, and the CSDINRF. Both UCT and UWC granted me six months paid sabbaticals which enabled me to pursue and finalize my research and the CSDINRF helped fund some of my research and conference expenses. I also owe an intellectual debt to the many conference organisers and delegates who attended my conference presentations and gave valuable constructive feedback. Finally thanks to three special friends. Jean-Louis Van Belle, motivated me by challenging me to finish before him (he owes me a Belgian beer). Dr Stephen Craven did an excellent proofreading job, provided unlimited homebrew and served as a living example that it is possible to do your thesis part-time. Last, but definitely not least, Caroline Brawner helped with some of the data entry, revitalized my energies by dragging me off to many mountain runs, proofread the entire thesis and provide some valuable criticism. iii Un ive rsi ty of Ca pe To wn