Effects of Domain Familiarity on Conceptual Modeling Performance

Conceptual modeling is currently considered a significant phase in information systems development. Several modeling grammars and methods have been studied extensively in the information systems discipline. However, previous research on conceptual models has focused on certain grammar (syntax) or discovering a way to deliver the meaning of a model (semantic) more clearly and completely. With regard to the construct overload issue in conceptual modeling, past studies have had some deficiencies in research methods and even presented contradicting results. The objective of the present study is twofold. First, the authors researched the interaction effect among syntax, semantics, and pragmatics to discover the preferred design, context, and user knowledge with which models are more likely to be understood or interpreted. Second, they performed an experiment to reconcile conflicting outcomes and acquire a more complete and accurate understanding of construct overload. Specifically, the authors focused on understanding the end users' modeling performance between ontologically clear and unclear models. They applied an improved experimental methodology that integrates three features (i.e., syntax, semantic, pragmatic) rather than treat them individually and employs different degrees of domain familiarity in the conceptual model (i.e., familiar domain vs. unfamiliar domain). The result of this study will broaden the perspective on usability in the context of the conceptual model and may serve as a modeler's ontological guidance in terms of whether or not to contain construct overload when they create a model. In addition, this study makes the theoretical contribution by verifying the domain extensibility towards the theory of ontological clarity.

An Experimental Study on the Influence That Failure Number, Specialization, and Controls Have on Confidence in Predicting System Failures1

Design reviews are typically used for three types of design activities: (1) identifying errors, (2) assessing the impact of the errors, and (3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Nondomain generalists (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to provide their own opinion or confidence on whether the system would still successfully achieve the stated objectives. The confidence level for all domain populations decreased significantly as the number of design errors increased (largest p-value = 0.0793), and this decrease in confidence is more significant as the number of design errors increases. The impact on confidence is lower when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334) as the confidence decreased faster for domain general engineers as compared to domain specialists (p = < 0.0001). The domain specialists showed higher confidence in making decisions than domain generalists and nondomain generalists as the design errors increase.

User Study: Influence of Number of Design Errors on Ability to Predict Performance With and Without Controls

Design reviews are typically used for three types of design activities: 1) identifying errors, 2) assessing the impact of the errors, and 3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Non-domain individuals (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain-specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to estimate the likelihood that the system would still successfully achieve the stated objectives. Primary results from the study include the following: the confidence level for all domain population decreased significantly as the number of design errors increased (largest p-value = 0.0793) and this decrease in confidence is more significant as the design errors increase. The impact on confidence is less when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334), the confidence decreased faster for domain general engineers as compared to domain specialists (p = <0.0001). The domain specialists showed higher confidence in making decisions than domain generals and non-domain generalists as the design errors increase.