Reliability and validity issues in analysis of IT spending using IT managerial control ratios

Purpose The purpose of this paper is to critically evaluate the practice of IT cost benchmarking using IT managerial control ratios. First, numerous reliability and validity issues that this practice is plagued by are discussed. Second, recommendations on addressing some of these reliability and validity issues are also proposed. Design/methodology/approach This is a conceptual paper that draws on literature utilizing IT managerial control ratios to discuss various reliability problems associated with these measures and also makes use of IT strategy theory to challenge the validity of common interpretations of IT managerial control ratios. Findings This study explains that IT managerial control ratios are subject to numerous reliability and validity issues. The reliability issues arise from the inherent volatility of some of the variables used in these ratios and difficulties that executives have in providing a reliable estimate of those variables when they are approached by market research firms. The validity issues arise from the fact that high and low values of these ratios are subject to different and, at times, somewhat orthogonal theoretical interpretations. Practical implications IT managerial control ratios are often at the heart of important capital allocation decisions as well as studies which form important stakeholders’ perceptions regarding the business value of IT. These important uses of IT managerial control ratios should not be carried out without understanding the reliability and validity issues discussed in this paper. Originality/value While IT managerial control ratios are used extensively in benchmarking and research, very few researchers and practitioners possess a full understanding of the reliability and validity issues associated with these measures. This can potentially lead to sub-optimal capital allocation decisions and erroneous findings in studies. This paper provides a comprehensive overview of these issues and recommends possible remedies.