A Survey on Automated Log Analysis for Reliability Engineering

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems, because they are often the only data available that record software runtime information. As modern software is evolving into a large scale, the volume of logs has increased rapidly. To enable effective and efficient usage of modern software logs in reliability engineering, a number of studies have been conducted on automated log analysis. This survey presents a detailed overview of automated log analysis research, including how to automate and assist the writing of logging statements, how to compress logs, how to parse logs into structured event templates, and how to employ logs to detect anomalies, predict failures, and facilitate diagnosis. Additionally, we survey work that releases open-source toolkits and datasets. Based on the discussion of the recent advances, we present several promising future directions toward real-world and next-generation automated log analysis.

Research on Various Tools in Big Data

The field big data has wonderful changes in the last few years. Data are collected very massive amount and cheaply through network devices such as mobile, camera, microphone, software logs etc. Information is coming various sources and also the nature of the information is different. It is very difficult to perform with very huge data sets and different nature by using traditional data application software techniques. For many field we need to take out the valuable information from enormous and noisy data sets. In this paper we analyses the characteristics among four different tools and the comparison value is very useful to determine the efficient analytic tool. Based on the value which made user to select the best tool for to advance the performance of big data in a easiest way