Mastering Real-Time Data Quality Control - How to Measure and Manage the Quality of (Rig) Sensor Data

2007 ◽  
Author(s):  
Wolfgang Mathis ◽  
Gerhard Thonhauser
Keyword(s):  
Quality Control ◽  
Data Quality ◽  
Real Time ◽  
Sensor Data ◽  
Data Quality Control ◽  
Time Data ◽  
Real Time Data

Real time data quality control applied on an IOT sensor water quality network

2020 ◽  
Author(s):  
Matthias Maeyens ◽  
Brianna Pagán ◽  
Piet Seuntjens ◽  
Bino Maiheu ◽  
Nele Desmet ◽  
...  
Keyword(s):  
Water Quality ◽  
Quality Control ◽  
Surface Water ◽  
Data Quality ◽  
Real Time ◽  
Water Quality Monitoring ◽  
Data Quality Control ◽  
Time Data ◽  
Data Set ◽  
Real Time Data

<p>In recent years, extend periods of drought have been affecting the water quality and availability in  the Flanders region in Belgium. Especially the coastal region experienced an increased salinization of ground and surface water. The Flemish government therefore decided to invest in a dense IoT water quality monitoring network aiming to deploy 2500 water quality sensors  primarily in surface water but also in ground water and sewers. The goal of this "Internet of Water" project is to establish an operational state of the art monitoring and prediction system in support of future water policy in Flanders. </p><p>Since Flanders is a relatively small region (13,522 km²), placing this many sensors will result in one of the most dense surface water quality sensor networks in the world. Each sensor will continuously measure several indicators of water quality and transmit the data wirelessly. This allows us to continuously monitor the water quality and build a big enough data set to be able to use a more data driven approach to predicting changes  in water quality. However, as with any sensor system, the quality of the data can vary in time due to problems with the sensors, incorrect calibration or unforeseen issues. Real-time data quality control is crucial to prevent unsound decisions due to faulty data.</p><p>This contribution will give a general overview of the network and it’s specifications, but mainly focus on the implementation of the data stream as well as methods that are implemented to guarantee good data quality. More specifically the architecture and setup of a real-time data quality control system is described. Which will add quality control flags to measurements.  This system is  integrated with the NGSI API introduced by FIWARE, which forces us to make specific design decisions to acommodate to the NGSI API.</p>

Automated Drilling Data Quality Control Using Application of AI Technologies

2021 ◽  
Author(s):  
Francesco Battocchio ◽  
Jaijith Sreekantan ◽  
Arghad Arnaout ◽  
Abed Benaichouche ◽  
Juma Sulaiman Al Shamsi ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Quality Control ◽  
Data Quality ◽  
Information Loss ◽  
Sensor Data ◽  
Data Quality Control ◽  
Drilling Data ◽  
Advanced Analytics ◽  
Quality Issues

Abstract Drilling data quality is notoriously a challenge for any analytics application, due to complexity of the real-time data acquisition system which routinely generates: (i) Time related issues caused by irregular sampling, (ii) Channel related issues in terms of non-uniform names and units, missing or wrong values, and (iii) Depth related issues caused block position resets, and depth compensation (for floating rigs). On the other hand, artificial intelligence drilling applications typically require a consistent stream of high-quality data as an input for their algorithms, as well as for visualization. In this work we present an automated workflow enhanced by data driven techniques that resolves complex quality issues, harmonize sensor drilling data, and report the quality of the dataset to be used for advanced analytics. The approach proposes an automated data quality workflow which formalizes the characteristics, requirements and constraints of sensor data within the context of drilling operations. The workflow leverages machine learning algorithms, statistics, signal processing and rule-based engines for detection of data quality issues including error values, outliers, bias, drifts, noise, and missing values. Further, once data quality issues are classified, they are scored and treated on a context specific basis in order to recover the maximum volume of data while avoiding information loss. This results into a data quality and preparation engine that organizes drilling data for further advanced analytics, and reports the quality of the dataset through key performance indicators. This novel data processing workflow allowed to recover more than 90% of a drilling dataset made of 18 offshore wells, that otherwise could not be used for analytics. This was achieved by resolving specific issues including, resampling timeseries with gaps and different sampling rates, smart imputation of wrong/missing data while preserving consistency of dataset across all channels. Additional improvement would include recovering data values that felt outside a meaningful range because of sensor drifting or depth resets. The present work automates the end-to-end workflow for data quality control of drilling sensor data leveraging advanced Artificial Intelligence (AI) algorithms. It allows to detect and classify patterns of wrong/missing data, and to recover them through a context driven approach that prevents information loss. As a result, the maximum amount of data is recovered for artificial intelligence drilling applications. The workflow also enables optimal time synchronization of different sensors streaming data at different frequencies, within discontinuous time intervals.

scholarly journals QoS Management for Embedded Databases in Multicore-Based Embedded Systems

2015 ◽  
Vol 2015 ◽  
pp. 1-14 ◽  
Author(s):  
Woochul Kang ◽  
Jaeyong Chung
Keyword(s):  
Embedded Systems ◽  
Real Time ◽  
Control Method ◽  
Network Connectivity ◽  
Time Data ◽  
Quality Of Data ◽  
Multiple Control ◽  
Data Intensive ◽  
Real Time Data

With ubiquitous deployment of sensors and network connectivity, amounts of real-time data for embedded systems are increasing rapidly and database capability is required for many embedded systems for systematic management of real-time data. In such embedded systems, supporting the timeliness of tasks accessing databases is an important problem. However, recent multicore-based embedded architectures pose a significant challenge for such data-intensive real-time tasks since the response time of accessing data can be significantly affected by potential intercore interferences. In this paper, we propose a novel feedback control scheme that supports the timeliness of data-intensive tasks against unpredictable intercore interferences. In particular, we use multiple inputs/multiple outputs (MIMO) control method that exploits multiple control knobs, for example, CPU frequency and the Quality-of-Data (QoD) to handle highly unpredictable workloads in multicore systems. Experimental results, using actual implementation, show that the proposed approach achieves the target Quality-of-Service (QoS) goals, such as task timeliness and Quality-of-Data (QoD) while consuming less energy compared to baseline approaches.

Traffic Controller for Handling Service Quality in Multimedia Network

2010 ◽  
pp. 96-112
Author(s):  
Manjunath Ramachandra ◽  
Vikas Jain
Keyword(s):  
Real Time ◽  
Long Range ◽  
Multimedia Data ◽  
Time Data ◽  
Self Similarity ◽  
Real Time Data ◽  
Data Transfers ◽  
Multimedia Network ◽  
Different Origin

The present day Internet traffic largely caters for the multimedia traffic throwing open new and unthinkable applications such as tele-surgery. The complexity of data transactions increases with a demand for in time and real time data transfers, demanding the limited resources of the network beyond their capabilities. It requires a prioritization of data transfers, controlled dumping of data over the network etc. To make the matter worse, the data from different origin combine together imparting long lasting detrimental features such as self similarity and long range dependency in to the traffic. The multimedia data fortunately is associated with redundancies that may be removed through efficient compression techniques. There exists a provision to control the compression or bitrates based on the availability of resources in the network. The traffic controller or shaper has to optimize the quality of the transferred multimedia data depending up on the state of the network. In this chapter, a novel traffic shaper is introduced considering the adverse properties of the network and counteract with the same.

scholarly journals Fossel: Efficient Latency Reduction in Approximating Streaming Sensor Data

2020 ◽  
Vol 12 (23) ◽  
pp. 10175
Author(s):  
Fatima Abdullah ◽  
Limei Peng ◽  
Byungchul Tak
Keyword(s):  
Real Time ◽  
Data Analytics ◽  
Streaming Data ◽  
Sensor Data ◽  
Simple Type ◽  
Data Sampling ◽  
Time Data ◽  
Latency Reduction ◽  
Network Delays ◽  
Real Time Data

The volume of streaming sensor data from various environmental sensors continues to increase rapidly due to wider deployments of IoT devices at much greater scales than ever before. This, in turn, causes massive increase in the fog, cloud network traffic which leads to heavily delayed network operations. In streaming data analytics, the ability to obtain real time data insight is crucial for computational sustainability for many IoT enabled applications such as environmental monitors, pollution and climate surveillance, traffic control or even E-commerce applications. However, such network delays prevent us from achieving high quality real-time data analytics of environmental information. In order to address this challenge, we propose the Fog Sampling Node Selector (Fossel) technique that can significantly reduce the IoT network and processing delays by algorithmically selecting an optimal subset of fog nodes to perform the sensor data sampling. In addition, our technique performs a simple type of query executions within the fog nodes in order to further reduce the network delays by processing the data near the data producing devices. Our extensive evaluations show that Fossel technique outperforms the state-of-the-art in terms of latency reduction as well as in bandwidth consumption, network usage and energy consumption.

Intelligent Data Quality Control of Real-time Rig Data

2013 ◽  
Author(s):  
Arghad Arnaout ◽  
Philipp Zoellner ◽  
Gerhard Thonhauser ◽  
Neil Johnstone
Keyword(s):  
Quality Control ◽  
Data Quality ◽  
Real Time ◽  
Data Quality Control

scholarly journals A Framework for Real Time Processing of Sensor Data in the Cloud

2015 ◽  
Vol 2015 ◽  
pp. 1-11 ◽  
Author(s):  
Supun Kamburugamuve ◽  
Leif Christiansen ◽  
Geoffrey Fox
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Cloud Services ◽  
Sensor Data ◽  
Smart Devices ◽  
Time Data ◽  
Real Time Processing ◽  
Real Time Data ◽  
Autonomous Robot Navigation ◽  
Real Time Data Processing

We describe IoTCloud, a platform to connect smart devices to cloud services for real time data processing and control. A device connected to IoTCloud can communicate with real time data analysis frameworks deployed in the cloud via messaging. The platform design is scalable in connecting devices as well as transferring and processing data. With IoTCloud, a user can develop real time data processing algorithms in an abstract framework without concern for the underlying details of how the data is distributed and transferred. For this platform, we primarily consider real time robotics applications such as autonomous robot navigation, where there are strict requirements on processing latency and demand for scalable processing. To demonstrate the effectiveness of the system, a robotic application is developed on top of the framework. The system and the robotics application characteristics are measured to show that data processing in central servers is feasible for real time sensor applications.

Sign in / Sign up

Export Citation Format

Share Document