Soft Real-Time Hadoop Scheduler for Big Data Processing in Smart Cities

2016 ◽  
Author(s):  
Ciprian Barbieru ◽  
Florin Pop
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Smart Cities ◽  
Big Data Processing

scholarly journals The real-time big data processing method based on LSTM or GRU for the smart job shop production process

2020 ◽  
Vol 14 ◽  
pp. 174830262096239 ◽  
Author(s):  
Chuang Wang ◽  
Wenbo Du ◽  
Zhixiang Zhu ◽  
Zhifeng Yue
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Production Process ◽  
Real Time ◽  
Job Shop ◽  
Production Data ◽  
Time Data ◽  
Big Data Processing ◽  
The Real ◽  
Data Processing Method

With the wide application of intelligent sensors and internet of things (IoT) in the smart job shop, a large number of real-time production data is collected. Accurate analysis of the collected data can help producers to make effective decisions. Compared with the traditional data processing methods, artificial intelligence, as the main big data analysis method, is more and more applied to the manufacturing industry. However, the ability of different AI models to process real-time data of smart job shop production is also different. Based on this, a real-time big data processing method for the job shop production process based on Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) is proposed. This method uses the historical production data extracted by the IoT job shop as the original data set, and after data preprocessing, uses the LSTM and GRU model to train and predict the real-time data of the job shop. Through the description and implementation of the model, it is compared with KNN, DT and traditional neural network model. The results show that in the real-time big data processing of production process, the performance of the LSTM and GRU models is superior to the traditional neural network, K nearest neighbor (KNN), decision tree (DT). When the performance is similar to LSTM, the training time of GRU is much lower than LSTM model.

scholarly journals Big Data Usage Patterns in the Health Care Domain: A Use Case Driven Approach Applied to the Assessment of Vaccination Benefits and Risks

2014 ◽  
Vol 23 (01) ◽  
pp. 27-35 ◽  
Author(s):  
S. de Lusignan ◽  
S-T. Liaw ◽  
C. Kuziemsky ◽  
F. Mold ◽  
P. Krause ◽  
...  
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Predictive Analytics ◽  
Regulatory Compliance ◽  
Use Cases ◽  
Data Sources ◽  
Use Case ◽  
Crowd Sourcing ◽  
Big Data Processing

Summary Background: Generally benefits and risks of vaccines can be determined from studies carried out as part of regulatory compliance, followed by surveillance of routine data; however there are some rarer and more long term events that require new methods. Big data generated by increasingly affordable personalised computing, and from pervasive computing devices is rapidly growing and low cost, high volume, cloud computing makes the processing of these data inexpensive. Objective: To describe how big data and related analytical methods might be applied to assess the benefits and risks of vaccines. Method: We reviewed the literature on the use of big data to improve health, applied to generic vaccine use cases, that illustrate benefits and risks of vaccination. We defined a use case as the interaction between a user and an information system to achieve a goal. We used flu vaccination and pre-school childhood immunisation as exemplars. Results: We reviewed three big data use cases relevant to assessing vaccine benefits and risks: (i) Big data processing using crowd-sourcing, distributed big data processing, and predictive analytics, (ii) Data integration from heterogeneous big data sources, e.g. the increasing range of devices in the “internet of things”, and (iii) Real-time monitoring for the direct monitoring of epidemics as well as vaccine effects via social media and other data sources. Conclusions: Big data raises new ethical dilemmas, though its analysis methods can bring complementary real-time capabilities for monitoring epidemics and assessing vaccine benefit-risk balance.

Modeling Big Data Analytics with a Real-Time Executable Specification Language

2015 ◽  
pp. 289-312
Author(s):  
Amir A. Khwaja
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Data Analytics ◽  
Big Data Analytics ◽  
Exception Handling ◽  
Specification Language ◽  
Big Data Processing ◽  
Business Decisions ◽  
Concurrent Processes

Big data explosion has already happened and the situation is only going to exacerbate with such a high number of data sources and high-end technology prevalent everywhere, generating data at a frantic pace. One of the most important aspects of big data is being able to capture, process, and analyze data as it is happening in real-time to allow real-time business decisions. Alternate approaches must be investigated especially consisting of highly parallel and real-time computations for big data processing. The chapter presents RealSpec real-time specification language that may be used for the modeling of big data analytics due to the inherent language features needed for real-time big data processing such as concurrent processes, multi-threading, resource modeling, timing constraints, and exception handling. The chapter provides an overview of RealSpec and applies the language to a detailed big data event recognition case study to demonstrate language applicability to big data framework and analytics modeling.

Modeling Big Data Analytics with a Real-Time Executable Specification Language

Big Data ◽  
2016 ◽  
pp. 418-440
Author(s):  
Amir A. Khwaja
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Data Analytics ◽  
Big Data Analytics ◽  
Exception Handling ◽  
Specification Language ◽  
Big Data Processing ◽  
Business Decisions ◽  
Concurrent Processes

Big data explosion has already happened and the situation is only going to exacerbate with such a high number of data sources and high-end technology prevalent everywhere, generating data at a frantic pace. One of the most important aspects of big data is being able to capture, process, and analyze data as it is happening in real-time to allow real-time business decisions. Alternate approaches must be investigated especially consisting of highly parallel and real-time computations for big data processing. The chapter presents RealSpec real-time specification language that may be used for the modeling of big data analytics due to the inherent language features needed for real-time big data processing such as concurrent processes, multi-threading, resource modeling, timing constraints, and exception handling. The chapter provides an overview of RealSpec and applies the language to a detailed big data event recognition case study to demonstrate language applicability to big data framework and analytics modeling.

Trends and Technologies in Big Data Processing

2020 ◽  
pp. 17-42
Author(s):  
Amitava Choudhury ◽  
Kalpana Rangra
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Internet Of Things ◽  
Data Processing ◽  
Real Time ◽  
Computing Technology ◽  
Time Data ◽  
Big Data Processing ◽  
Real Time Data ◽  
Real Time Data Processing

Data type and amount in human society is growing at an amazing speed, which is caused by emerging new services such as cloud computing, internet of things, and location-based services. The era of big data has arrived. As data has been a fundamental resource, how to manage and utilize big data better has attracted much attention. Especially with the development of the internet of things, how to process a large amount of real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention to high performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. In this chapter, various big data processing techniques are discussed.

scholarly journals Urban Planning and Smart City Decision Management Empowered by Real-Time Data Processing Using Big Data Analytics

Sensors ◽  
2018 ◽  
Vol 18 (9) ◽  
pp. 2994 ◽  
Author(s):  
Bhagya Silva ◽  
Murad Khan ◽  
Changsu Jung ◽  
Jihun Seo ◽  
Diyan Muhammad ◽  
...  
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Smart City ◽  
Data Analytics ◽  
Smart Cities ◽  
Big Data Analytics ◽  
Time Data ◽  
Real Time Data

The Internet of Things (IoT), inspired by the tremendous growth of connected heterogeneous devices, has pioneered the notion of smart city. Various components, i.e., smart transportation, smart community, smart healthcare, smart grid, etc. which are integrated within smart city architecture aims to enrich the quality of life (QoL) of urban citizens. However, real-time processing requirements and exponential data growth withhold smart city realization. Therefore, herein we propose a Big Data analytics (BDA)-embedded experimental architecture for smart cities. Two major aspects are served by the BDA-embedded smart city. Firstly, it facilitates exploitation of urban Big Data (UBD) in planning, designing, and maintaining smart cities. Secondly, it occupies BDA to manage and process voluminous UBD to enhance the quality of urban services. Three tiers of the proposed architecture are liable for data aggregation, real-time data management, and service provisioning. Moreover, offline and online data processing tasks are further expedited by integrating data normalizing and data filtering techniques to the proposed work. By analyzing authenticated datasets, we obtained the threshold values required for urban planning and city operation management. Performance metrics in terms of online and offline data processing for the proposed dual-node Hadoop cluster is obtained using aforementioned authentic datasets. Throughput and processing time analysis performed with regard to existing works guarantee the performance superiority of the proposed work. Hence, we can claim the applicability and reliability of implementing proposed BDA-embedded smart city architecture in the real world.

scholarly journals Real-time Big Data Processing System to Improve Semiconductor Production Efficiency in Smart Factory

2018 ◽  
Vol 7 (3.33) ◽  
pp. 243
Author(s):  
Hyeopgeon Lee ◽  
Young-Woon Kim ◽  
Ki-Young Kim
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Production Line ◽  
Production Efficiency ◽  
Processing System ◽  
Data Processing System ◽  
Big Data Processing ◽  
Defect Rate ◽  
Semiconductor Production

Semiconductor production efficiency is closely related to the defect rate in the production process. The temperature and humidity control in the production line are very important because these affect the defect rate. So many smart factory of semiconductor production uses sensor. It is installed in the semiconductor process, which send huge amounts of data per second to a central server to carry out temperature and humidity control in each production line. However, big data processing systems that analyze and process large-scale data are subject to frequent delays in processing, and transmitted data are lost owing to bottlenecks and insufficient memory caused by traffic concentrated in the central server. In this paper, we propose a real-time big data processing system to improve semiconductor production efficiency. The proposed system consists of a production line collection system, task processing system and data storage system, and improves the productivity of the semiconductor manufacturing process by reducing data processing delays as well as data loss and discarded data.  

Sign in / Sign up

Export Citation Format

Share Document