Establishing the Advanced Disaster Reduction Management System by Fusion of Real-Time Disaster Simulation and Big Data Assimilation

2016 ◽  
Vol 11 (2) ◽  
pp. 164-174 ◽  
Author(s):  
Shunichi Koshimura ◽  
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Data Assimilation ◽  
Real Time ◽  
Big Data Analysis ◽  
Disaster Mitigation ◽  
Observation Data ◽  
Real Time Simulation ◽  
Time Simulation ◽  
Analysis Platform

A project titled “Establishing the advanced disaster reduction management system by fusion of real-time disaster simulation and big data assimilation,” was launched as Core Research for Evolutional Science and Technology (CREST) by the Japan Science and Technology Agency (JST). Intended to save as many lives as possible in future national crises involving earthquake and tsunami disasters, the project works on a disaster mitigation system of the big data era, based on cooperation of large-scale, high-resolution, real-time numerical simulations and assimilation of real-time observation data. The world’s most advanced specialists in disaster simulation, disaster management, mathematical science, and information science work together to create the world’s first analysis platform for real-time simulation and big data that effectively processes, analyzes, and assimilates data obtained through various observations. Based on quantitative data, the platform designs proactive measures and supports disaster operations immediately after disaster occurrence. The project was launched in 2014 and is working on the following issues at present.Sophistication and fusion of simulations and damage prediction models using observational big data: Development of a real-time simulation core system that predicts the time evolution of disaster effect by assimilating of location information, fire information, and building collapse information which are obtained from mobile terminals, satellite images, aerial images, and other new observation data in addition to sensing data obtained by the undersea high-density seismic observation network.Latent structure analysis and major disaster scenario creation based on a huge amount of simulation results: Development of an analysis and extraction method for the latent structure of a huge amount of disaster scenarios generated by simulation, and creation of severe scenarios with minimum “unexpectedness” by controlling disaster scenario explosion (an explosive increase in the number of predicted scenarios).Establishment of an earthquake and tsunami disaster mitigation big data analysis platform: Development of an earthquake and tsunami disaster mitigation big data analysis platform that realizes analyses of a huge number of disaster scenarios and increases in speed of data assimilation, and clarifies the requirements for operation of the platform as a disaster mitigation system.The project was launched in 2014 as a 5-year project. It consists of element technology development and system fusion, feasibility study as a next-generation disaster mitigation system (validation with/without introduction of the developed real-time simulation and big data analysis platform) in the affected areas of the Great East Japan Earthquake, and test operations in affected areas of the Tokyo metropolitan earthquake and the Nankai Trough earthquake.

Fusion of Real-Time Disaster Simulation and Big Data Assimilation – Recent Progress

2017 ◽  
Vol 12 (2) ◽  
pp. 226-232 ◽  
Author(s):  
Shunichi Koshimura ◽  
Keyword(s):  
Big Data ◽  
Data Assimilation ◽  
Real Time ◽  
Quantitative Data ◽  
Large Scale ◽  
Recent Progress ◽  
Big Data Analysis ◽  
Disaster Reduction ◽  
Public Agencies ◽  
Time Simulation

This paper reports the latest outcomes of the project “Establishing the Advanced Disaster Reduction Management System by Fusion of Real-time Disaster Simulation and Big Data Assimilation” that started in 2014. The objectives of targeting various kinds of damage due to earthquakes and tsunami, fusion of large-scale high-resolution numerical simulation, effective processing and analysis of big data from various observations, and data assimilation were achieved. The outcomes will be utilized to create the world’s first real-time simulation and big data analysis basis that would potentially assist with designing preliminary measures based on quantitative data and disaster responses to a disaster. Case studies using recent disasters were used in this endeavor and validation were performed. In the future, environments that rapidly provide information on possible damage situations in real time for public agencies, corporations, and citizens facing a catastrophic disaster in Japan will be developed by integrating these studies.

scholarly journals A monitoring method of semiconductor manufacturing processes using Internet of Things–based big data analysis

2017 ◽  
Vol 13 (7) ◽  
pp. 155014771772181 ◽  
Author(s):  
Seok-Woo Jang ◽  
Gye-Young Kim
Keyword(s):  
Time Series ◽  
Big Data ◽  
Data Analysis ◽  
Internet Of Things ◽  
Real Time ◽  
Semiconductor Manufacturing ◽  
Reference Model ◽  
Big Data Analysis ◽  
Monitoring Method ◽  
Manufacturing Equipment

This article proposes an intelligent monitoring system for semiconductor manufacturing equipment, which determines spec-in or spec-out for a wafer in process, using Internet of Things–based big data analysis. The proposed system consists of three phases: initialization, learning, and prediction in real time. The initialization sets the weights and the effective steps for all parameters of equipment to be monitored. The learning performs a clustering to assign similar patterns to the same class. The patterns consist of a multiple time-series produced by semiconductor manufacturing equipment and an after clean inspection measured by the corresponding tester. We modify the Line, Buzo, and Gray algorithm for classifying the time-series patterns. The modified Line, Buzo, and Gray algorithm outputs a reference model for every cluster. The prediction compares a time-series entered in real time with the reference model using statistical dynamic time warping to find the best matched pattern and then calculates a predicted after clean inspection by combining the measured after clean inspection, the dissimilarity, and the weights. Finally, it determines spec-in or spec-out for the wafer. We will present experimental results that show how the proposed system is applied on the data acquired from semiconductor etching equipment.

scholarly journals A Microservice-Based Big Data Analysis Platform for Online Educational Applications

2020 ◽  
Vol 2020 ◽  
pp. 1-13
Author(s):  
Kehua Miao ◽  
Jie Li ◽  
Wenxing Hong ◽  
Mingtao Chen
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Data Science ◽  
Modular Design ◽  
Science Research ◽  
Big Data Analysis ◽  
Research Field ◽  
Traditional Work ◽  
Educational Applications ◽  
Analysis Platform

The booming development of data science and big data technology stacks has inspired continuous iterative updates of data science research or working methods. At present, the granularity of the labor division between data science and big data is more refined. Traditional work methods, from work infrastructure environment construction to data modelling and analysis of working methods, will greatly delay work and research efficiency. In this paper, we focus on the purpose of the current friendly collaboration of the data science team to build data science and big data analysis application platform based on microservices architecture for education or nonprofessional research field. In the environment based on microservices that facilitates updating the components of each component, the platform has a personal code experiment environment that integrates JupyterHub based on Spark and HDFS for multiuser use and a visualized modelling tools which follow the modular design of data science engineering based on Greenplum in-database analysis. The entire web service system is developed based on spring boot.

scholarly journals BDAP: A Big Data Analysis Platform Based On Spark

2018 ◽  
Vol 1060 ◽  
pp. 012023
Author(s):  
Zhixiang Wang ◽  
Yao Bu ◽  
Demeng Bai ◽  
Bin Wu ◽  
Jiafeng Qin
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Big Data Analysis ◽  
Analysis Platform

Research of Big Data Processing Platform

2014 ◽  
Vol 484-485 ◽  
pp. 922-926
Author(s):  
Xiang Ju Liu
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Data Analysis ◽  
Network Architecture ◽  
Data Analytics ◽  
Big Data Analytics ◽  
Big Data Analysis ◽  
Computing Platform ◽  
Operational Characteristics ◽  
Analysis Platform

This paper introduces the operational characteristics of the era of big data and the current era of big data challenges, and exhaustive research and design of big data analytics platform based on cloud computing, including big data analytics platform architecture system, big data analytics platform software architecture , big data analytics platform network architecture big data analysis platform unified program features and so on. The paper also analyzes the cloud computing platform for big data analysis program unified competitive advantage and development of business telecom operators play a certain role in the future.

Towards an integrated regulatory framework for water and energy security with digital maps: a case study of the Guangdong-Hong Kong-Macao Greater Bay Area, China

2020 ◽  
Author(s):  
Pingyu Fan ◽  
Kwok Pan Chun ◽  
Ana Mijic ◽  
Daphne Ngar-Yin Mah
Keyword(s):  
Big Data ◽  
Information Retrieval ◽  
Data Analysis ◽  
Real Time ◽  
Energy Security ◽  
Regulatory System ◽  
Big Data Analysis ◽  
Regulatory Framework ◽  
Digital Maps ◽  
Bay Area

<p>Digital water and energy maps allow fast information retrieval, big data analysis and resources demand prediction for real time responses in 5-G networks. A regulatory systems framework is needed to enable and promote integrated actions grounded on map-based feedback information, to facilitate resources movements and knowledge transfer for water and energy security. At the same time, the proposed regulatory system needs to safeguard national security and personal privacy when general public and the private sectors have access to big databases.</p><p>The Guangdong-Hong Kong-Macao Greater Bay Area (GBA) in China is an initiative on regional economic development involving nine mainland cities and two Special Administrative Regions (SARs). As central policies cannot be efficiently executed in the whole regions, institutional fragmentation could be a prominent barrier to achieve regional water and energy optimum rather than individual city maxima for the water and energy nexus.</p><p>In this study, we propose a systems regulatory framework that integrates natural, urban and social systems across multiple scales in which the relevant laws, policies, decisions and actions are supported by digital maps. On a planning scale, our new regulatory system based on spatial map information promotes optimum uses of natural capitals and ecosystem services (ES). For linking different urban spatial processes on different scales, satellite images and Local Climate Zone (LCZ) maps are used to describe natural environment and urban characteristics from 200km to 10km resolutions for supporting land-use planning laws and estimating regional development carrying capacity to mitigate water and energy insecurity.</p><p>On an operational scale, smart meters and remote sensor systems provide real time water and energy information from a fast developing 5-G network for the proposed digital maps. Forecasted energy and water demands from the digital maps can be used for regional or local environment regulation reinforcement. Proposed spatial maps also improve transboundary collaboration by providing visualisation of legal targets and emission limits. Through digital maps, key agencies and sectors will have a capacity to share transboundary knowledge, information and responsibility, to foster smooth system flows in terms of culture, economy, policy and technology, by active participations and decentralized actions.</p><p>On an evaluation scale, open map information increases the transparency of legal targets and pollution limits. By rapid information retrieval and big data analysis from digital maps, regulators can assess the performance of water and energy security practices.</p><p>In summary, the proposed framework based on LCZ maps for the GBA can be applied to other rapidly developing regions with emerging 5-G networks. The integrated regulatory framework also guides water and energy security practices and transfer central policies to local actions by rapid information retrieval, big data analysis and prediction of demand for real time responses based on digital water and energy maps.</p><p></p><p></p><p></p><p></p><p></p>

Design and Implementation of Financial Big Data Visualization Analysis Platform

2019 ◽  
Vol 3 (1) ◽  
Author(s):  
Xi Chen ◽  
Bo Fan ◽  
Jie Zheng ◽  
Hongyan Cui
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Data Visualization ◽  
Production Efficiency ◽  
Big Data Analysis ◽  
Research Field ◽  
Visualization Analysis ◽  
Front End ◽  
Big Data Visualization ◽  
Analysis Platform

At present, it has become a hot research field to improve production efficiency and improve life experience through big data analysis. In the process of big data analysis, how to vividly display the results of the analysis is crucial. So, this paper introduces a set of big data visualization analysis platform based on financial field. The platform adopts the MVC system architecture, which is mainly composed of two parts: the background and the front end. The background part is built on the Django framework, and the front end is built with html5, css3, and JavaScript. The chart is rendered by Echarts. The platform can realize the classification of customers' savings potential through bank data, and make portraits of customers with different savings levels. The data analysis results can be dynamically displayed and interact wit

Sign in / Sign up

Export Citation Format

Share Document