GDS-based Mask Data Preparation Flow: Data Volume Containment by Hierarchical Data Processing

2002 ◽  
Author(s):  
Steffen F. Schulze ◽  
Pat LaCour ◽  
Peter D. Buck
Keyword(s):  
Data Processing ◽  
Hierarchical Data ◽  
Flow Data ◽  
Data Preparation ◽  
Data Volume

Software platform for automated preparation and analytical processing of heterogeneous datasets

2020 ◽  
pp. 15-20 ◽  
Author(s):  
E. D. Avedyan ◽  
I. V. Voronkov
Keyword(s):  
Data Processing ◽  
Analytical Data ◽  
Volume Growth ◽  
Software Platform ◽  
External Data ◽  
Heterogeneous Datasets ◽  
Data Volume ◽  
Processing Resources ◽  
Modular Platform ◽  
Network Technologies

Summary: the article proposes new software platform for automating the processes of preprocessing and marking up datasets with the aim of further solving analytical problems such as image classification and processing textual and parametric information using neural network technologies. The software platform uses modern technologies and combines a large number of methods in the form of a modular platform, which can be supplemented as the tasks of analytical data processing become more complicated. The need to develop such a software platform is dictated primarily by the fact that, given the current level of data volume growth, the actual transition to deep data analytics remains unattainable without such software platforms, since confidentiality, access to information and the use of external data processing resources are required.

scholarly journals Analysis of Tools for The Description and Analytical Processing of Enviromental Data

2020 ◽  
Vol 20 (2) ◽  
pp. 129-132
Author(s):  
Vugar Abdullayev ◽  
N.A. Ragimova N.A ◽  
V.H Abdullayev ◽  
T.K Askerov
Keyword(s):  
Data Processing ◽  
Environmental Monitoring ◽  
Analytical Data ◽  
Environmental Data ◽  
Volume Data ◽  
Star Schema ◽  
Data Processing Tool ◽  
Data Volume ◽  
Speed Up ◽  
Analytical Processing

The objects of the research are tools that support the description and analytical processing of environmental data requests. These tools are used for environmental monitoring. Analytical processing of environmental data is necessary for this monitoring by the persons concerned. Here, a star schema is used to describe the data. Analytical data processing tools are required for analysis and research of environmental data. The results of analytical processing of environmental data are used to speed up decision-making. This article also describes the structure of the analytical data processing tool. Therefore, one of the problem points is how to describe the data. For this purpose, an environmental data relay scheme is defined, and the data description is implemented in multidimensional cubes. Due to the growth of data volume, data processing is carried out using multi-dimensional visualization methods. In addition, a visual user interface has been created for analytically processing queries based on scale data. The result of this research is to find a method for describing environmental data. At the end of the research, a hypercube was obtained, with the help of which it was possible to structure environmental data and carry out analytical processing of them. To this end, environmental data have been described using a multi-dimensional visualization method. And OLAP technologies were used to carry out analytical processing of this data. OLAP technologies allow aggregate data to be used and presented as a hypercube. The results of the research can be used as a basis for an environmental information system that is used for environmental monitoring.

scholarly journals Stochastic calibration of radio interferometers

2020 ◽  
Vol 493 (4) ◽  
pp. 6071-6078 ◽  
Author(s):  
Sarod Yatawatta
Keyword(s):  
Data Processing ◽  
Radio Frequency ◽  
Computational Cost ◽  
Spatial Localization ◽  
Radio Frequency Interference ◽  
Radio Telescopes ◽  
Data Rates ◽  
Data Volume ◽  
Radio Transients ◽  
Processing Steps

ABSTRACT With ever-increasing data rates produced by modern radio telescopes like LOFAR and future telescopes like the SKA, many data-processing steps are overwhelmed by the amount of data that needs to be handled using limited compute resources. Calibration is one such operation that dominates the overall data processing computational cost; none the less, it is an essential operation to reach many science goals. Calibration algorithms do exist that scale well with the number of stations of an array and the number of directions being calibrated. However, the remaining bottleneck is the raw data volume, which scales with the number of baselines, and which is proportional to the square of the number of stations. We propose a ‘stochastic’ calibration strategy where we read only in a mini-batch of data for obtaining calibration solutions, as opposed to reading the full batch of data being calibrated. None the less, we obtain solutions that are valid for the full batch of data. Normally, data need to be averaged before calibration is performed to accommodate the data in size-limited compute memory. Stochastic calibration overcomes the need for data averaging before any calibration can be performed, and offers many advantages, including: enabling the mitigation of faint radio frequency interference; better removal of strong celestial sources from the data; and better detection and spatial localization of fast radio transients.

scholarly journals Muting the noise cone in near‐surface reflection data: An example from southeastern Kansas

Geophysics ◽  
1998 ◽  
Vol 63 (4) ◽  
pp. 1332-1338 ◽  
Author(s):  
Gregory S. Baker ◽  
Don W. Steeples ◽  
Matt Drake
Keyword(s):  
Data Processing ◽  
Seismic Data ◽  
Hydrocarbon Exploration ◽  
Seismic Reflection Profile ◽  
Seismic Data Processing ◽  
Near Surface ◽  
Surface Reflection ◽  
Reflection Data ◽  
Data Volume ◽  
Total Data

A 300-m near‐surface seismic reflection profile was collected in southeastern Kansas to locate a fault(s) associated with a recognized stratigraphic offset on either side of a region of unexposed bedrock. A substantial increase in the S/N ratio of the final stacked section was achieved by muting all data arriving in time after the airwave. Methods of applying traditional seismic data processing techniques to near‐surface data (200 ms of data or less) often differ notably from hydrocarbon exploration‐scale processing (3–4 s of data or more). The example of noise cone muting used is contrary to normal exploration‐scale seismic data processing philosophy, which is to include all data containing signal. The noise cone mute applied to the data removed more than one‐third of the total data volume, some of which contains signal. In this case, however, the severe muting resulted in a higher S/N ratio in the final stacked section, even though some signal could be identified within the muted data. This example supports the suggestion that nontraditional techniques sometimes need to be considered when processing near‐surface seismic data.

Sign in / Sign up

Export Citation Format

Share Document