scholarly journals Numerical behavior of NVIDIA tensor cores

2021 ◽  
Vol 7 ◽  
pp. e330
Author(s):  
Massimiliano Fasi ◽  
Nicholas J. Higham ◽  
Mantas Mikaitis ◽  
Srikara Pranesh
Keyword(s):  
Matrix Multiplication ◽  
Floating Point ◽  
Partial Sums ◽  
Hardware Accelerators ◽  
Floating Point Arithmetic ◽  
Mixed Precision ◽  
The Matrix ◽  
Custom Hardware ◽  
Intermediate Results ◽  
Point Arithmetic

We explore the floating-point arithmetic implemented in the NVIDIA tensor cores, which are hardware accelerators for mixed-precision matrix multiplication available on the Volta, Turing, and Ampere microarchitectures. Using Volta V100, Turing T4, and Ampere A100 graphics cards, we determine what precision is used for the intermediate results, whether subnormal numbers are supported, what rounding mode is used, in which order the operations underlying the matrix multiplication are performed, and whether partial sums are normalized. These aspects are not documented by NVIDIA, and we gain insight by running carefully designed numerical experiments on these hardware units. Knowing the answers to these questions is important if one wishes to: (1) accurately simulate NVIDIA tensor cores on conventional hardware; (2) understand the differences between results produced by code that utilizes tensor cores and code that uses only IEEE 754-compliant arithmetic operations; and (3) build custom hardware whose behavior matches that of NVIDIA tensor cores. As part of this work we provide a test suite that can be easily adapted to test newer versions of the NVIDIA tensor cores as well as similar accelerators from other vendors, as they become available. Moreover, we identify a non-monotonicity issue affecting floating point multi-operand adders if the intermediate results are not normalized after each step.

scholarly journals Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

2020 ◽  
Vol 476 (2243) ◽  
pp. 20200110
Author(s):  
Azzam Haidar ◽  
Harun Bayraktar ◽  
Stanimire Tomov ◽  
Jack Dongarra ◽  
Nicholas J. Higham
Keyword(s):  
Linear Systems ◽  
Numerical Stability ◽  
High Performance ◽  
Iterative Refinement ◽  
Floating Point ◽  
Double Precision ◽  
Floating Point Arithmetic ◽  
Mixed Precision ◽  
Scientific Simulations ◽  
Point Arithmetic

Double-precision floating-point arithmetic (FP64) has been the de facto standard for engineering and scientific simulations for several decades. Problem complexity and the sheer volume of data coming from various instruments and sensors motivate researchers to mix and match various approaches to optimize compute resources, including different levels of floating-point precision. In recent years, machine learning has motivated hardware support for half-precision floating-point arithmetic. A primary challenge in high-performance computing is to leverage reduced-precision and mixed-precision hardware. We show how the FP16/FP32 Tensor Cores on NVIDIA GPUs can be exploited to accelerate the solution of linear systems of equations Ax  =  b without sacrificing numerical stability. The techniques we employ include multiprecision LU factorization, the preconditioned generalized minimal residual algorithm (GMRES), and scaling and auto-adaptive rounding to avoid overflow. We also show how to efficiently handle systems with multiple right-hand sides. On the NVIDIA Quadro GV100 (Volta) GPU, we achieve a 4 × − 5 × performance increase and 5× better energy efficiency versus the standard FP64 implementation while maintaining an FP64 level of numerical stability.

scholarly journals Numerical algorithms for high-performance computational science

2020 ◽  
Vol 378 (2166) ◽  
pp. 20190066 ◽  
Author(s):  
Jack Dongarra ◽  
Laura Grigori ◽  
Nicholas J. Higham
Keyword(s):  
High Performance ◽  
Numerical Algorithms ◽  
Computational Science ◽  
Floating Point ◽  
Important Criterion ◽  
Data Movement ◽  
Floating Point Arithmetic ◽  
High Performance Computers ◽  
Point Arithmetic ◽  
Speed And Accuracy

A number of features of today’s high-performance computers make it challenging to exploit these machines fully for computational science. These include increasing core counts but stagnant clock frequencies; the high cost of data movement; use of accelerators (GPUs, FPGAs, coprocessors), making architectures increasingly heterogeneous; and multi- ple precisions of floating-point arithmetic, including half-precision. Moreover, as well as maximizing speed and accuracy, minimizing energy consumption is an important criterion. New generations of algorithms are needed to tackle these challenges. We discuss some approaches that we can take to develop numerical algorithms for high-performance computational science, with a view to exploiting the next generation of supercomputers. This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.

Fast and robust mesh arrangements using floating-point arithmetic

2020 ◽  
Vol 39 (6) ◽  
pp. 1-16
Author(s):  
Gianmarco Cherchi ◽  
Marco Livesu ◽  
Riccardo Scateni ◽  
Marco Attene
Keyword(s):  
Floating Point ◽  
Floating Point Arithmetic ◽  
Point Arithmetic

Floating-point arithmetic with 84-bit numbers

1964 ◽  
Vol 7 (1) ◽  
pp. 10-13 ◽  
Author(s):  
Robert T. Gregory ◽  
James L. Raney
Keyword(s):  
Floating Point ◽  
Floating Point Arithmetic ◽  
Point Arithmetic

scholarly journals Gaussian variant of Freivalds’ algorithm for efficient and reliable matrix product verification

2020 ◽  
Vol 26 (4) ◽  
pp. 273-284
Author(s):  
Hao Ji ◽  
Michael Mascagni ◽  
Yaohang Li
Keyword(s):  
False Positive ◽  
Measure Zero ◽  
Matrix Multiplication ◽  
Numerical Linear Algebra ◽  
Floating Point ◽  
Matrix Product ◽  
Computing Environment ◽  
Deterministic Algorithms ◽  
Product Verification ◽  
Point Arithmetic

AbstractIn this article, we consider the general problem of checking the correctness of matrix multiplication. Given three n\times n matrices 𝐴, 𝐵 and 𝐶, the goal is to verify that A\times B=C without carrying out the computationally costly operations of matrix multiplication and comparing the product A\times B with 𝐶, term by term. This is especially important when some or all of these matrices are very large, and when the computing environment is prone to soft errors. Here we extend Freivalds’ algorithm to a Gaussian Variant of Freivalds’ Algorithm (GVFA) by projecting the product A\times B as well as 𝐶 onto a Gaussian random vector and then comparing the resulting vectors. The computational complexity of GVFA is consistent with that of Freivalds’ algorithm, which is O(n^{2}). However, unlike Freivalds’ algorithm, whose probability of a false positive is 2^{-k}, where 𝑘 is the number of iterations, our theoretical analysis shows that, when A\times B\neq C, GVFA produces a false positive on set of inputs of measure zero with exact arithmetic. When we introduce round-off error and floating-point arithmetic into our analysis, we can show that the larger this error, the higher the probability that GVFA avoids false positives. Moreover, by iterating GVFA 𝑘 times, the probability of a false positive decreases as p^{k}, where 𝑝 is a very small value depending on the nature of the fault on the result matrix and the arithmetic system’s floating-point precision. Unlike deterministic algorithms, there do not exist any fault patterns that are completely undetectable with GVFA. Thus GVFA can be used to provide efficient fault tolerance in numerical linear algebra, and it can be efficiently implemented on modern computing architectures. In particular, GVFA can be very efficiently implemented on architectures with hardware support for fused multiply-add operations.

Sign in / Sign up

Export Citation Format

Share Document