Implications of the Regional Earthquake Likelihood Models test of earthquake forecasts in California

The Regional Earthquake Likelihood Models (RELM) test was the first competitive comparison of prospective earthquake forecasts. The test was carried out over 5 years from 1 January 2006 to 31 December 2010 over a region that included all of California. The test area was divided into 7682 0.1°x0.1° spatial cells. Each submitted forecast gave the predicted numbers of earthquakes <em>N_emi</em> larger than <em>M</em>=4.95 in 0.1 magnitude bins for each cell. In this paper we present a method that separates the forecast of the number of test earthquakes from the forecast of their locations. We first obtain the number <em>N_em</em> of forecast earthquakes in magnitude bin <em>m</em>. We then determine the conditional probability <em>λ_emi</em>=<em>N_emi/</em><em>N_em</em> that an earthquake in magnitude bin <em>m</em> will occur in cell <em>i</em>. The summation of <em>λ_emi</em> over all 7682 cells is unity. A random (no skill) forecast gives equal values of <em>λ_emi</em> for all spatial cells and magnitude bins. The <em>skill</em> of a forecast, in terms of the location of the earthquakes, is measured by the success in assigning large values of <em>λ_emi</em> to the cells in which earthquakes occur and low values of <em>λ_emi</em> to the cells where earthquakes do not occur. Thirty-one test earthquakes occurred in 27 different combinations of spatial cells <em>i</em> and magnitude bins <em>m</em>, we had the highest value of <em>λ_emi</em> for that <em>mi</em> cell. We evaluate the performance of eleven submitted forecasts in two ways. First, we determine the number of <em>mi</em> cells for which the forecast <em>λ_emi</em> was the largest, the best forecast is the one with the highest number. Second, we determine the mean value of <em>λ_emi</em> for the 27 <em>mi</em> cells for each forecast. The best forecast has the highest mean value of <em>λ_emi</em>. The success of a forecast during the test period is dependent on the allocation of the probabilities λemi between the mi cells, since the sum over the mi cells is unity. We illustrate the forecast distributions of <em>λ_emi</em> and discuss their differences. We conclude that the RELM test was successful in illustrating the choices required when a forecast of the location of a future earthquake is made.

Evaluating the RELM Test Results

We consider implications of the Regional Earthquake Likelihood Models (RELM) test results with regard to earthquake forecasting. Prospective forecasts were solicited forM≥4.95earthquakes in California during the period 2006–2010. During this period 31 earthquakes occurred in the test region withM≥4.95. We consider five forecasts that were submitted for the test. We compare the forecasts utilizing forecast verification methodology developed in the atmospheric sciences, specifically for tornadoes. We utilize a “skill score” based on the forecast scoresλfiof occurrence of the test earthquakes. A perfect forecast would haveλfi=1, and a random (no skill) forecast would haveλfi=2.86×10-3. The best forecasts (largest value ofλfi) for the 31 earthquakes had values ofλfi=1.24×10-1toλfi=5.49×10-3. The best mean forecast for all earthquakes wasλ̅f=2.84×10-2. The best forecasts are about an order of magnitude better than random forecasts. We discuss the earthquakes, the forecasts, and alternative methods of evaluation of the performance of RELM forecasts. We also discuss the relative merits of alarm-based versus probability-based forecasts.