QUALITY MONITORING
THE PRODUCT VALIDATION PROGRAM
Objective

The main goal of the Product Validation Group (PVG) is to structuralize the product validation activities of the European Countries involved in the project.

For all the H SAF products generated, the PVG is responsible:

- to monitor the progress in product quality as further development evaluating statistical scores and case study analysis on the base of comparison between satellite products and ground data;

- to provide validation service to end-users publishing on the H SAF web-page the statistical scores evaluated and the case studies analysed;

- to provide online quality control to end-users: quality is provided inside the product, with the resolution of IFOV, to be directly used and also included in automatic routines;

- to provide ground data service inside the project for algorithm calibration and validation activities;

- to investigate the H SAF product impact in end-user applications as Civil Protection activities for emergency management, precipitation event alerts, street monitoring, water balance evaluation, etc.



Participants

The cluster is coordinated by the Italian Civil Protection Department (DPC). The DPC is an expert user of near-real-time observations commonly used in the hydrological field, closely linked to national and local meteorological services. The DPC is actually involved as main user of national and international spatial projects.

The PVG is composed of experts from the National Meteorological and Hydrological Institutes of Austria, Belgium, Bulgaria, Finland, France, Germany, Hungary, Italy, Poland, Romania, Slovakia, and Turkey . Hydrologists, meteorologists, and precipitation, snow and soil moisture ground data experts, coming from these countries are involved in the product validation activities. ECMWF takes also part of the PVG.


Institutes Involved

The Validation activity is carried out by all participating countries through the continuous development of algorithms dedicated to the:

  • Quality filtering applied to ground field (from rain gauges, radar, snow and soil moisture measurements);
  • Interpolation techniques of in situ data;
  • Up-scaling of reference data vs satellite grid;
  • Evaluation of Statistical indexes.

All these algorithms are implemented in the framework of common codes used by all countries belonging to the PVG.

 

 

European Ground Data

The Precipitation PVG uses both rain gauge and radar data for validation of precipitation products.

The rain gauge network of PPVG is composed of approximately 8,400 nominal stations across 8 Countries.

The network of 8,404 rain gauges used for H-SAF precipitation products validation

A key characteristic of such networks is the distance between each rain gauge and the closest one, averaged over all the instruments considered in the network: it is inversely correlated with the rain gauge spatial density. Instruments number and average minimum distance are summarized below.

 

Country

Total number of gauges *

Average minimum distance (km)

Belgium

92

15.2

Bulgaria

123

25.2

Germany

2,299

12.9

Hungary

270

17.0

Italy

2,934

11.3

Poland

540

24.0

Slovakia

911

13.6

Turkey

1,235

26.5

* the number of rain gauges could vary from day to day due to operational efficiency within a maximum range of 10-15%.

Number and density of rain gauges within H SAF PPVG

 

 

Most of the gauges used in the National networks by the PPVG Partners are of the tipping bucket type, and hourly cumulated.

 

71 C-band radars are used by the H-SAF PPVG for assessing the satellite product accuracy. An inventory on radar data networks and products used in PPVG has pointed out that all the institutes involved in the PPVG declared the system are kept in a relatively good status and all of them apply some correction factors in their processing chain of radar data. Only the radar data, which passes the quality control of the owner Institute, are used by the PPVG for validation activities. Please note that the Validation procedure is the same for all countries of PPVG.

The networks of 71 C-band radars available in the H SAF PPVG

Instruments number and average minimum distance in each country are summarized in table.

 

Country

Total number of radar

Average minimum distance (km)

Belgium

1

-

Bulgaria

-

-

Germany

16

163

Hungary

4

190

Italy

22

141

Poland

8

186

Slovakia

4

137

Turkey

16

253

 

Number and density of radars used by the H SAF PPVG.

Statistical error description

The results of the common validation methodology are provided in form oflarge statistics (multi-categorical and continuous) and case studies analysis: these are complementary in assessing the accuracy of the implemented algorithms. Large statistics, in fact, helps in identifying existence of pathological behavior, while selected case studies are useful in identifying the roots of such behavior, when present.

a) Large statistical analysis

Statistical scores are evaluated for long (at least one year) time series. A referenced data set of one year was defined for all typologies of products (precipitation, soil moisture and snow) in order to compare the statistical scores obtained by different versions of the same product and to attest the quality improvement of the new versions.

The validation methodologies developed for precipitation, soil moisture and snow products are manly based on satellite product comparison with the ground data collected by the countries involved in the validation group.

A common validation methodology for all typologies of products (precipitation, soil moisture and snow) has been individuated defining:

- characteristics of ground data which can be used (quality control, spatial distribution, etc.);

- up-scaling techniques of ground data versus native grid of satellite products;

- data comparison methodologies (temporal and spatial);

- statistical scores (continuous and/or multi-categorical) valuation.

In order to assess the instantaneous and cumulated precipitation products for different precipitation regimes the statistical scores are evaluated for different precipitation classes as descripted in tables below.

Precipitation Class number

Precipitation Rate

0

PR < 1 mm/h

1

PR ≥ 1 mm/h

2

PR ≥ 5 mm/h

3

PR ≥ 10 mm/h

Classes for instantaneous rain rate class

Accumulated Precipitation Class number

Cumulated Rain

over 24 hours

0

CR < 1 mm

1

CR ≥ 1 mm

2

CR ≥ 5 mm

3

CR ≥ 10 mm

Classes for cumulated rain class

The impact of different background in the products performances are also taken into account. The statistical scores are calculated separately for land, sea and coast areas.

Continuous scores evaluated are here indicated:

where:

sat1 sat2 satk indicate the precipitation values estimated by satellite products;

ref1, ref2, refk, indicate the precipitation values observed by radar/rain gauges;

N indicates the number of observed/estimated precipitation data.

Multi categorical statistics are derived by the following contingency table:

Ground

yes

no

total

yes

hits

false alarms

forecast yes

Satellite

no

misses

correct negatives

forecast no

total

observed yes

observed no

total

where:

  • hit: Satk≥Rth and Obsk≥Rth
  • miss: Satk<Rth and Obsk≥R th
  • false alarm: miss: Satk≥Rth and Obsk <Rth
  • correct negative: miss: Satk<Rth and Obs k<Rth

Rth is the threshold between the “rain” and “no rain” conditions identified by a precipitation value of 0.25 mm/h for H01 new. rel., H02A and H03A, 5 mm/h for H15A and 1 mm/24h for H05A.

The scores evaluated from the contingency table are:

Score

Acronym

Range

Perfect score

Calculation

Probability Of Detection

POD

0 to 1

1

False Alarm Rate

FAR

0 to 1

0

Critical Success Index

CSI

0 to 1

1

b) Case study analysis

Each Institute produces case studies analysis based on its own knowledge and experience, but following a standard format based on:

  • data and products used;
  • comparison methodology;
  • plots (colors);
  • hydrological evaluation in accord with hydrological validation cluster (for some cases).

The results obtained by Quality Assessment Cluster are:

  • iscussed inside the VG and with product developers by email and annual meeting;
  • reported in the project documents;
  • published in the H-SAF web page section dedicated to the validation.

Validation reports will be produced any time the management and the scientific groups will decide that a new validation campaign is requested, or any time that the validation group decides that there is a need for further studies. It is envisaged that validation reports are issued before the Project Reviews and collected in the appropriate documentation.

RESULTS

Results of the comparison between H SAF Precipitation Products and European ground data on one full year of data are in this section summarized for each product as summarized in table below.

Products

H01 new rel. ; H02A ; H03A ; H05A 24h ; H15A

Covered period

01/06/2016 - 31/05/2017

Q.A. methods applied

Continuous statistics

ME, SD, MAE, MB, RMSE, FSE

Multi-categorical statistics

POD, FAR, CSI

Contributing countries

BE, BU, DE, HU, IT, PL, SK, TU

H01 new rel.

The validation procedure evaluates only liquid phase precipitation with quality “fair” or “good” (as indicated by the satellite product itself). Satellite Field Of Views not fully covered by ground data (or with percentage of coverage less than 50%) are discarded by the Q.A. procedure in order to increase the significance of the statistical sample.

The product requirement thresholds for PR ≥ 1 mm/h are reported on the left side of the table below. The accuracy obtained for H01 new rel. in the validation period with radar and rain gauge data is summarized.

Between target and optimal

Between threshold and target

Threshold exceeded by < 50 %

Threshold exceeded by ≥ 50 %

 

 

H01 new rel.

Annual average of FSE (%)

Precipitation Class

Requirement (FSE %)

radar   (Land)

radar     (Sea)

radar (Coast)

gauge (Land)

Overall

thresh

target

optimal

≥ 1 mm/h

200

150

100

132

200

205

131

144

 

The monthly accuracy, in terms of FSE%, is shown in figure below in comparison with radar (panel a) and rain gauge data (panel b). The background colours highlight the requirement thresholds. The horizontal black lines show the corresponding FSE values computed for the full period. The overall results are shown in panel c, while in the panel d is reported the percentage contribution of radar sea, land and coast data and for rain gauge data.

The Multi categorical scores for the overall data is here represented. The first column indicates the precipitation classes of the satellite product, while along the columns are reported the ground precipitation classes.

 

Overall

Multi-Categorical Statistics

 

[0 - 0.25[ mm/h

[0.25 - 1[ mm/h

[1 - 10[ mm/h

≥10 mm/h

[0 - 0.25[ mm/h

98%

64%

38%

12%

[0.25 - 1[ mm/h

1%

5%

4%

2%

[1 - 10[ mm/h

1%

31%

55%

62%

≥10 mm/h

0%

0%

3%

24%

 

H02A

The validation procedure evaluates only liquid phase precipitation with quality “fair” or “good” (as indicated by the satellite product itself). Satellite Field Of Views not fully covered by ground data (or with percentage of coverage less than 50%) are discarded by the Q.A. procedure in order to increase the significance of the statistical sample.

The product requirement thresholds for PR ≥ 1 mm/h are reported on the left side of the table below. The accuracy obtained for H02A in the validation period with radar and rain gauge data is summarized.

Between target and optimal

Between threshold and target

Threshold exceeded by < 50 %

Threshold exceeded by ≥ 50 %

 

 

 

H02A

Annual average of FSE (%)

Precipitation Class

Requirement (FSE %)

radar   (Land)

radar     (Sea)

radar (Coast)

gauge (Land)

Overall

thresh

target

optimal

≥ 1 mm/h

200

150

100

121

122

121

108

118

 

The monthly accuracy, in terms of FSE%, is shown in figure below in comparison with radar (panel a) and rain gauge data (panel b). The background colours highlight the requirement thresholds. The horizontal black lines show the corresponding FSE values computed for the full period. The overall results are shown in panel c, while in the panel d is reported the percentage contribution of radar sea, land and coast data and for rain gauge data.

The Multi categorical scores for the overall data is here represented. The first column indicates the precipitation classes of the satellite product, while along the columns are reported the ground precipitation classes.

 

Overall

Multi-Categorical Statistics

 

[0 - 0.25[ mm/h

[0.25 - 1[ mm/h

[1 - 10[ mm/h

≥10 mm/h

[0 - 0.25[ mm/h

93%

27%

11%

3%

[0.25 - 1[ mm/h

6%

48%

40%

18%

[1 - 10[ mm/h

2%

25%

48%

70%

≥10 mm/h

0%

0%

1%

9%

 

H03A

The accuracy of H03A has been evaluated by each country for ground precipitation rates detected above 1 mm/h. Only satellite FOVs covered by ground data for more than 50% are evaluated in the validation procedure in order to increase the significance of the statistical sample.

The H03A accuracy with respect to radar and rain gauge data is summarized.

Between target and optimal

Between threshold and target

Threshold exceeded by < 50 %

Threshold exceeded by ≥ 50 %

 

 

H03A

Annual average of FSE (%)

Precipitation Class

Requirement (FSE %)

radar   (Land)

radar     (Sea)

radar (Coast)

gauge (Land)

Overall

thresh

target

optimal

≥ 1 mm/h

200

150

100

142

140

143

123

138

 

The monthly accuracy, in terms of FSE%, is shown in figure below in comparison with radar (panel a) and rain gauge data (panel b). The background colours highlight the requirement thresholds. The horizontal black lines show the corresponding FSE values computed for the full period. The overall results are shown in panel c, while in the panel d is reported the percentage contribution of radar sea, land and coast data and for rain gauge data.

The Multi categorical scores for the overall data is here represented. The first column indicates the precipitation classes of the satellite product, while along the columns are reported the ground precipitation classes.

Overall

Multi-Categorical Statistics

 

[0 - 0.25[ mm/h

[0.25 - 1[ mm/h

[1 - 10[ mm/h

≥10 mm/h

[0 - 0.25[ mm/h

86%

52%

36%

15%

[0.25 - 1[ mm/h

8%

22%

22%

18%

[1 - 10[ mm/h

6%

26%

41%

62%

≥10 mm/h

0%

0%

1%

6%

 

H05A 24h

The accuracy of H05A has been evaluated by each country for ground precipitation accumulated above 1 mm over 24 hours. Only satellite FOVs covered by ground data for more than 50% of extension are evaluated in the validation procedure in order to increase the significance of the statistical sample.

The H05A 24h accuracy is here summarized.

Between target and optimal

Between threshold and target

Threshold exceeded by < 50 %

Threshold exceeded by ≥ 50 %

 

H05A 24h

Annual average of FSE (%)

Precipitation Class

Requirement (FSE %)

radar   (Land)

radar     (Sea)

radar (Coast)

gauge (Land)

Overall

thresh

target

optimal

≥ 1 mm over 24h

200

150

100

142

169

155

148

147

 

The monthly accuracy, in terms of FSE%, is shown in figure below in comparison with radar (panel a) and rain gauge data (panel b). The background colours highlight the requirement thresholds. The horizontal black lines show the corresponding FSE values computed for the full period. The overall results are shown in panel c, while in the panel d is reported the percentage contribution of radar sea, land and coast data and for rain gauge data.

The Multi categorical scores for the overall data is here represented. The first column indicates the precipitation classes of the satellite product, while along the columns are reported the ground precipitation classes.

 

Overall

Multi-Categorical Statistics

 

< 1 mm/24h

[1 - 10[ mm/24h

≥10 mm/24h

< 1 mm/24h

86%

32%

16%

[1 - 10[ mm/24h

12%

42%

41%

≥10 mm/24h

2%

16%

43%

 

H15A

The accuracy of H15A has been evaluated by each country for ground precipitation rates detected above 5 mm/h. Only satellite FOVs covered by ground data for more than 50% are evaluated in the validation procedure in order to increase the significance of the statistical sample.

The results are here summarized.

Between target and optimal

Between threshold and target

Threshold exceeded by < 50 %

Threshold exceeded by ≥ 50 %

 

H15A

Annual average of FSE (%)

Precipitation Class

Requirement (FSE %)

radar   (Land)

radar     (Sea)

radar (Coast)

gauge (Land)

Overall

thresh

target

optimal

≥ 5mm/h

200

170

120

191

171

164

160

182

 

The monthly accuracy, in terms of FSE%, is shown in figure below in comparison with radar (panel a) and rain gauge data (panel b). The background colours highlight the requirement thresholds. The horizontal black lines show the corresponding FSE values computed for the full period. The overall results are shown in panel c, while in the panel d is reported the percentage contribution of radar sea, land and coast data and for rain gauge data.

The Multi categorical scores for the overall data is here represented. The first column indicates the precipitation classes of the satellite product, while along the columns are reported the ground precipitation classes.

Overall

Multi-Categorical Statistics

 

[1 - 5[ mm/h

[5 - 10[ mm/h

≥10 mm/h

[1 - 5[ mm/h

66%

62%

56%

[5 - 10[ mm/h

13%

13%

14%

≥10 mm/h

21%

25%

31%