Quality Assessment

Objective

The main goal of the Product Validation Group (PVG) is to structure the product validation activities of the European Countries involved in the project.

For all the H SAF products generated, precipitation, soil moisture and snow cover products, the PVG is responsible:

- to monitor the progress in product quality as further development evaluating statistical scores and case study analysis on the base of comparison between satellite products and ground data;

- to provide validation service to end-users publishing on the H SAF web-page the statistical scores evaluated and the case studies analysed;

- to provide online quality control to end-users: quality is provided inside the product, with the resolution of IFOV for products over European area, to be directly used and also included in automatic routines; with the resolution of 0.5° x 0.5° for products over MSG full-disk area;

- to provide ground data service inside the project for algorithm calibration and validation activities;

- to investigate the H SAF product impact in end-user applications as Civil Protection activities for emergency management, precipitation event alerts, street monitoring, water balance evaluation, etc.

Participants

The cluster is coordinated by the Italian Civil Protection Department (DPC). The DPC is an expert user of near-real-time observations commonly used in the hydrological field, closely linked to national and local meteorological services. The DPC is actually involved as main user of national and international spatial projects.

The PVG is composed of experts from the National Meteorological and Hydrological Institutes of Austria (ZAMG), Belgium (IRM), Bulgaria (NIMH), Finland (FMI), France (Meteo France), Germany (BfG), Hungary (OMSZ), Italy (ITAF MET, DPC, UniBo, CNR-IRPI, CIMA), Poland (IMWM), Slovakia (SHMU), and Turkey (ITU, METU, AU) . Hydrologists, meteorologists, and precipitation, snow and soil moisture ground data experts, coming from these countries are involved in the product validation activities. ECMWF takes also part of the PVG.

Precipitation Products
The PPVG (Precipitation Product Validation Group) evaluates yearly the operational precipitation products (PP) to guarantee a continuous high-level quality standard (Mugnai et al., 2013). The actually operational PP are:

  • P-IN-SSMIS: Precipitation rate at ground by MW conical scanners
  • P-IN-MHS: Precipitation rate at ground by MW cross track scanners
  • P-IN-SEVIRI: Precipitation rate at ground by GEO/IR supported by LEO/MW
  • P-AC-SEVIRI: Accumulated precipitation at ground by blended MW and IR

The first three products estimate the instantaneous precipitation, in particular: the first and the second ones are based on PMW sensors, SSMIS and MHS respectively, while the third product uses the blending technique to produce instantaneous rain rate estimates every 15’ intersecting pixels from SEVIRI and LEO MW sensors. Finally, the P-AC-SEVIRI product returns maps of accumulated precipitation in the previous 3, 6, 12 and 24 hours every 3 hours performing time integration of frequent precipitation data available as products P-IN -SEVIRI. The validation process is based on the comparison with precipitation measurements by ground station over the European area and with precipitation estimates by radar aboard satellite platforms over hemispherical or global coverage. All PP cover the MSG Full-Disk area therefore the comparison is performed over Europe, Africa, part of Atlantic Ocean. Both methodologies are below descripted in detail.
Validation methodology over European area
From the beginning of the project it was clear the importance to define a common validation procedure in order to make the results obtained by several institutes comparable and to better understand their meanings (Puca et al., 2014). The main steps of this methodology have been identified during the development phase inside the validation group, in collaboration with the product developers, and with the support of ground data experts. This common procedure has given rise to a single common code for all members of the PPVG, named Unique Common Code (UCC) (Petracca et al., 2018). This common validation methodology is based on ground data (radar and rain gauge) comparisons to produce large statistic (multi-categorical and continuous), and case study analysis. Both components (large statistic and case study analysis) are considered complementary in assessing the accuracy of the implemented algorithms. Large statistics helps in identifying existence of pathological behaviour, selected case studies are useful in identifying the roots of such behaviour, when present. The main steps of the validation procedure (Rinollo et al., 2013) respect to ground data are:

  • ground data error analysis: radar and rain gauge (Porcù et al., 2014; Vulpiani et al, 2008; 2012; 2014);
  • point measurements (rain gauge) spatial interpolation (Pignone et al., 2010);
  • up-scaling of radar data versus native product satellite grid;
  • temporal comparison of precipitation products (satellite and ground);
  • statistical scores (continuous and multi-categorical) evaluation;
  • case study analysis.

EUROPEAN GROUND DATA
The rain gauge network of PPVG is composed of approximately 8,400 nominal stations across 8 Countries.
The Precipitation PVG uses both rain gauge and radar data for validation of precipitation products.
A key characteristic of such networks is the distance between each rain gauge and the closest one, averaged over all the instruments considered in the network: it is inversely correlated with the rain gauge spatial density. Instruments number and average minimum distance are summarized below.
Number and density of rain gauges within H SAF PPVG

Country

Total number of gauges *

Average minimum distance (km)

Belgium

92

15.2

Bulgaria

123

25.2

Germany

2,299

12.9

Hungary

270

17.0

Italy

2,934

11.3

Poland

540

24.0

Slovakia

911

13.6

Turkey

1,235

26.5

* the number of rain gauges could vary from day to day due to operational efficiency within a maximum range of 10-15%.

71 C-band radars are used by the H-SAF PPVG for assessing the satellite product accuracy. An inventory on radar data networks and products used in PPVG has pointed out that all the institutes involved in the PPVG declared the system are kept in a relatively good status and all of them apply some correction factors in their processing chain of radar data.
Number and density of radars used by the H SAF PPVG.

Country

Total number of gauges *

Average minimum distance (km)

Belgium

1

-

Bulgaria

-

-

Germany

16

163

Hungary

4

190

Italy

22

141

Poland

8

186

Slovakia

4

137

Turkey

16

253

Validation methodology over MSG Full-Disk area

The validation over the MSG full-disk area is performed by DPC for all HSAF precipitation products. The methodology was developed in communion with European experts belonging to the PVG. The comparison is performed respect to DPR products freely available by GPM website. The main steps of the procedure are:

  • regridding of DPR and H-SAF data versus a regular 0.5° equi-distance grid;
  • temporal and spatial matching between precipitation products;
  • statistical scores (continuous and multi-categorical) evaluation;
The methodology produces large statistic (multi-categorical and continuous) scores.

DPR products used as reference for comparison
The spatial coverage of both rain gauge and ground radar networks is not suitable to detect precipitation on a global scale. At the contrary, satellite observations provide estimates on a synoptic scale, although there are some issues related to their accuracy. It was discussed in the Visiting Associated analysis (Sebastianelli, 2016) in comparison with ground radar network. The DPR is a Dual frequency Precipitation Radar located on board of the GPM Core Observatory.
It uses the Ka (~35 GHz) and Ku bands (~13 GHz) to construct three-dimensional precipitation and drop size distribution maps. The GPM Core Observatory flies in a non-sun-synchronous orbit at 65° inclination to cover a larger latitudinal extension with respect to the TRMM orbit, which extended from 35°S to 35°N. Both Ku- and Ka-band radars perform cross-track type scans (perpendicular to the direction of the satellite motion) estimating the precipitation during the day and the night over land and ocean. The Ku-band radar performs a normal scan (NS) acquisition mode that is composed by 49 footprints (IFOV) of 5 km in diameter. In fact, away from the scanning center, footprints tend to widen and overlap (edge effects) because of a geometric distortion. The term swath indicates the width of each scan of 245 km. The range resolution is 250 m. The Ka-band radar can perform a matched scan (MS) or a high sensitivity scan (HS) acquisition mode. The MS footprints match the central 25 footprints of the Ku-band and the range resolution is 250 m. Therefore, MS scan is composed of 25 footprints of 5 km in diameter and the swath is 125 km. When Ka-band radar operates in HS mode footprints are interlaced with the matched beams, the range resolution is 500 m and there are 24 footprints along a swath. Figure shows the different DPR scanning modes with respect the flight direction. It must to be noted that the range resolution is different from the spatial resolution. In fact, the sampling is carried out for 19 km above the sea level and then along the vertical there are many footprints of 250 m height (range resolution). In addition, footprint size decreases as the sampling height increases due to the antenna aperture. The sampling distance between the centers of two adjacent footprints is 5.2 km, and it is constant throughout the scan to the edges. Apart the other problems which affects the DPR estimates, the main issues deal with the attenuation and the ground clutter. The K-band radar estimates are affected by attenuation when they sample through very intense precipitations (convective cells). Ground clutter is a non-meteorological echo which causes an overestimate of precipitations. DPR products (level 2A) referred to single frequency radar are 2A-Ku, 2A-Ka-MS and 2A-Ka-HS, as showed in figure below.
Three different DPR products combining Ka and Ku bands precipitation rate estimates (prEs) also exist depending on the IFOV to which data are referred. The IFOV can be related to the NS Ku-band, or to the MS or HS Ka-band, and the corresponding DPR products for prEs are 2A-DPR-NS, 2A-DPR-MS and 2A-DPR-HS, respectively. Results of Visiting Associated activity highlight as 2A-DPR-NS product performs better with respect to ground-based radar estimates. For this reason, the prEs by 2A-DPR-NS product (hereafter also referred as DPR-NS) was used as precipitation reference to validate the H-SAF satellite precipitation products.
Statistical error description
The results of the common validation methodology are provided in form of large statistics (multi-categorical and continuous) and case studies analysis: these are complementary in assessing the accuracy of the implemented algorithms. Large statistics, in fact, helps in identifying existence of pathological behavior, while selected case studies are useful in identifying the roots of such behavior, when present. a) Large statistical analysis Statistical scores are evaluated for long (at least one year) time series. A referenced data set of one year was defined for all typologies of products (precipitation, soil moisture and snow) in order to compare the statistical scores obtained by different versions of the same product and to attest the quality improvement of the new versions. The validation methodologies developed for precipitation, soil moisture and snow products are manly based on satellite product comparison with the ground data collected by the countries involved in the validation group. A common validation methodology for all typologies of products (precipitation, soil moisture and snow) has been individuated defining:

  • characteristics of ground data which can be used (quality control, spatial distribution, etc.);
  • up-scaling techniques of ground data versus native grid of satellite products;
  • data comparison methodologies (temporal and spatial);
  • statistical scores (continuous and/or multi-categorical) valuation.

In order to assess the precipitation products for different precipitation regimes the statistical scores are evaluated for different precipitation classes as descripted in table below.
Classes for instantaneous rain rate class.

Precipitation Rate Classes

1 2 3
≥ 1 mm/h ≥ 5 mm/h ≥ 10 mm/h
The impact of different background in the products performances are also taken into account. The statistical scores are calculated separately for land, sea and coast areas. The Precipitation Product Validation Leader collects all validation results as computed by European institutes, verifies the consistency of these results and evaluates the monthly and seasonal common statistical results as reported in OR and PVR documents. Continuous statistics are provided for each month and season of assessment. The main statistical scores are here indicated.
the index “k” represents the spatial and temporal grid point at the scale of the common reference grid. “obs” and “sat” represent the observed (or estimated) rainfall respectively by reference and satellite sensors. The FSE% score represents the accuracy for all precipitation products. The User requirements are recorded in table:
Multi categorical statistics are derived by the following contingency table:
where:

  • hit: Satk≥Rth and Obsk≥
  • miss: Satk< and Obsk≥Rth
  • false alarm: miss: Satk≥Rth and Obsk<Rth
  • correct negative: miss: Satk<Rth and Obsk<Rth

Rth is the threshold between the “rain” and “no rain” conditions identified by a precipitation value of 0.25 mm/h for H01 new. rel., H02B and H03B and 1 mm/24h for H05B. The scores evaluated from the contingency table are:
b) Case study analysis Each Institute produces case studies analysis based on its own knowledge and experience, but following a standard format based on:

  • data and products used;
  • comparison methodology;
  • plots;
  • hydrological evaluation in accord with hydrological validation cluster (for some cases).

The results obtained by Quality Assessment Cluster are:

  • discussed inside the VG and with product developers by email and annual meeting;
  • reported in the project documents;
  • published in the H-SAF web page section dedicated to the validation results.

Validation reports will be produced any time the management and the scientific groups will decide that a new validation campaign is requested, or any time that the validation group decides that there is a need for further studies. It is envisaged that validation reports are issued before the Project Reviews and collected in the appropriate documentation.
References
Mugnai, A., Casella, D., Cattani, E., Dietrich, S., Laviola, S., Levizzani, V., Panegrossi, G., Petracca, M., Sanò, P., Di Paola, F., Biron, D., De Leonibus, L., Melfi, D., Rosci, P., Vocino, A., Zauli, F., Pagliara, P., Puca, S., Rinollo, A., Milani, L., Porcù, F., and Gattari, F., Precipitation products from the hydrology SAF, Nat. Hazards Earth Syst. Sci., 13, 1959–1981, 2013 www.nat-hazards-earth-systsci.net/13/1959/2013/ doi:10.5194/nhess-13-1959-2013
Petracca, M., D'Adderio, L.P., Porcù, F., Vulpiani, G., Sebastianelli, S., Puca, S., Validation of GPM Dual-frequency Precipitation Radar (DPR) rainfall products over Italy, Journal of Hydrometeorology, 19, 907-925, 2018, doi: 10.1175/JHM-D-17-0144.1
Pignone, F., N. Rebora, F. Silvestro, and F: Castelli, 2010: GRISO (Generatore Random di 873 Interpolazioni Spaziali da Osservazioni incerte)-Piogge, Relazione delle attività del I anno 874 inerente la Convenzione 778/2009 tra Dipartimento di Protezione Civile e Fondazione CIMA 875 (Centro Internazionale in Monitoraggio Ambientale), report no. 272/2010, p. 353, 2010.
Porcù F., L. Milani, M. Petracca, 2014: On the uncertainties in validating satellite 878 instantaneous rainfall estimates with raingauge operational network, Atmospheric Research, 879 http://dx.doi.org/10.1016/j.atmosres.2013.12.007.
Puca, S., Baguis, P., Campione, E., Ertürk, A., Gabellani, S., Iwański, R., Jurašek, M., Kaňák, J., Kerényi, J., Koshinchanov, G., Kozinarova, G., Krahe, P., Łapeta, B., Lábó, E., Milani, L., Okon, L., Öztopal, A., Pagliara, P., Pignone, F., Porcù, F., Rachimow, C., Rebora, N., Rinollo, A., Roulin, E., Sönmez, İ., Toniazzo, A., Vulpiani, G., Biron, D., Casella, D., Cattani, E., Dietrich, S., Laviola, S., Levizzani, V., Melfi, D., Mugnai, A., Panegrossi, G., Petracca, M., Sanò, P., Zauli, F., Rosci, P., De Leonibus, L. “The validation service of the hydrological SAF geostationary and polar satellite precipitation products”, Nat. Hazards Earth Syst. Sci., 14, 2014 www.nat-hazards-earth-systsci.net/14/871/2014/ doi:10.5194/nhess-14-871-2014
Rinollo, A., Vulpiani, G., Puca, S., Pagliara, P., Kaˇnák, J., Lábó, E., Okon, L’., Roulin, E., Baguis, P., Cattani, E., Laviola, S., and Levizzani, V.: Definition and impact of a quality index for radar-based reference measurements in the H-SAF precipitation product validation, Nat. Hazards Earth Syst. Sci., 13, 2695–2705, doi:10.5194/nhess-13-2695-2013, 2013.
Sebastianelli S., Potentials and limitations of the use of GPM-DPR for validation of H-SAF precipitation products: study over the italian territory, H_AS16_03 DPC/CNR-ISAC 2016, http://hsaf.meteoam.it/documents/visiting-scientist/Final_Report_Stefano_Sebastianelli.pdf
Vulpiani, G., P. Tabary, J. P. D. Chatelet, and F. S. Marzano, 2008: Comparison of advanced radar polarimetric techniques for operational attenuation correction at C band. J. Atmos. Oceanic Technol., 25, 1118–1135, https://doi.org/10.1175/2007JTECHA936.1.
Vulpiani, G.,M. Montopoli, L. D. Passeri, A. Gioia, P. Giordano, and F. S. Marzano, 2012: On the use of dual-polarized C-band radar for operational rainfall retrieval in mountainous areas. J. Appl. Meteor. Climatol., 51, 405–425, https://doi.org/10.1175/JAMC-D-10-05024.1
Vulpiani, G., A. Rinollo, S. Puca, and M. Montopoli, 2014: A quality-based approach for radar rain field reconstruction and the H-SAF precipitation products validation. Proc. Eighth European Radar Conf., Garmish-Partenkirchen, Germany, ERAD, Ab- stract 220, 6 pp., http://www.pa.op.dlr.de/erad2014/programme/ExtendedAbstracts/220_Vulpiani.pdf
Soil Moisture Products
In the framework of the H SAF project several soil moisture products, with different timeliness (e.g. NRT, offline, data records), spatial resolution, format (e.g. time series, swath orbit geometry) or the representation of the water content in various soil layers (e.g. surface, root-zone), are generated on a regular basis and distributed to users. The Soil Moisture Products Validation Group evaluates yearly the operational soil moisture products (SMP) to guarantee a continuous high-level quality standard.
The actually operational SMP are:
  • Surface soil moisture (SSM) from the radar backscattering coefficients measured by the Advanced Scatterometer (ASCAT) on-board the series of Metop satellite:
    • SSM ASCAT-A(-B)(-C) NRT O12.5(O25): Metop-A(-B, -C) ASCAT NRT SSM orbit geometry 12.5 (25) km sampling
    • SSM ASCAT DR2019TS12.5: Metop ASCAT data record SSM time series 12.5 km sampling for the period 2007-01-01 to 2018-12-31
    • SSM ASCAT DR2019 EXT TS12.5: Metop ASCAT data record extension SSM time series 12.5 km sampling for the year 2019
    • SSM-ASCAT-NRT-DIS: Disaggregated Metop ASCAT NRT SSM at 1 km NRT
  • Root zone soil moisture (RZSM) product from the assimilation of scatterometer Surface Soil Moisture (SSM) in a ECMWF Land Data Assimilation System (LDAS)
    • SM-DAS-2: Soil Wetness Profile Index in the roots region retrieved by Metop ASCAT surface wetness scatterometer assimilation method
    • SM-DAS-3: Soil Wetness Index in the roots region by ERS/SCAT and Metop ASCAT-A scatterometer assimilation in a Land Data Assimilation System - Offline product
The first 5 products are Level 2 surface soil moisture product derived from the radar backscattering coefficients measured by the Advanced Scatterometer (ASCAT) on-board the series of Metop satellites using a change detection method, developed at the Research Group Remote Sensing, Department for Geodesy and Geoinformation (GEO), Vienna University of Technology (TU Wien). In the TU Wien soil moisture retrieval algorithm, longterm Scatterometer data are used to model the incidence angle dependency of the radar backscattering signal. Knowing the incidence angle dependency, the backscattering coefficients are normalized to a reference incidence angle. Finally, the relative soil moisture data ranging between 0% and 100% are derived by scaling the normalized backscattering coefficients between the lowest/highest values corresponding to the driest/wettest soil conditions (more information can be found on the Product User manual PUM).
More information on the soil moisture retrieval algorithm can be found in the Algorithm Theoretical Baseline Document (ATBD).
The other two products are index of root zone soil moisture. In the soil moisture assimilation system of ECMWF, the surface observation from ASCAT is propagated towards the roots region down to 2.89 m below surface, providing estimates for 4 layers (thicknesses 0.07, 0.21, 0.72 and 1.89 m).
The ECMWF model generates soil moisture profile information according to the Hydrology Tiled ECMWF Scheme for Surface Exchanges over Land (HTESSEL). SM-DAS-2 is available at a 24-hour time step, with a global daily coverage at 00:00 UTC. SM-DAS-2 is produced in a continuous way in order to ensure the time series consistency of the product (and also to provide values when there is no satellite data, from the model propagation).
Validation methodology

The validation of space-based soil moisture products is typically based on the comparison of related space-based soil moisture data sets, in situ data, land surface models or other parameters related to surface soil moisture (e.g. precipitation). In the framework of the HSAF project the Soil Moisture Products are yearly validated in two ways:

  1. Comparing the satellite products against in situ soil moisture measurements where available.
  2. With a triple collocation analysis using global dataset of soil moisture.
Validation methodology using ground data

The validation with ground data is performed through the comparison with in situ sensors located in Europe (e.g., SMOSMANIA) and worldwide. The performance metrics used for the evaluation of the accuracy of the test data set are the correlation coefficient, CC, the Root Mean Square Difference, RMSD, and the BIAS or mean error (satellite minus in situ), BIAS. Before the comparison, satellite data are rescaled to the same range of in situ observations through a linear approach.

Validation methodology using triple collocation analysis

The triple collocation analysis (TCA) is a statistical tool used for error characterization, first introduced by Stoffelen in 1988. It simultaneously estimates the error structure of three spatially and temporally collocated data sets θ1, θ2 and θ3, which are linearly related to the hypothetical (unknown) truth θ and contaminated by additive zero-mean noise, which must not be correlated with the signal or between sets (Scipal et al., 2010):

αi and βi are systematic additive and multiplicative biases of data set i with respect to the true state Θ, and εi represents zero-mean random noise. The underlying assumptions of the error model are: (i) linearity between the true signal and the observations, (ii) signal and error stationarity, (iii) independence between the errors and the signal, and (iv) independence between the errors of θ1, θ2 and θ3, (zero error cross-correlation). The mean squared random error (MSE) of all three data sets is estimated individually by TCA. The error variance can also be expressed in a normalised form as Signal-to-Noise Ratio (SNR). This allows to compare the error variances between the data sets. SNR is usually given in decibel, which can be easily interpreted: a SNR value of zero means that the signal variance is equal to the noise variance, and every 3 dB increase implies a doubling of the signal variance compared to the noise variance.

The soil moisture retrieval algorithm is applied on a global basis (i.e. for each grid point over land), but in certain situations (dense forest, snow cover, frozen soil, open water or topographic complex area dominating the satellite footprint), the retrieval of meaningful soil moisture values is not always possible. Nonetheless, soil moisture is computed and additional information (e.g. on the soil state) is provided to mask invalid soil moisture measurements. The triple collocation analysis is performed globally and the statistics are provided for committed and non-committed areas. The committed area represents a restricted geographical region with high confidence in the successful retrieval of surface soil moisture information from Metop ASCAT. The area is limited to low and moderate vegetation regimes, unfrozen and no snow cover, low to moderate topographic variations, as well as no wetlands and coastal areas. All the statistical results of the TC analysis are reported both on global scale and committed areas.

In green the committed areas (a restricted geographical region with high confidence in the successful retrieval of surface soil moisture information from Metop ASCAT)

The validation is performed globally on the WARP 5 grid (v2.3). As reference data set the NOAH GLDAS land surface model (v2.1) and the passive CCI soil moisture product (v04.3) are used. The NOAH model provided by the Global Land Data Assimilation System (GLDAS) contains atmospheric and land surface parameters stored on a regular global grid (spacing 0.25◦). From 2000-ongoing, the GLDAS NOAH version 2.1 data set provides soil moisture at a 3-hourly temporal resolution (daily at 00:00, 03:00, 06:00, 09:00, 12:00, 15:00, 18:00 and 21:00 UTC) (Roddel et al., 2004). The data is publicly available at GES DISC (Goddard Earth Sciences Data and Information Services Center). The first soil moisture layer (0.00 - 0.07 m) of NOAH GLDAS is used for the validation.
The Soil Moisture CCI project is part of the ESA Programme on Global Monitoring of Essential Climate Variables (ECV), better known as the Climate Change Initiative (CCI). The CCI Passive Soil Moisture product (v4.3) was generated by the VU University Amsterdam in collaboration with NASA based on passive microwave observations from Nimbus 7 SMMR, DMSP SSM/I, TRMM TMI, Aqua AMSR-E, Coriolis WindSat, and GCOM-W1 AMSR2. The ECV soil moisture production system generates soil moisture at a spatial resolution of approximately 25 × 25 km for top < 2 cm of the soil, expressed in volumetric units (m3m-3). The soil temperature and snow depth information are used for filtering for non-frozen (soil temperature > 4°) and snow-free (snow depth = 0) time periods.

The results of the validation in H SAF are reported as Signal-to-Noise Ratio (SNR) and Pearson’s correlation coefficient (R). Pearson’s R is the most commonly used measure of linear dependencies between two data sets (Helsel et al., 2002, Wilcox, 2009). Three threshold values, optimal, target and threshold, have been selected to evaluate the performances of the products. For further information on validation and specific results of each product refer to the Product Validation Reports (PVR). Product validated with tripe collocation are SSM ASCAT-A(-B)(-C) NRT O12.5(O25), SSM ASCAT DR2019 EXT TS12.5, SM-DAS-2 and SM-DAS-3.

The boxplots indicate the distribution of the quality benchmarks globally and just for the committed area. A percentage of locations exceeding each of the three thresholds is indicated as well.

SNR of the test data set.

Exemple of Pearson Correlation coefficent for a test data set.

Exemple of Pearson Correlation coefficent for a test data set for committed areas.

D. Helsel and R. M. Hirsch, Statistical Methods in Water Resources, 2002, no. Book 4.

M. Rodell, P. R. Houser, U. Jambor, J. Gottschalck, K. Mitchell, C.-J. Meng, K. Arsenault, B. Cosgrove, J. Radakovich, M. Bosilovich, J. K. Entin*, J. P. Walker, D. Lohmann, and D. Toll, “The Global Land Data Assimilation System,” Bulletin of the American Meteorological Society, vol. 85, pp. 381–394, Mar. 2004.

K. Scipal, W. Dorigo, and R. deJeu, “Triple collocation: A new tool to determine the error structure of global soil moisture products.” IEEE, Jul. 2010, pp. 4426–4429.

A. Stoffelen, “Toward the true near-surface wind speed: Error modeling and calibration using triple collocation,” Journal of Geophysical Research Oceans, vol. 103, pp. 7755–7766, 1998.

R. R. Wilcox, Basic statistics: understanding conventional methods and modern insights, 2009

Snow cover Products
DPC and the international validation group does yearly the Operational Reviews (OR) of the Snow Products of H-SAF Satellite products. All four of the Snow Cluster products are now operational and the continuous validation guarantees that they will satisfy the quality standards in the future. Six countries from H-SAF consortium Ground data from National Weather Stations of: Finland, Turkey, Italy, Poland, Germany and Belgium.

The products are:

  • SE-E-SEVIRI: Snow detection (snow mask) by VIS/IR radiometry;
  • WS-E: Snow status (dry/wet) by MW radiometry;
  • FSC-E: Effective snow cover by VIS/IR radiometry;
  • SWE-E: Snow water equivalent by MW radiometry;
Each of the product have a dedicated validation procedure:
Products SE-E-SEVIRI, WS-E and SWE-E are validated using a methodology based on ground data Station over the European area. Product FSC-E is validated using high resolution satellite Data from COPERNICUS Sentinel satellites. This validation methodology will be used for new full disk products and hemispheric products, as already made by precipitation products validation (using NOAA DPR – Satellites).
Validation methodology based on ground station data
From the beginning of the project, a common validation procedure using ground station data was established, convening that each participating country in the validation process validate each of the snow products with ground stations of their own country separately. The results are then collected and checked. For Product FSC-E Effective Snow Cover a different validation procedure with Satellite data is applied from 2017, this is described in the following section.

The common validation methodology is based on ground data from automatic weather and snow Stations in the Winter period (1 October – 1 May) to produce large statistic (multi-categorical and continuous), and case study analysis. Both components (large statistic and case study analysis) are considered complementary in assessing the accuracy of the implemented algorithms. Large statistics helps in identifying existence of systematic errors, mainly derived from inhomogeneity of snow cover in complex terrain (spatial inhomogeneity) and rapid snow melting in some areas (mainly in southern and costal near the sea). Selected case studies are useful to identify the roots of such errors.

European Ground Data
The number and type of stations for each of the 6 involved Countries is showed below, in sum more than 3000 station are evaluated each year for the OR.
Ground Data Station and Countries for Snow Validation

Country

Country

Number of Stations

Finland Synoptic 190
Turkey Synoptic 85
Italy Snow/Avalanche 264
Poland Synoptic 595
Germany Synoptic 1863
Belgium Teleclim 84
TOTAL 3081

Ground Data Station Network for Snow Validation

An Example of a Station for Automatic and manual Snow data is showed below, in some countries WMO Weather Stations are used, in other counties dedicated Snow Station Networks provide the data for validation.
Most Weather Station for snow data monitoring in Italy are automated and sited in mountainous areas, were during the winter season there is a stable snow cover. In addition to normal meteorological instruments, these weather stations have a snow depth sensor and a snow temperature profiler (the two instruments in the left side of the snowfield). Data is transmitted via Radio, Satellite of Mobile Phone Network to the regional weather and avalanche centres in Italy, and further transmitted and collected in the Operative Room (Centro Funzionale) of the National Civil Protection in Rome.
Snow field in the Italian Alps with an Automatic Snow and Meteorological Station (Italy)
An Example for a Snow Water Equivalent instrument from a Weather Station is showed below. For validating SWE high quality ground data is used, in some countries they give a detailed analysis of the annual snow cove.

Example of Ground Station in Turkey - Snow Pack Analyser (SPA) for H13 SWE validation.

The complete SPA System consists of two (2) SPA sensing bands with one installed horizontally 10 cm above the ground and the other installed at an angle (referred to as the sloping band), an impedance analyzer, an ultra-sonic snow depth sensor and mounting accessories to assure proper tension of the SPA bands (Figure 3). Each of the SPA bands sends frequencies into the snow pack and measures the complex impedance. Snow consists of ice, air and water. Each of these elements have different dielectric constants and when the band sends out the measuring frequencies it is able to read the returned value to determine the percentage of liquid water, ice and the remaining value as air.

For Products SE-E-SEVIRI and SWE-E validation is done separately for Mountainous and Flat/Forested areas, to provide complete error information on the product performances related to the orography.

Mask flat/forested versus mountainous regions

SE-E-SEVIRI Snow Detection
The validation is based on measurements at ground stations (SYNOP and other lower level posts) made on a daily basis at 0600 UTC. Metadata concerning the method and instrument used for snow measurement as well as accuracy and frequency are included. The comparison between the observation data and the satellite product is a point to pixel comparison. To compare the satellite product with observation data, the measurement from the station that is the nearest to the satellite pixel is used. From satellite product, only pixels with code 0 (snow) and 85 (ground) are taken into consideration. Cloudy and data-missing pixels are discarded from comparison. The measurement is considered as ‘snow occurrence’ if the snow depth (SD) parameter value is equal or greater than 2 cm, SD ≥ 2 cm.

Each Country/Team contributes to long statistic validation by providing the monthly contingency table and the statistical scores. The results are showed in flat and mountainous areas, and for merged product to provide a complete error information to the user on the product performances related to the orography.

To assess the degree of compliance of the product with user requirements all the SPVG members provided the long statistic results following the validation methodology.

For product H10 the User requirements are recorded in table 2:

Product requirements for product SE-E-SEVIRI in Flat/Forest areas

Score

Threshold

Target

Optimal

POD 0.80 0.85 0.99
FAR 0.20 0.15 0.05
Product requirements for product SE-E-SEVIRI in Mountainous areas

Score

Threshold

Target

Optimal

POD 0.60 0.70 0.99
FAR 0.30 0.20 0.05
WS-E: Snow status (dry/wet)
Snow Status is validated by and indirect temperature-based validation procedure. This approach is based on air temperature data, which does not directly describe the status of the snow pack, especially on wetlands. Secondly, the calibration of the thresholds is based on data for whole Finland over several years, but is still validated for a single country only and might not hold for places with very different winter profile. In areas where there is not a homogenous and stable snow cover the validation group retains that product H11 cannot be considered fully reliable. Therefore, it’s suggested to use the product only in Nordic areas where during the winter season snow cover is constant and homogenous, and a clear snow melting period - that is areas with snow cover going from dry to wet - is detected by product H11.

The validation conditions for existence of dry snow are:
  1. Automatic snow depth measurement states that there is one centimeter or more of snow during the day. Observations with less snow are discarded from validation
  2. The average 2 meter air temperature between the 6 hour period between 00-06 UTC has been equal or lower than -0.5 degrees centigrade
  3. The 2 meter air temperature is not above 0 degrees centigrade for more than two hour (e.g. two hourly or 8 15-minutely measurements) during this same period

If rule 1) is not true, there is no snow on the ground and observation is discarded from validation. If 1) is true, but either one of the rules 2) or 3) are false, the snow is considered wet. If all the rules are true, snow is considered dry.

The comparison between the station data and the H11 products are made daily on pixel level. For a single day, consider one pixel with several weather stations. All different stations increment corresponding snow status counters for that individual product pixel. There are counter arrays for
  1. The number of no snow cases
  2. The number of dry snow cases
  3. The number of wet snow cases

There are four arrays for every day: the three listed above and of course the product data itself. Each pixel forms a single validation data point. To avoid the bias that would be caused by mixed-pixel cases, only the unambiguous pixels (the pixels where all the weather stations agree on the status) are included when calculating the statistical metrics. From the classification results, different statistical scores can be calculated: Probability of detection, False alarm ratio, Probability of false detection, Accuracy, Critical success index, and Heidke skill score.

The degree of compliance of the product with product requirements is reported in Table 1

Score

Threshold

Target

Optimal

POD 0.60 0.80 0.90
FAR 0.20 0.10 0.05
SWE-E Snow Water Equivalent
Snow Water Equivalent product is evaluated using selected Station with special Snow Water equivalent instruments, with case studies for a better definition of error source.

Below is an example for a SWE instrument in Turkey.

Validation is currently over Finland, Poland, Germany and Turkey. Each Country/Team contributes to long statistic validation by providing the seasonal statistical scores. The results are showed separately for flat and mountainous areas (if applicable), since the product requirements are different in value for the two areas. Where not applicable (no distinction is made) the stricter rules of the mountainous areas are considered

The degree of compliance of the product with product requirements is reported in Table 1

Area

Threshold

Target

Optimal

flat (RMSE) 40 mm 20 mm 10 mm
mountain (RMSE) 45 mm 25 mm 15 mm
Validation methodology based on high resolution satellite data
FSC-E: Effective snow cover
For the Fractional Snow Cover Product, a ground station-based validation is not possible with conventional ground data from meteorological stations, and a Satellite based validation with Copernicus Sentinel 2 data was developed and validated against in situ webcam photography from the ground. The is illustrated in Piazzi et al., 2019, in which also SE-E-SEVIRI Snow Detection product has been considered. Main results about Sentinel 2 show that: (i) Sentinel2 can properly used for continuous validation of medium/coarse resolution satellite snow products because it has a significant consistency with both ground-based snow measurements and in-situ webcam photography; (ii) dense cloud cover can undermine the reliability of Sentinel-2 snow maps; (iii) patchy snow cover and melting period may lead to an overestimation of snow cover. Main results about satellite snow products show that they are highly consistent with S-2 imagery with a higher agreement over flat areas than in mountainous regions; (ii) complex topography significantly hinders snow detection; (iii) vegetation cover has less relevant impact on the consistency among remotely-sensed observations, even in presence of dense evergreen forest.

This procedure and is now the basis for the validation of the product, and will also be the basis for the new global and hemispheric products in extra- European areas; in 2019/20 two new global products H34 and H35 will be added to the portfolio of H-SAF, validation group will check quality and operative status.

Effective snow cover (FSC-E)

RESULTS
Results of the comparison between H SAF Operational Products and reference data are yearly computed and reported in OR documents. Last Operations Reports are here available:
•   OR-8 (2018)
Download
•   OR-7 (2017)
Download
•   OR-6 (2016)
Download
Validation results for each operational product are summarized in the Product Validation Report (PVR) as below listed:
PRECIPITATION PRODUCTS
•   P-IN-SSMIS: Precipitation rate at ground by MW cross track scanners
Download
•   P-IN-MHS Precipitation rate at ground by MW cross-track scanners AMSU/MHS
Download
•   P-IN-SEVIRI Precipitation rate at ground by blended MW and IR
Download
•   P-AC-SEVIRI Accumulated precipitation at ground by blended MW and IR
Download
SOIL MOISTURE PRODUCTS
•   SSM-ASCAT-NRT-DIS Disaggregated Metop ASCAT NRT SSM at 1 km NRT
Download
•   SM-DAS-2 Soil Wetness Profile Index in the roots region retrieved by Metop ASCAT surface wetness scatterometer assimilation method
Download
•   SSM-ASCAT-B-NRT-O12.5 Metop-B ASCAT NRT SSM orbit geometry 12.5 km sampling
Download
•   SSM-ASCAT-A-NRT-O12.5 Metop-A ASCAT NRT SSM orbit geometry 12.5 km sampling
Download
•   SSM-ASCAT-A-NRT-O25 Metop-A ASCAT NRT SSM orbit geometry 25 km sampling
Download
•   SSM-ASCAT-B-NRT-O25 Metop-B ASCAT NRT SSM orbit geometry 25 km sampling
Download
SNOW COVER PRODUCTS
•   SE-E-SEVIRI Snow detection (snow mask) by VIS/IR radiometry
Download
•   WS-E Snow status (dry/wet) by MW radiometry
Download
•   FSC-E Effective snow cover by VIS/IR radiometry
Download
•   SWE-E Snow water equivalent by MW radiometry
Download
•   SE-D-SEVIRI Snow detection for flat land (snow mask) by VIS/NIR of SEVIRI
Download
•   SE-G-AVHRR EPS Daily Snow Cover
Download