Quality Assessment

In the framework of the H SAF project several soil moisture products, with different timeliness (e.g. NRT, offline, data records), spatial resolution, format (e.g. time series, swath orbit geometry) or the representation of the water content in various soil layers (e.g. surface, root-zone), are generated on a regular basis and distributed to users. The Soil Moisture Products Validation Group evaluates yearly the operational soil moisture products (SMP) to guarantee a continuous high-level quality standard.

The actually operational SMP are:

• Surface soil moisture (SSM) from the radar backscattering coefficients measured by the Advanced Scatterometer (ASCAT) on-board the series of Metop satellite:

  • SSM ASCAT-A(-B)(-C) NRT O12.5(O25): Metop-A(-B, -C) ASCAT NRT SSM orbit geometry 12.5 (25) km sampling

  • SSM ASCAT DR2019TS12.5: Metop ASCAT data record SSM time series 12.5 km sampling for the period 2007-01-01 to 2018-12-31

  • SSM ASCAT DR2019 EXT TS12.5: Metop ASCAT data record extension SSM time series 12.5 km sampling for the year 2019

  • SSM-ASCAT-NRT-DIS: Disaggregated Metop ASCAT NRT SSM at 1 km NRT

• Root zone soil moisture (RZSM) product from the assimilation of scatterometer Surface Soil Moisture (SSM) in a ECMWF Land Data Assimilation System (LDAS)

  • SM-DAS-2: Soil Wetness Profile Index in the roots region retrieved by Metop ASCAT surface wetness scatterometer assimilation method

  • SM-DAS-3: Soil Wetness Index in the roots region by ERS/SCAT and Metop ASCAT-A scatterometer assimilation in a Land Data Assimilation System - Offline product

The first 5 products are Level 2 surface soil moisture product derived from the radar backscattering coefficients measured by the Advanced Scatterometer (ASCAT) on-board the series of Metop satellites using a change detection method, developed at the Research Group Remote Sensing, Department for Geodesy and Geoinformation (GEO), Vienna University of Technology (TU Wien). In the TU Wien soil moisture retrieval algorithm, longterm Scatterometer data are used to model the incidence angle dependency of the radar backscattering signal. Knowing the incidence angle dependency, the backscattering coefficients are normalized to a reference incidence angle. Finally, the relative soil moisture data ranging between 0% and 100% are derived by scaling the normalized backscattering coefficients between the lowest/highest values corresponding to the driest/wettest soil conditions (more information can be found on the Product User manual PUM).

More information on the soil moisture retrieval algorithm can be found in the Algorithm Theoretical Baseline Document (ATBD).

The other two products are index of root zone soil moisture. In the soil moisture assimilation system of ECMWF, the surface observation from ASCAT is propagated towards the roots region down to 2.89 m below surface, providing estimates for 4 layers (thicknesses 0.07, 0.21, 0.72 and 1.89 m).

The ECMWF model generates soil moisture profile information according to the Hydrology Tiled ECMWF Scheme for Surface Exchanges over Land (HTESSEL). SM-DAS-2 is available at a 24-hour time step, with a global daily coverage at 00:00 UTC. SM-DAS-2 is produced in a continuous way in order to ensure the time series consistency of the product (and also to provide values when there is no satellite data, from the model propagation).

The validation of space-based soil moisture products is typically based on the comparison of related space-based soil moisture data sets, in situ data, land surface models or other parameters related to surface soil moisture (e.g. precipitation). In the framework of the HSAF project the Soil Moisture Products are yearly validated in two ways:

1. Comparing the satellite products against in situ soil moisture measurements where available.

2. With a triple collocation analysis using global dataset of soil moisture.

The validation with ground data is performed through the comparison with in situ sensors located in Europe (e.g., SMOSMANIA) and worldwide. The performance metrics used for the evaluation of the accuracy of the test data set are the correlation coefficient, CC, the Root Mean Square Difference, RMSD, and the BIAS or mean error (satellite minus in situ), BIAS. Before the comparison, satellite data are rescaled to the same range of in situ observations through a linear approach.

The triple collocation analysis (TCA) is a statistical tool used for error characterization, first introduced by Stoffelen in 1988. It simultaneously estimates the error structure of three spatially and temporally collocated data sets θ1, θ2 and θ3, which are linearly related to the hypothetical (unknown) truth θ and contaminated by additive zero-mean noise, which must not be correlated with the signal or between sets (Scipal et al., 2010):

αi and βi are systematic additive and multiplicative biases of data set i with respect to the true state Θ, and εi represents zero-mean random noise. The underlying assumptions of the error model are: (i) linearity between the true signal and the observations, (ii) signal and error stationarity, (iii) independence between the errors and the signal, and (iv) independence between the errors of θ1, θ2 and θ3, (zero error cross-correlation). The mean squared random error (MSE) of all three data sets is estimated individually by TCA. The error variance can also be expressed in a normalised form as Signal-to-Noise Ratio (SNR). This allows to compare the error variances between the data sets. SNR is usually given in decibel, which can be easily interpreted: a SNR value of zero means that the signal variance is equal to the noise variance, and every 3 dB increase implies a doubling of the signal variance compared to the noise variance.

The soil moisture retrieval algorithm is applied on a global basis (i.e. for each grid point over land), but in certain situations (dense forest, snow cover, frozen soil, open water or topographic complex area dominating the satellite footprint), the retrieval of meaningful soil moisture values is not always possible. Nonetheless, soil moisture is computed and additional information (e.g. on the soil state) is provided to mask invalid soil moisture measurements. The triple collocation analysis is performed globally and the statistics are provided for committed and non-committed areas. The committed area represents a restricted geographical region with high confidence in the successful retrieval of surface soil moisture information from Metop ASCAT. The area is limited to low and moderate vegetation regimes, unfrozen and no snow cover, low to moderate topographic variations, as well as no wetlands and coastal areas. All the statistical results of the TC analysis are reported both on global scale and committed areas.

The validation is performed globally on the WARP 5 grid (v2.3). As reference data set the NOAH GLDAS land surface model (v2.1) and the passive CCI soil moisture product (v04.3) are used. The NOAH model provided by the Global Land Data Assimilation System (GLDAS) contains atmospheric and land surface parameters stored on a regular global grid (spacing 0.25◦). From 2000-ongoing, the GLDAS NOAH version 2.1 data set provides soil moisture at a 3-hourly temporal resolution (daily at 00:00, 03:00, 06:00, 09:00, 12:00, 15:00, 18:00 and 21:00 UTC) (Roddel et al., 2004). The data is publicly available at GES DISC (Goddard Earth Sciences Data and Information Services Center). The first soil moisture layer (0.00 - 0.07 m) of NOAH GLDAS is used for the validation.

The Soil Moisture CCI project is part of the ESA Programme on Global Monitoring of Essential Climate Variables (ECV), better known as the Climate Change Initiative (CCI). The CCI Passive Soil Moisture product (v4.3) was generated by the VU University Amsterdam in collaboration with NASA based on passive microwave observations from Nimbus 7 SMMR, DMSP SSM/I, TRMM TMI, Aqua AMSR-E, Coriolis WindSat, and GCOM-W1 AMSR2. The ECV soil moisture production system generates soil moisture at a spatial resolution of approximately 25 × 25 km for top < 2 cm of the soil, expressed in volumetric units (m3m-3). The soil temperature and snow depth information are used for filtering for non-frozen (soil temperature > 4°) and snow-free (snow depth = 0) time periods.

The results of the validation in H SAF are reported as Signal-to-Noise Ratio (SNR) and Pearson’s correlation coefficient (R). Pearson’s R is the most commonly used measure of linear dependencies between two data sets (Helsel et al., 2002, Wilcox, 2009). Three threshold values, optimal, target and threshold, have been selected to evaluate the performances of the products. For further information on validation and specific results of each product refer to the Product Validation Reports (PVR). Product validated with tripe collocation are SSM ASCAT-A(-B)(-C) NRT O12.5(O25), SSM ASCAT DR2019 EXT TS12.5, SM-DAS-2 and SM-DAS-3.

D. Helsel and R. M. Hirsch, Statistical Methods in Water Resources, 2002, no. Book 4.

M. Rodell, P. R. Houser, U. Jambor, J. Gottschalck, K. Mitchell, C.-J. Meng, K. Arsenault, B. Cosgrove, J. Radakovich, M. Bosilovich, J. K. Entin*, J. P. Walker, D. Lohmann, and D. Toll, “The Global Land Data Assimilation System,” Bulletin of the American Meteorological Society, vol. 85, pp. 381–394, Mar. 2004.

K. Scipal, W. Dorigo, and R. deJeu, “Triple collocation: A new tool to determine the error structure of global soil moisture products.” IEEE, Jul. 2010, pp. 4426–4429.

A. Stoffelen, “Toward the true near-surface wind speed: Error modeling and calibration using triple collocation,” Journal of Geophysical Research Oceans, vol. 103, pp. 7755–7766, 1998.

R. R. Wilcox, Basic statistics: understanding conventional methods and modern insights, 2009

DPC and the international validation group does yearly the Operational Reviews (OR) of the Snow Products of H SAF Satellite products. All four of the Snow Cluster products are now operational and the continuous validation guarantees that they will satisfy the quality standards in the future. Six countries from H SAF consortium Ground data from National Weather Stations of: Finland, Turkey, Italy, Poland, Germany and Belgium.

The products are:

• SE-E-SEVIRI: Snow detection (snow mask) by VIS/IR radiometry;

• WS-E: Snow status (dry/wet) by MW radiometry;

• FSC-E: Effective snow cover by VIS/IR radiometry;

• SWE-E: Snow water equivalent by MW radiometry;

Each of the product have a dedicated validation procedure:

Products SE-E-SEVIRI, WS-E and SWE-E are validated using a methodology based on ground data Station over the European area. Product FSC-E is validated using high resolution satellite Data from COPERNICUS Sentinel satellites. This validation methodology will be used for new full disk products and hemispheric products, as already made by precipitation products validation (using NOAA DPR – Satellites).

The validation is based on measurements at ground stations (SYNOP and other lower level posts) made on a daily basis at 0600 UTC. Metadata concerning the method and instrument used for snow measurement as well as accuracy and frequency are included. The comparison between the observation data and the satellite product is a point to pixel comparison. To compare the satellite product with observation data, the measurement from the station that is the nearest to the satellite pixel is used. From satellite product, only pixels with code 0 (snow) and 85 (ground) are taken into consideration. Cloudy and data-missing pixels are discarded from comparison. The measurement is considered as ‘snow occurrence’ if the snow depth (SD) parameter value is equal or greater than 2 cm, SD ≥ 2 cm.

Each Country/Team contributes to long statistic validation by providing the monthly contingency table and the statistical scores. The results are showed in flat and mountainous areas, and for merged product to provide a complete error information to the user on the product performances related to the orography.

To assess the degree of compliance of the product with user requirements all the SPVG members provided the long statistic results following the validation methodology.

For product H10 the User requirements are recorded in table 2:

Snow Status is validated by and indirect temperature-based validation procedure. This approach is based on air temperature data, which does not directly describe the status of the snow pack, especially on wetlands. Secondly, the calibration of the thresholds is based on data for whole Finland over several years, but is still validated for a single country only and might not hold for places with very different winter profile. In areas where there is not a homogenous and stable snow cover the validation group retains that product H11 cannot be considered fully reliable. Therefore, it’s suggested to use the product only in Nordic areas where during the winter season snow cover is constant and homogenous, and a clear snow melting period - that is areas with snow cover going from dry to wet - is detected by product H11.

The validation conditions for existence of dry snow are:

1. Automatic snow depth measurement states that there is one centimeter or more of snow during the day. Observations with less snow are discarded from validation

2. The average 2 meter air temperature between the 6 hour period between 00-06 UTC has been equal or lower than -0.5 degrees centigrade

3. The 2 meter air temperature is not above 0 degrees centigrade for more than two hour (e.g. two hourly or 8 15-minutely measurements) during this same period

If rule 1) is not true, there is no snow on the ground and observation is discarded from validation. If 1) is true, but either one of the rules 2) or 3) are false, the snow is considered wet. If all the rules are true, snow is considered dry.

The comparison between the station data and the H11 products are made daily on pixel level. For a single day, consider one pixel with several weather stations. All different stations increment corresponding snow status counters for that individual product pixel. There are counter arrays for

1. The number of no snow cases

2. The number of dry snow cases

3. The number of wet snow cases

There are four arrays for every day: the three listed above and of course the product data itself. Each pixel forms a single validation data point. To avoid the bias that would be caused by mixed-pixel cases, only the unambiguous pixels (the pixels where all the weather stations agree on the status) are included when calculating the statistical metrics. From the classification results, different statistical scores can be calculated: Probability of detection, False alarm ratio, Probability of false detection, Accuracy, Critical success index, and Heidke skill score.

Snow Water Equivalent product is evaluated using selected Station with special Snow Water equivalent instruments, with case studies for a better definition of error source.

Below is an example for a SWE instrument in Turkey.

Validation is currently over Finland, Poland, Germany and Turkey. Each Country/Team contributes to long statistic validation by providing the seasonal statistical scores. The results are showed separately for flat and mountainous areas (if applicable), since the product requirements are different in value for the two areas. Where not applicable (no distinction is made) the stricter rules of the mountainous areas are considered

For the Fractional Snow Cover Product, a ground station-based validation is not possible with conventional ground data from meteorological stations, and a Satellite based validation with Copernicus Sentinel 2 data was developed and validated against in situ webcam photography from the ground. The is illustrated in Piazzi et al., 2019, in which also SE-E-SEVIRI Snow Detection product has been considered. Main results about Sentinel 2 show that: (i) Sentinel2 can properly used for continuous validation of medium/coarse resolution satellite snow products because it has a significant consistency with both ground-based snow measurements and in-situ webcam photography; (ii) dense cloud cover can undermine the reliability of Sentinel-2 snow maps; (iii) patchy snow cover and melting period may lead to an overestimation of snow cover. Main results about satellite snow products show that they are highly consistent with S-2 imagery with a higher agreement over flat areas than in mountainous regions; (ii) complex topography significantly hinders snow detection; (iii) vegetation cover has less relevant impact on the consistency among remotely-sensed observations, even in presence of dense evergreen forest.

This procedure and is now the basis for the validation of the product, and will also be the basis for the new global and hemispheric products in extra- European areas; in 2019/20 two new global products H34 and H35 will be added to the portfolio of H SAF, validation group will check quality and operative status.