Chapter33 - CM Box User Guide

(Difference between revisions)

Revision as of 10:26, 4 October 2006

5.2. Filling gaps in agricultural statistics.

René Gommes, Linda See, Peter Hoefsloot

The idea behind this chapter is to use pixel-based information that is generally available from non-national sources, like satellite imagery, or topographic information from digital terrain models, etc., to assist with the estimation of the missing statistical information by administrative unit (AU).
First level AUs are not necessarily the most appropriate choice for meaningful presentation of agricultural statistics, because the criteria adopted by the countries to define such units are very rarely based on agro-ecological zones. In addition, their size varies over orders of magnitude and some are so heterogeneous that the climate may vary from desert to tropical humid. AU’s are nevertheless the unit used by national statistical services to report most data.

Examples of “missing” statistical data

National agricultural statistics data are easily available, as they are published by the countries as statistical yearbooks and systematically assembled by FAO into the annual Production Yearbooks (FAOSTAT ). Sub-national data, on the other hand, are usually difficult to retrieve, and subject to a number of complications when they have to be consolidated into a homogeneous and consistent set at the continental level.
Several difficulties are associated with language, units and national typologies; they can be dealt with rather easily, and only rarely contribute to data being “missing” in a strict sense. Some examples are given below:

“language” (Guinea corn is actually sorghum, makopa is dried cassava chops...)
units (‘bags per acre”, where the weight of a “bag” is variable from crop to crop and from country to country);
typologies (walo and dieri in Mauritania, which refer to the type of water control...).

It occurs rather often that the data are unavailable altogether. Possible causes:

no sampling is carried out at the national level. Sometimes subjective estimates are produced, but they are of uncertain quality;
data are collected at the national level, but never documented or actually published in national statistical yearbooks. However, some of those data are available nationally from the concerned services;
different data are collected for different geographic units (for instance, not all AUs collect data for all crops, or sometimes the different AUs apply for agriculture and, say population;
data are aggregated (by areas or by crops) in a way which is not compatible with the reporting of other countries. A typical example would be “millet” which can be bulrush millet or finger millet, or both together, although the crops are rather different from an auto-ecological point of view. The worst example being “millet and sorghum” reported as either “millet” or “sorghum”;
data are not available for all the years from 1986 to 1990 in the reference period, or they are available only for years outside the reference period
the AU units were modified during the reference period. This creates a minor difficulty when AU are aggregated, but when they are split, it is not always possible to redistribute the statistics between the new AU;
the amounts cultivated and harvested are deemed to be negligible if they are below a cut-off that varies according to countries;
crops are not reported on separately, for instance white potatoes can sometimes be lumped with vegetables and appear nowhere in the statistics.

For a continental study, these practices lead to a high percentage of “missing” data. In order to fill the gaps, a number of items had thus to be interpolated or otherwise estimated.

Methods for spatial interpolation of missing agricultural statistics

As indicated, the spatial interpolation can either be purely geo-statistical, or take advantage of the additional knowledge obtained from external variables. In the first category, the method known as “inverse distance weighting”. In the second, the method was Satellite Enhanced Data Interpolation (SEDI).

External variables useable for spatial interpolation

In semi-arid areas, good correlations can be found between between environmental conditions and yield (K/H). Once average AU values of NDVI, elevation, etc. are available for surrounding areas, yields can be regressed against the external environmental variables . The method is not applicable if cultivars vary in the same agroclimatic area. Generally this information is not included in the statistics, and the database was thus considered cultivar- independent. It was also not feasible to distinguish between irrigated and rainfed crops, subsistence farming and large scale modern agricultural production.
The main external variable of interest was NDVI (Normalized Difference vegetation Index), created by NASA/GIMMS and regularly available every 10 days since 1981. It represents one of the most popular remotely sensed indicators for monitoring the response of vegetation to weather condition in several parts of Africa.
NDVI is an indicator of the density of living green mass; in theory, it varies from -1 to +1, but in practice only values between 0 to 0.7 are found on land areas.
1981-1991 average monthly NDVI data from ARTEMIS (FAO, 1993) were used to derive the NDVI variables included in the AGDAT database. The most relevant, in the current context, are NDVI monthly average, maximum and minimum together with the same values relative to the value of 0.12. This threshold corresponds to the occurrence of green vegetation on the ground. The interpretation of NDVI in humid tropical regions is difficult due to the absorption of infra-red light by water vapor and because the response of the index as a unction of biomass reaches a plateau (saturation) at high biomass values.
Figure 1 below illustrates a typical relation between yield and NDVI, showing, among others, NDVI values starting at about 0.07 and yields levelling off from values above 0.2. Note that the values indicated correspond to the spatial average over a whole AU, where no crops are actually grown below about 0.12. This explains why relatively high uields can be found at low NDVI when some crops are irrigated or grown in the wettest parts of the AU only.

Coarse grain yield and NDVI in Burkina Faso and the countries surrounding Burkina Faso. The figure was restricted to yields below 1200 Kg/Ha.

Inverse distance weighting

Geostatistical interpolation of missing data consists in the estimation of missing values at one point in space based on the known values of neighboring entities. The inverse-distance weighting is one of the most straightforward methods; it takes into account the distance between the “known” and “unknown” points and their relative importance in the estimation. For instance, close-by AU’s of the same country are assigned a higher weight than the AU’s of neighboring countries distance. The reason for this is that it is considered that production and feeding behavior are more homogeneous within the country. For the AGDAT database, after testing, conditions for interpolation are at least three to ten neighboring AU’s and maximum distance between the AU’s centers is 600 km.
For the geo-statistical interpolation, it was generally considered that the reported crop yield corresponds to the centre of gravity of the AU . The software used for the inverse-distance weighting was mostly FAOMET (Gommes and See, 1993). Inverse-distance weighting was applied mainly to Area cultivated per capita, per capita production and relative area.

Satellite Enhanced Data Interpolation, or SEDI

SEDI takes advantage of the correlation between and environmental variable, for instance the above mentioned NDVI/biomass and agricultural yields. One of the ways to approach this is co-kriging, a variant of kriging using one or more ausiliary variable and exploiting both the spatial features of the variable to be interpolated and the correlations between the variable and the ausiliary variables (Bogaert et al, 1995)
The SEDI interpolation method originated in a Harare based FAO Regional Remote Sensing Project. It was originally developed to interpolate rainfall data collected at station level using the additional information provided by METEOSAT cold cloud duration images. The methods proved powerful and versatile, and it is now regularly used by FAO to spatially interpolate other parameters as well (e.g. potential evapotranspiration, crop yields, actual crop evapotranspiration estimates etc.
The concepts of this interpolation method and software implementing the technique have been described by Hoefsloot, 1996. The SEDI functions were recently incorporated into the WINDISP_3 software (Pfirman and Hogue, 1998)
SEDI is a simple and straightforward method for 'assisted' interpolation. The method can be applied to any parameter of which the values are available for a number of geographical locations, as long as a 'background' field is available that has a negative or positive relation to the parameter that needs to be interpolated.
Three requirements are a prerequisite for the successful application of the SEDI method:

The availability of the parameter to interpolate as point data at different geographical locations (e.g. rainfall, potential evapotranspiration, crop yields). In the present case of statistical variables, they were assigned a co-ordinate corresponding to the centre of gravity of the AU;
The availability of a background parameter in the form of a regularly spaced grid (or field) for the same geographical area (e.g. the above-mentioned NDVI variables , altitude).
A relation between the two parameters (negative or positive; Yield/NDVI is positive, temperature/altitude would be negative). A Spearman rank correlation test can reveal whether a relation exists, and how strong this relation is.

The SEDI method yields the parameter mentioned under point 1 as a field (i.e. an image covering the whole area under consideration. The average of the field value over the AU provides the estimation of the spatialized statistic. The method is illustrated below using rainfall and “Cold Cloud Duration”.
Rainfall is reported on a dekadal (10-day) basis in most countries of the world for agrometeorological purposes. A typical longitude-latitude plot showing the locations of the stations and the rainfall amounts is given in figure 2 below.
The geostationary METEOSAT satellite takes infrared temperature “pictures” of the earth every half hour. In tropical regions it can be assumed that areas with temperatures lower than about minus 40C correspond to convective cloud systems of the type that produces rainfall. The accumulated number of hours in a dekad below this low temperature threshold is known as 'Cold Cloud Duration' (CCD). It can be represented as an “image”, i.e. regularly spaced rows and columns the intersection of which corresponds to the picture elements, or 'pixels'. A pixel represents one data value . Pixels can be assigned a colour depending on the value they represent, as represented by a grey-scale in figure 3.

Rainfall (mm) for Zimbabwe second dekad of January 1991.

The relation between rainfall and CCD is a positive one. In other words: high rainfall values generally coincide with high CCD values.
The SEDI process is done in three steps:
1. Extracting values from the image and calculating the ratio of point and image values 2. Gridding the ratio's to form a regularly spaced grid. 3. Multiplying Grid with image to obtain estimated image.
1.1.2.1. Step 1 : Extracting values from the image and calculating the ratio's

Figure 4. Extracting 5 or 9 pixels per point value

For every point value in the input rainfall data, a value can be extracted from the CCD image. The SEDI method will find the pixel that coincides with a rainfall station and extract the pixel value. In some cases the value of one pixel does not give satisfactorily results. Therefore the SEDI software allows the user to extract the values of more than one pixel from the image, and take its average as image value for the station (figure 4). For every station we now have a rainfall value and a CCD value. The Spearman rank correlation coefficient (using the rainfall/CCD data pairs) yields a positive value. This means the relation between rainfall and CCD is positive (as to be expected). The ratio between rainfall and CCD value is now calculated as shown in table 1)
Table 1 : ratio between Rainfall and CCD (mm/hour)
Station Name Rainfall (mm) CCD value (hours) Ratio Station 1 23.4 56 0.42 Station 2 12.4 12 1.03 Station 3 54.3 96 0.57 Station 4 6.7 8 0.84
Should the relation have been negative, the ratio is calculated as follows:

1.1.2.2. Step 2: Creating a regularly spaced grid from the ratios
The second step constitutes of the creation of a grid from the irregularly spaced ratio's, i.e. the spatial interpolation, of the ratios at regularly spaces points contituting a grid (figure 5).

Figure 5. Creation of ratio grid from point values
The ratio grid is created with the inverse distance method mentioned under 3.3.2 with a weighting power of 2. The software allows the user to set:
• The distance between the grid lines. A low distance creates a accurate, dense grid, while a high value creates a coarse, less accurate and more general grid; • The number of stations per grid-point determines the number of stations included in the calculation of a point in the grid matrix;. • The maximum radius for interpolation determines whether a value is calculated for a point in the grid matrix. If the number of stations around this grid-point within this radius is higher than the specified number of stations, a value is calculated. Otherwise the grid-point is assigned a missing value, and the resulting image will be 'empty' at that particular point.

1.1.2.3. Step 3: Creating the SEDI image
The last step encompasses the creation of the SEDI image. The process is simple. By multiplying the grid (step 2) with the background image, an estimate for the value to interpolate is obtained. In terms of rainfall and CCD: a rainfall image is obtained by multiplying the ratio grid with the background image (figure 6).

Figure 6 Creation of a SEDI image from a ratio grid and a background image

Retrieved from "http://hoefsloot.com/wiki/index.php?title=Chapter33"

CM Box User Guide	Main Page \| About \| Special pages \| Log in
	Printable version \| Disclaimer \| Privacy policy \| Current revision

 Rainfall is reported on a dekadal (10-day) basis in most countries of the world for agrometeorological purposes. A typical longitude-latitude plot showing the locations of the stations and the rainfall amounts is given in figure 2 below.
+The geostationary METEOSAT satellite takes infrared temperature “pictures” of the earth every half hour. In tropical regions it can be assumed that areas with temperatures lower than about minus 40C correspond to convective cloud systems of the type that produces rainfall. The accumulated number of hours in a dekad below this low temperature threshold is known as 'Cold Cloud Duration' (CCD). It can be represented as an “image”, i.e. regularly spaced rows and columns the intersection of which corresponds to the picture elements, or 'pixels'. A pixel represents one data value . Pixels can be assigned a colour depending on the value they represent, as represented by a grey-scale in figure 3.
 {|"class=prettytable" cellpadding="15" border="1" style="border-collapse:collapse"
 |width="225"|Rainfall (mm) for Zimbabwe second dekad of January 1991.
 ||[[Image:graph103.jpg|400 px]]
 |}
-The geostationary METEOSAT satellite takes infrared temperature “pictures” of the earth every half hour. In tropical regions it can be assumed that areas with temperatures lower than about minus 40C correspond to convective cloud systems of the type that produces rainfall. The accumulated number of hours in a dekad below this low temperature threshold is known as 'Cold Cloud Duration' (CCD). It can be represented as an “image”, i.e. regularly spaced rows and columns the intersection of which corresponds to the picture elements, or 'pixels'. A pixel represents one data value . Pixels can be assigned a colour depending on the value they represent, as represented by a grey-scale in figure 3.
-Figure 3. Cold Cloud Duration image for Zimbabwe, second dekad of January 1991
 The relation between rainfall and CCD is a positive one. In other words: high rainfall values generally coincide with high CCD values.