Validity and Reliability of Life Cycle Assessment: A Case Study of New Zealand Dairy Production

Accepted: 23 March 2020 The purpose of this work is to evaluate the validity and reliability of life cycle assessment as a research methodology to measure the ecological footprint of New Zealand’s dairy production (by the dairy farming and dairy processing sectors). Environmental Input-Output Analysis was used as a macro-level life cycle assessment tool and applied to the dairy production over a period of six years. Eco-efficiencies were used as performance metrics to assess the sustainability of the dairy production. Over the six year period, New Zealand’s annual milk production increased from 11.4 to 15.1 million tonnes and its annual production of dairy products increased from 1.9 to 2.6 million tonnes. Eco-efficiencies indicate that over this time dairy farming became significantly more efficient in terms of land use (-27%), electricity use (-12%), water use (-21%) and lime use (-16%) and produced significantly less water-based effluent (-20%). At the same time fuel and fertilizer use were slightly less efficient (increasing by 2% and 6% respectively). The dairy processing industry used 21% less water and discharged 21% less effluent water. Fuel used in milk transportation was 14% more efficient. The internal validity of the research was good despite significant structural changes to the dairy processing sector but commercial sensitivity had a negative impact on the results. External validity was affected by different boundaries, different climates and different time frames for published studies but some comparisons were possible. The underlying data was generally accurate, reproducible and representative of the entire sectors with checks for anomalies to ensure good reliability. The choice of Environmental Input-Output Analysis for the life cycle assessment tool and error analysis of all underlying data contributed to both the validity of the research method and the reliability of the data. These, in turn, give good credibility to the research findings.


INTRODUCTION
The dairy farming sector and the dairy processing sector contribute substantially to New Zealand's GDP, but have concurrently large environmental impacts, particularly those relating to deteriorating soil ecosystems, greenhouse gas emissions and water pollution. While GDP is commonly used as an indicator of a country's economic well-being, the overall well-being is captured more precisely in the country's Genuine Progress Indicator (GPI) (Bagstad et al., 2014;Kubiszewski et al., 2013). For many countries, including New Zealand, the GPI has increased less rapidly than the GDP over the past 40 years (Patterson et al., 2019).
Life cycle assessment (LCA) is commonly used to measure environmental impacts arising from all stages in the production of a commodity or process. In the context of dairy production, traditional LCA has been used in small-scale studies on milk production (deVries & de Boer, 2010;Yan et al., 2011;Iribarren et al., 2011) and on a range of dairy products such as cheese (Milani et al., 2011;van Middelaar et al., 2011). This type of LCA is frequently coupled with Data Envelopment Analysis (DEA) to measure the efficiency of decision making units (DMUs). For example LCA-DEA has been used to compare different energy-producing companies (Vasquez-Rowe & Iribarren, 2015) or different wheat-producing concerns (Pishgar-Komleh et al., 2020) where the desirable outputs are energy and wheat respectively and the undesirable output is often the carbon-footprint based on greenhouse gas (GHG) emissions. Environmental Input-Output Analysis (EIOA) is a macro-level life cycle assessment involving the use of national economic data coupled with national physical data (such as data on land use, water use, greenhouse gas emissions, etc.), all aggregated at the sector-level over a period of one year. It is used to show the annual 'ecological footprint' of a country's interacting sectors and provides measures of the resource use and environmental impact on a national scale (Schaffartzik et al., 2013). The footprint encompasses the total impact, that arises both from the sector itself (the direct contribution) and from the sectors it interacts with (the indirect contribution). The metrics arising from EIOA depend upon the amount of goods produced annually by each sector; for example, production of a small amount of cheese will generally use less resources and produce less waste than the production of a large amount of cheese. Normalising the use of resources and the production of wastes in terms of production units yields a set of 'eco-efficiencies', such as land use per tonne of milk solids or effluent production per tonne of milk solids and these can be used as metrics to assess the sustainability of the sectors, and of the country as a whole, over a period of time (Yu et al., 2013).
In 2005, a government-funded investigation into the ecological footprint of New Zealand's food and fibre sectors began. The first phase was a retrospective cross-sectional study of the sectors; a snap shot of their activity in 1998 using EIOA to establish a baseline of performance for comparison with future years. The second phase (as envisaged at the beginning of the research) was to be a similar study on the sectors' performance six years later, in 2004. For various reasons, discussed later in this article, the second phase was only partially completed. The case study reported here is a part of the overall investigation, and looks at the dairy sectors' ecological footprint in 1998. It then looks at the dairy sectors' sustainability, expressed in terms of eco-efficiency changes over the sixyear period. The specific factors investigated were land use, electricity use, water use, fuel use, fertilizer use, dairy production and water-based effluent production. Details of the research are reported in Flemmer (2012).
The remainder of this article briefly discusses the rationale behind the research methodology and presents an overview of the main findings. It then focuses on two important aspects of the research, namely the validity of the research methodology and the reliability of the data collected -both of which are critical to the value of the research findings. Many LCA practitioners ignore these aspects (Lloyd & Ries, 2007).
For New Zealand's dairy sectors, the validity of the methodology was affected by several circumstances such as the profound change in the corporate structure of the country's dairy processing sector over the six year period and the extreme commercial sensitivity of the collected data. The effect of these on the reported eco-efficiencies as accurate metrics for judging the sustainability of the dairy sectors is discussed.
Assessing the error in the accuracy of the data collected in a particular study allows judgement to be made as to whether the underlying data is reliable. However, data reliability may also be determined by comparison with other published international studies. In this case study, error assessments were made on the collected data but significant problems arose when trying to compare the data with other published studies; New Zealand milk collection is undertaken by the dairy processing sector while it is part of the dairy farming sector activity for most countries. This aspect has a massive influence on the boundaries that are crucial in life cycle assessment.

RESEARCH METHODOLOGY
The research goal was to determine the ecological footprint (or overall economic and environmental performance) of New Zealand's food and fibre sectors over a period of time in order to judge whether their performance was becoming more or less sustainable. The metric chosen to express the performance was a set of eco-efficiencies. In this case study, the performance of the dairy farming and dairy processing sectors were examined. Examples of the eco-efficiencies (or units of the study) were the use of water, land and energy and the emission of carbon dioxide from energy use, all expressed per kilogram of milk.
The epistemology (or underlying literature relating to this type of research) suggested that EIOA as a macro-level life cycle assessment methodology was the most appropriate way to generate eco-efficiencies. This then led to an examination of the required tools and, ultimately, to the ontology or detailed development of the research methodology used in the research project as described in Andrew et al. (2005).
The research approach can be explained in terms of Saunder's research onion (Saunders et al. 2009), in terms of its philosophy (the outermost layer of the onion), approach, strategy, method and tools (the innermost core of the onion) as summarised on Table 1. The research began with a retrospective cross-sectional study of the sectors; a snap shot of the performance of the sectors in 1998 to establish a baseline of performance for comparison with future years. A positivist philosophy was adopted, involving very structured data collection, error analyses of the collected data and extensive data storage protocols. Emphasis was placed on using large samples that were representative of each sector in its entirety. Further emphasis was placed on ensuring both internal and external validity in the data collection; the researchers individually sought the most accurate data and there was careful oversight of the central database to check for anomalies in the data. The validity of the chosen EIOA approach as a measurement of environmental performance and the validity of eco-efficiencies as a measure of sector performance is discussed later in the article. The research involved a mixed method; primarily the collection of quantitative data supplemented with qualitative information regarding the reliability of the data collected.
For the case study on dairy production, the underlying data sources are discussed in  and Flemmer (2012). The research established the total (direct and indirect) environmental impacts in terms of inputs, such as land, energy and water use, and outputs such as waste effluent and greenhouse gas emissions. The methodology identified those sectors supplying the dairy industry which make significant indirect contributions to its total inputs and outputs. The latter could be used in targeting future attempts to improve the ecological performance of dairy production.
The next phase of the research project aimed at a second cross-sectional study six years later (in 2004) using the same EIOA methodology with an interpretivist philosophy and an inductive approach in order to form an assessment of whether New Zealand's food and fibre sectors were becoming more or less sustainable. The complexities in this assessment are discussed later in the article in analysing the validity of the approach. There is also a detailed discussion of the error estimates and reliability of the underlying data. For the case study on dairy production this produced eco-efficiencies for the dairy farming sector and the dairy processing sector over the six year period and these are presented in the next section.

RESULTS AND MAIN FINDINGS
The full EIOA results for the baseline 1998 performance of New Zealand's dairy farming and dairy processing sectors are presented in Flemmer 2012.
For the dairy farming sector, Table 2 summarises the direct and the total (direct and indirect) inputs and outputs of the sector in 1998. The inputs are the use of resources by the sector. In the case of land, direct use of land is the land used for dairy farming, while indirect use of land is the land embodied in other sectors (such as those producing feed crops) that supply the dairy farming sector. The outputs are the useful product (namely, milk) and the discharges of water-based effluent and greenhouse gases.
Note that in Table 2 "fuel use" is the use of petrol and diesel for the operation of the dairy farms. In New Zealand, the dairy processing sector manages the transport of the milk from the farms to the processing centres so fuel used

Philosophy Positivist
Knowledge of the ecological footprint of New Zealand's food and fibre sectors will come from positive interpretation of the research results. The methodology accurately represents the reality of the sector performance and eco-efficiencies represent a good description of sustainability The results need to be repeatable and will only be acceptable if they are based on data sample sizes that are sufficiently large to be representative of the sector-level scale. Validity will rely on detailed error analysis of the data and well-structured data collection protocols.

Approach Inductive
The data gathered in the research is used to estimate eco-efficiencies which are used to construct a theory on the performance of the dairy sectors using inductive reasoning.

Strategy Archival research and interviews
The sectoral data comes from finding and examining archival government and corporate databases, coupled with interviews of people involved in the collection of the original databases.

Quantitative
The use of Environmental Input-Out analysis as a mathematical model capturing all intra-sector flows that allows life cycle assessment at the national level. The use of eco-efficiencies as metrics for sector performance over the six-year period.

Tools Databases and interviews
Government and corporate databases, scaled to the national level. Interviews to perform error assessments on the data for internal validity. Protocols for data storage and data oversight by project managers for external validity.
in milk transportation is allocated to the dairy processing sector.
The high percentages of indirect contributions given in Table 2 show that New Zealand's dairy farming sector imports many of its inputs and outputs from other sectors of the economy. The exception is the use of water; 96.3% of the total water is used directly by the dairy farming sector itself. Similarly, 91.9% of the water effluent is produced directly by dairy farming operation.
The direct inputs and outputs for the New Zealand dairy farming sector for the years 1998 and 2004 are listed in Table 3. The performance of the sector, expressed as the change in eco-efficiencies is given in Table 4.
When the performance of the farming sector is expressed in terms of eco-efficiencies, it is clear (Table 4) that the sector has improved in terms of its use of land, electricity, water and lime and in its generation of water-based effluent and animal methane. However, over the 6 year period it is less efficient in its use of fertilizer (excluding lime) and fuel and in its emission of carbon dioxide from fuel use.
For the dairy processing sector, Table 5 lists the direct and the total (direct and indirect) inputs and outputs of the sector in 1998.
In terms of inputs, Table 5 shows that the dairy processing sector uses very little land for its operation and imports an enormous amount of land use from the dairy farming sector in the supply of the milk for processing. There are similar large indirect contributions to the electricity and water use. For outputs, there are large indirect contributions to the total water effluent and to the carbon dioxide emissions from fuel used predominantly in milk collection.
The direct inputs and outputs for the New Zealand dairy processing sector are listed in Table 6. The change in eco-efficiencies are given in Table 7.
There are 8 inputs and 7 outputs in Table 5 for 1998 and yet the data in Table 6 for 2004 has only one input and three outputs, so it is clear that there was a problem with the 2004 data. All of the missing data was embargoed on the grounds of commercial sensitivity and this is discussed in the next section. Table 7 shows that over the six year period the New Zealand dairy processing sector became more efficient in its use of water, its discharge of water effluent and its discharge of carbon dioxide from milk collection. It produced 4.73% fewer kilograms of processed dairy products per kilogram of raw milk but there was a trend over this period towards manufacturing higher value dairy products.
Having presented the main findings from the research, there are several aspects that need to be considered. Firstly, the accuracy of the underlying data needs to be examined. Secondly, the life cycle assessment using EIOA could only be performed on the 1998 data and not on the 2004 data. Thirdly, only a very small portion of the dairy processing data collected in 2004 was ever published    and finally, there is the question of whether EIOA gives an accurate assessment and whether eco-efficiencies are a measure of the performance of New Zealand's dairy sectors. These aspects are discussed in the next section.

Validity and Reliability
Assessing the validity of the research involves judging whether both the research design and the research method produced an accurate measurement of the performance of the dairy industry. Both internal and external validity are important. Internal validity is an assessment of the flaws within the study itself arising from weaknesses in the design of the research method and also from weaknesses in the data collected. External validity is a measure of the extent to which the findings from the research can be generalized to a larger group.
Factors affecting internal validity include: • Variability of the dairy sectors over the time frame of the research project • The data sample size having to be representative of the sector as a whole • The time taken for data collection and analysis • The accuracy of the data collected (expressed in terms of the error analyses in the component data but also affected by factors such as commercial sensitivity and human bias) External validity is affected by aspects such as: • The intrinsic differences between the dairy sector operation in different countries with different boundaries, different practices, different scales of operation and different climates. • Interaction between the researcher and the companies managing the research data • The effect of time; the research analysis in this case occurred over a different time frame from other international studies Reliability means consistency or repeatability of the research measures, in the context of evaluating the quality of research (rather than as a statistical measure). It is affected by: • The availability of independent data sources • Data sample sizes that are large enough to represent the entire dairy industry • Protocols for eliminating data errors and for checking data values Each of these aspects is discussed below.

Validity of the research design.
The research was designed around the use of EIOA as a macro-level life cycle assessment for the New Zealand food and fibre sectors (dairy, sheep, beef and grain farming, forestry and the processing of products from these sectors) and the use of eco-efficiencies to express sector performance. EIOA is generally held to be the most accurate method for determining the total (direct and indirect) use of resources (or inputs) and the total outputs of both valuable products and of ecological impacts. There are several metrics for sustainability, as discussed in , but eco-efficiencies are regarded as one of the valid proxies for sustainability since they reflect the balance between minimizing environmental impact while optimizing production processes. Thus the research design is valid and is an accurate method for assessing the ecological performance of the food and fibre sectors. However EIOA relies on the availability of accurate and extensive data in the form of annual monetary input-output (MIOT) tables and annual physical input-output (PIOT) tables for each and every sector of a country's economy. Many countries have extensive MIOTs but very few PIOTS exist and those that do exist are limited to just a few entries such as land use, electricity use and carbon dioxide emissions.  In this case study, the 1998 MIOT for 48 sectors of the New Zealand economy was provided by Market Economics Ltd who also provided some base data for the 1998 PIOT. Superior data for the PIOT was collected for the New Zealand food and fibre sectors as a collaborative research effort involving six research groups. Careful protocols were put in place to minimize data errors in transcription to the central database and to check the data values against other data sources.
The data collection was a much lengthier process than originally envisaged and some three years into the project, a number of problems had surfaced. Chief amongst these were the personnel changes within the six research groups, the changes in the corporations providing data to the research groups and the many interdependencies which meant that the failure of any group to get their data impacted the entire project. Ultimately this meant that the full EIOA was only done by the entire group for the baseline year of the study (1998) and that the research project ended before the second phase EIOA (for 2004) was done. For this case study, namely the dairy sectors, data was collected for both 1998 and 2004 which allowed computation of eco-efficiencies and an evaluation of the sectors' performance over time. The remaining discussion on validity and reliability focuses just on the case study and not on the entire research project.

Factors affecting the internal validity and reliability of the dairy sector research.
The first factor to be considered is the change in the dairy sectors over the time frame of the research. For the dairy farming sector, milk production became substantially more efficient in terms of use of land, electricity, water and lime primarily because of more intensive farming using more fuel and more fertilizer. Milk production increased from 11.4 million tonnes to 15.1 million tonnes over the 6-year period. The most profound change happened in the structure of the dairy processing sector (Figure 1).
In 1998 there were 14 major dairy processing companies in New Zealand and all dairy exports were handled  In 2004 these had coalesced into just 3 major processing companies (Fonterra, Westland and Tatua Co-operatives), each in charge of its own export business. Fonterra is significantly larger than the other two co-operatives, processing about 95% of the country's milk. Despite such a profound structural change, a comparison of 1998 and 2004 parameters is still valid for the dairy processing sector because the processing facilities themselves did not change significantly; they merely changed corporate governance. The second factor affecting the validity and reliability of the case study results is the quality of the data including the data source, sample size and accuracy. For the dairy farming sector, the data sources include government reports, corporate reports, internal company datasets, company sales data, research articles and personal communication with engineers, accountants, environmental strategists and agricultural sales representatives. Estimating the data quality is a very difficult task. For example, the economic data for dairy farming was derived from the MAF (2004) dairy farm monitoring report. The report merely states that the data for each of 5 regions came from 20 farmers with input from agribusiness and is thought to be representative of about 83% of the farms in a particular region. There is no information on the standard deviation of the data, either within the 83% of farms or over the region as a whole. The New Zealand government collects this and other data annually and uses it to monitor the country's agricultural production, which implies that the data is important and valid but still gives no quantification of its accuracy. In a personal communication, the MAF officer responsible for the Canterbury region data suggested that the 2011 data was accurate to roughly +/-5% over the region. In some cases, a second source of data was found and used to judge the accuracy of the reported data. However, even this method has the intrinsic problem that generally there is an error in the second source of data and usually this error is not reported. Despite the difficulties and limitations, best estimates of the quality of the data are summarised in Table 8.
For the dairy processing sector, Fonterra provided access to their internal databases on annual (2004) financial statements, energy surveys, production surveys, transport accounts, waste reduction programme and major site water balances. The economic data from Fonterra corresponded to the processing of 95.3% of the total milk collected and this data was scaled up to account for the processing of domestic milk (1.7%) and for the processing by Tatua and Westland (3%) (MAF, 2004b). Fonterra also provided the historical records for the 12 companies in 1998 that later merged into Fonterra. Access to the Fonterra data was granted with the condition that they The assumed ratio of diesel to petrol (discussed above) will affect the accuracy of the CO 2 emissions by as much as an additional 17%.
nd: not determined no other source of data was found at the national level. CO 2 : carbon dioxide could embargo any commercially sensitive data. Under this condition, they released the 1998 data for publication but only allowed a small subset of the 2004 data to be published because of commercial sensitivity constraints. Table 9 summarises the quality of the data.

External validity of the case study.
Research findings that agree with other published studies are generally held to demonstrate good external validity in the research. However, comparisons are usually difficult because of differences in the studies from factors such as differing operation parameters and differing time frames. Further, in the case of national scale eco-efficiencies for dairy processing there are no comparable published studies. Therefore the remainder of this discussion will consider comparison of the findings for the dairy farming sector only. MacLeod and Moller (2006) report that for New Zealand dairy farming from 1994 to 2002, there was a 34% increase in the number of animals and a 12% increase in the area used for dairy farming. They also report an increase in bovine milk yield of 0.78% per annum. From this the change in the land eco-efficiency in hectares per kilogram of milk over the eight years can be computed as follows: Let the number of dairy cows in 1994 be x, the milk produced per cow be y kg/cow and the area used for dairy be z hectares. This means that MacLeod and Moller (2006) found that the New Zealand dairy farming sector used 21.5% less land per kilogram of milk produced over the 8-year period from 1994 to 2002. This is close to the 27.0% reduction in land eco-efficiency reported here for the 6-year period from 1998 to 2004.
The remaining dairy farming studies report LCA results which look at total (direct plus embodied) data while the results reported here are for the direct contribution only. With these limitations in mind, the data presented here is compared with the data reported in DeVries and de Boer (2010) on LCA studies for non-organic farming in OECD countries, including 14 studies on milk over the period from 1996 to 2005. Of these 14 studies, one (by Basset-Mens et al., 2009) is for an average New Zealand Table 9. Dairy processing sector data quality Input or Output Accuracy (% +/-)

Data source and justification
raw milk and CO 2 emissions from fuel 1 From detailed annual energy, production and transport records kept by Fonterra (for 95% of the milk processed), with slight error due to scale-up.
Water use 10 Derived from water balance data from 8 processing sites (handling 69% of the total milk processed) for 2002 and 2004. The milk processed in these two years was very similar but water use decreased dramatically from 2002 to 2004, indicating a trend towards more efficient use of water. The average was assumed for 2004 and was scaled up by a factor of 100/69 to give the water used nationally.
Water-based effluent 10 The 2002 water balance was used to get a ratio of water in: water out which was assumed to be constant for 2004.
Dairy products 1 From Fonterra production records (for 95% of the milk processed), with slight error due to scale-up.
CO 2 : carbon dioxide; Dairy products include milk, butter, anhydrous milkfat, frozen cream, cheese, whole milk powder, nutritional products/infant food, skim milk powder, buttermilk powder, casein products, lactose, whey products and ethanol dairy farm for the 2004/05 season. New Zealand has a temperate climate which makes its dairy farming significantly more eco-efficient than dairy farming in Europe. The eco-efficiencies from the LCA studies are shown in Table 10 together with those found in this work. There is close agreement between the results reported here and those of Basset-Mens et al. (2009) for land eco-efficiency but the fuel energy eco-efficiencies are markedly different, being much lower in this work. The reason for this is that eco-efficiency data reported here does not include any embodied contribution. Total fossil energy consists of the direct fuel used by the dairy farming sector plus a substantial fuel use embodied in resources such as feed and fertilizer used by the dairy farming sector.

MAIN FINDINGS AND CONCLUSIONS
The main findings of the case study show that over the six-year period from 1998 to 2004, New Zealand milk production increased from 11.4 to 15.1 million tonnes (a 32% increase). Over that time period the dairy farming sector became significantly more efficient in terms of its use of land (-27%), electricity (-12.4%), water (-20.5%) and lime (-15.6%) and produced significantly less water-effluent (-19.7%). It was slightly less efficient in its use of fuel (1.59%) and fertiliser (6.34%). Production from the New Zealand dairy processing sector increased from 1.9 to 2.6 million tonnes (a 37% increase) and for each kilogram of milk processed, it produced almost 5% more dairy product, used 21% less water and discharged 21% less effluent water. The fuel use for milk transportation from the farms to the processing centres was 14% more efficient.
The case study was part of a much larger research project involving six research groups working independently on a very complex collaborative effort to measure the ecological footprint of the country's food and fibre sectors using Environmental Input-Output analysis. The original goal was to assess the ecological footprint in 1998 and again in 2004 but the lengthy data collection process and the interdependency of the six groups meant that the project ended before completion of the analysis for 2004. The time taken for data collection and analysis were factors that ultimately had an adverse effect on the internal validity of the research design.
For the dairy sector case study, the internal validity of the research was good despite significant changes to the structure of the dairy processing sector over the six year period. Commercial sensitivity had a negative impact on the results for 2004. External validity was examined in the context of differences in dairy sector operation such as different boundaries, different climates and different time frames for published studies. The New Zealand dairy processing sector is unique because it controls the milk collection from the farms; in other OECD countries, this activity is controlled by the dairy farming sector. Despite some variation in external factors, the case study findings for the dairy farming sector were compared with other studies and showed some agreement.
The reliability of the case study results was examined in terms of the quality of the data looking at factors such as the data sources, sample sizes and data accuracy. Data for the dairy farming sector came from a range of sources, with widely varying accuracy that was sometimes hard to quantify. Data for the dairy processing sector was extremely reliable, with most data coming from Fonterra's databases and representing the processing of 95.3% of the total milk collected in New Zealand.
In conclusion, this case study provided useful information on the performance of New Zealand's dairy industry over a six-year time period, despite being limited by commercially sensitive issues. It was part of a much larger research project, with a very complex research methodology that ultimately proved to be fragile because of the inter-dependencies within the project. This is a common failing in large projects involving collaboration amongst several different research groups.

Disclosure statement
No potential conflict of interest was reported by the author.

Notes on contributor
Claire Flemmer holds a PhD in mechanical and aerospace engineering and is now a senior lecturer at Massey University's School of Built Environment in New Zealand.