February 25, 1999

EPA-SAB-EEC-LTR-99-002

Honorable Carol M. Browner Administrator U.S. Environmental Protection Agency 401 M Street, SW Washington, DC 20460

Subject: Science Advisory Board Review of the Implementation of the Agency-Wide Quality System

Dear Ms. Browner:

In response to a request from the National Center for Environmental Research and Quality Assurance (NCERQA) of EPA's Office of Research and Development, the Quality Management Subcommittee of the Science Advisory Board's (SAB) Environmental Engineering Committee (EEC) reviewed the implementation of the Agency-wide Quality System (QS). The Subcommittee met in Washington, DC on September 22-24, 1998.

In the Charge for this meeting, the EEC Quality Management Subcommittee was asked to use available information to evaluate the Agency's success in implementing the Agency-wide Quality System. The Subcommittee restricted its review of implementation to data collection because the Agency's Quality System presently does not address other activities, such as modeling or environmental technology.

At an earlier meeting, the Subcommittee addressed the relevance, completeness and practicality of the Agency's Quality System in a letter report (EPA-SAB-EEC-LTR-98-003). The focus of the Agency's Quality System on data collection, rather than other activities, was documented as an issue of concern in the previously cited SAB report (The complete Charge for both meetings is provided in Enclosure A, and Enclosure B contains a summary of the findings detailed in the previous report).

The Subcommittee commends the Agency for developing its Quality Program and recognizes the Quality Assurance Division (QAD) for its precedent-setting work. The Subcommittee also credits the QAD for its efforts to implement the Quality System, including:

a) the generation of widely-accepted policy and project-level guidance and requirements;

b) a training program that has introduced the Quality System concepts to many Agency personnel;

c) a review and approval process for Quality Management Plans (QMPs) that has guided implementation of the Agency's Quality System;

d) the use of management system reviews (MSRs) to identify areas of needed improvement and success stories for Quality System implementation; and

e) the use of Quality Assurance Annual Reports and Work Plans (QAARWP) to document past and planned efforts to implement the Quality System.

The Subcommittee wishes to draw the following overarching findings to your attention:

a) Implementation of the Quality System is uneven within the Agency, increasing the likelihood of problems with data quality and the associated decisions.

b) The majority of states that have primacy for implementing numerous environmental programs are generating data of unknown quality because they lack approved Quality Management Plans (QMP).

c) A well-implemented Quality System is necessary to accurately measure progress under the Government Performance Review Act (GPRA).

d) Incomplete implementation of the Agency's Quality System precludes proper evaluation of internal and external activities; this results in potential for waste, fraud and abuse.

e) The present reporting status of the Quality System function within the Agency organization lowers the profile of the Quality System and denies access to the proper level of authority and the independence necessary to oversee the quality of the Agency's services and products.

f) Senior Management needs to be a champion for successful implementation of the Agency's Quality System or acceptance and implementation of the Quality System will remain uneven and incomplete.

These findings are discussed in more detail below (in addition, findings and recommendations specific to Quality System Organizational and Program components will be found in Enclosure C; Enclosure D addresses Project level components).

a) Implementation of the Quality System is uneven across the Agency: After reviewing program and regional Quality Management Plans (QMP), management system review (MSR) reports, reviews, surveys, and conducting interviews, the Subcommittee concluded that implementation of the Agency's Quality System varies across the Agency, within regions and programs, and among states and grantees.

Because the QMP is the blueprint for implementing an organization's Quality System, the QMP is an indicator of the importance an organization assigns to the Quality System. Timely, coherent, and approved QMPs are reassuring. Organizations that lack a QMP, that are tardy in preparing the QMP, that operate for extended periods without an approved QMP, or that suffer from internal inconsistencies between QMPs at different levels are worrisome. Such behaviors imply that compliance with the Agency's Quality System is unimportant to the organization. The Subcommittee found actual implementation of the QMP requirement varied significantly, from some Agency organizations that functioned under a detailed and approved QMP to some states that lacked QMPs, in their entirety.

The organizational authority for Quality Assurance Managers varies from organization to organization as does the source and availability of funding. Typically, funding of regional Quality System activities is negotiated annually with the programs. The Superfund program, which is commonly the largest source of funding for Quality System activities, not surprisingly has implemented more Quality System activities than other programs.

Implementation of the Quality System at the project level depends upon the manager or supervisor. Where champions exist, Quality System activities such as a structured planning process and the use of quality assurance project plans (QAPP) are more likely to be employed. At the project level, the unevenness of implementation may also reflect the different emphasis on training in quality management that exists across the Agency.

The Subcommittee believes that this uneven and inadequate implementation of the Quality System can impede achievement of Agency goals.

b) The majority of states that have primacy for implementing numerous environmental programs are lacking approved Quality Management Plans (QMP), and most likely are generating data of unknown quality: States with primacy for the implementation of many of the Agency's environmental programs have legal responsibility to protect the environment and human health. More than 75% of states lack approved Quality Management Plans (QMP) for all or significant numbers of their environmental programs. Subcommittee fact-finding with a limited number of states discovered that some states not only lacked an approved QMP but also lacked quality assurance managers (or equivalent positions) and any semblance of an independent quality assurance function. States lacking a Quality System for environmental programs are unlikely to document the quality of data. Such a state is exposing itself, the reliability of its decisions, and its credibility, to criticisms due to its reliance upon data of unknown quality. The same is true for those Agency programs that depend upon those data.

c) A well-implemented Quality System is necessary to accurately measure progress towards compliance with the Government Performance and Results Act (GPRA): The intent of GPRA is to improve public confidence in federal agency performance by holding agencies accountable for achieving program results. GPRA's goals include:

(1) improving the accountability of federal agencies by setting annual performance objectives;

(2) developing a means of measuring progress; and then

(3) measuring and documenting the progress made toward reaching objectives.

Thorough implementation of the Agency's Quality System will produce data that account for progress made towards GPRA-related objectives. This implementation should include a structured planning process (e.g., Data Quality Objective, DQO, process), project plans, and appropriate data assessment. While the DQO process cannot assure that the objectives chosen are the appropriate ones or that the tools are available to measure progress towards the environmental objectives, the DQO process can result in the ability to benchmark and measure progress in achieving objectives and therefore will increase public confidence and facilitate oversight by Congress.

Data collected under an appropriate Quality System will be of known quality, and as such a determination can be made about the usability of the data in terms of their intended use. Data of unknown quality are problematic, since the data may lead to unjustified confidence and inappropriate use of the data. It is not possible to evaluate the quality of data or the quality of associated decisions, projects and activities, unless the data are generated under the auspices of a program that defines their quality. Questions such as whether a site has been cleaned, a contractor has completed a project, a facility is in compliance can not be confidently answered.

d) Incomplete implementation of the Agency's Quality System precludes proper evaluation of internal and external activities, with a resulting potential for waste, fraud and abuse: The Agency's Quality System contains the necessary components to generate data of known quality. These, or equivalent components, are essential for identifying the objectives of a data-gathering activity, documenting important planning issues for subsequent implementation, and assessing the suitability of the resulting data for their intended use.

When the Agency's Quality System is not properly implemented, one or more of the following events is likely to occur:

When the Agency's Quality System is not properly implemented, one or more of the following events is likely to occur:

(1) the objectives for collecting data are poorly or incorrectly identified, data are collected for the wrong reasons and the resulting data are of diminished value;

(2) the incorrect type, too much or too little data are collected;

(3) the spatial and temporal boundaries of the study are not properly defined;

(4) constraints and limitations are not accounted for by the plan or during data collection;

(5) the necessary degree of confidence in decision-making is not considered nor accommodated;

(6) the sampling and analytical activities are not implemented according to a well-designed plan; and

(7) the data are not properly assessed to determine their usability. These events result in data of unknown quality.

Data of unknown quality are problematic. Typically, such data are unsuitable for decision making or determining whether a project has been successfully completed. Collecting data without adherence to the Agency's Quality System (or an equivalent system), is apt to result in data that cannot achieve the objectives for which they were intended.

The Subcommittee is convinced that a more complete application of the Agency's Quality System to data collection projects will provide more usable data, better decisions, and improved oversight of internal and external data collection activities. The Quality System is designed to reduce EPA's vulnerabilities by producing data of known quality, an evidentiary trail, and reduced sampling and analytical costs (by specifying a better match between EPA's information needs and data collection). At present the potential for waste, fraud and abuse exists due to the incomplete implementation of the Quality System.

e) The reporting status of the Quality System function within the Agency organization lowers the profile and diminishes the effectiveness of the Quality System: The Subcommittee's earlier letter report concluded that the Agency's current organization doesn't allow national level QA managers adequate access to the proper level of authority within the Agency (Refer to Appendix B). Additional documents examined by the Subcommittee and the Subcommittees's discussions with Agency and state personnel indicate that the problem of access also exists at the regional, program and state levels. The Subcommittee found the reporting status of the QA authority correlates with the degree to which the Quality System had been implemented. The need to have quality assurance managers report at a senior level is expressed clearly in EPA Order 5360.1, and is an accepted practice within the Quality Assurance community.

"The Quality Manager shall have direct access to the highest level of management" (International Standards Organization Guide 25, "General Requirements for the Competence of Calibration and Testing Laboratories"): The need for quality assurance managers to report to a high level is also implied by the repeated requirement that the QA function be independent of those doing the work.

(1) "It is highly desirable that the QA function or person be independent of the functional groups that generate the data." (40 CFR 1546.2 Contract Quality Requirements);

(2) "The quality assurance unit shall be entirely separate from and independent of the personnel engaged in the direction and conduct of the study." (21 CFR 58 Federal Drug Administration and 40 CFR Part 160.35, Federal Insecticide, Fungicide, Rodenticide Act); and

(3) "The QA monitoring function shall be entirely separate from, and independent of personnel engaged in direct supervision or performance of the work being monitored." (ASTM D5283-92)

The predominant philosophy within the QA community emphasizes the need for independence of the QA function and direct access to the highest levels of management. It is inappropriate to argue that these accepted practices are applicable to other and lower tier organizations but not applicable to the Agency, the leading QA authority on environmental issues.

The Subcommittee recommends that the Agency place the Quality System at a higher level within the Agency structure so that it can function properly and so its position reflects the importance of its role within the Agency. The Agency's ability to collect data of known quality adequate for its decisions depends upon a longterm Agency-wide commitment to quality that will withstand changes in administrations. If the Quality System is perceived as a passing management fad or a factor that can be ignored in doing the Agency's business, the Agency will remain at risk on data quality issues.

f) Senior Management needs to be a champion for successful implementation of the Agency's Quality System: The Subcommittee found that the Quality Assurance Division (QAD) has generated great enthusiasm for quality assurance amongst a cadre of officials who are distributed throughout the EPA and have data collection responsibility. These are the early adopters motivated by their shared vision of the potential benefits that the Quality System can provide. To date, quality assurance has been primarily a social movement within the EPA (and associated state and tribal organizations) with QAD as the chief cheerleader. To gain wider diffusion will require senior management to implement a more complex web of persuasion, administrative mandates, and furtherance of rewarded quality practices. Studies of innovation call this moving up an "S" curve from early adopters to wider diffusion. However, these studies also indicate that most innovations fail on the rocky shoals of bureaucratic inertia. The committee is concerned that this might be a real possibility for the Quality System. Clearly, the benefits of the Quality System have not been sold to a large number of EPA officials, state and tribal officials. While this may be understandable given constraints on time and resources, it also poses dangers for the sustainability of quality assurance as a function within the Agency.

The following ideas emerged in the Subcommittee's review and may assist senior management in providing the means to counter incomplete buy-in and encourage wider acceptance of the Quality System within the Agency, states and tribes:

a) Consider revisiting the reporting status of the Quality System and institutionalizing it within the Agency structure.

b) Create senior and lower level champions for the Quality System within EPA, states and tribal organizations. Recognize and reward those exemplars who incorporate the Quality System so other EPA, state and contractors can emulate them. The Agency may wish to expand its present QA awards program to included exemplars from the states, tribes, grantees and contractors.

c) Emphasize the bench marking and oversight advantages of the Quality System as management tools and for complying with requirements such as those established by the GPRA. The Subcommittee was impressed with the potential of a wellimplemented Quality System to facilitate compliance with GPRA. A better understanding of the synergistic relationship between GPRA and an operable Quality System should advance the movement of the quality system from a peripheral activity to a central role within the Agency.

d) Articulate the need to have independent oversight of the quality of the Agency's products and services and how this oversight will add to the Agency's ability to protect human health and the environment as well as to gain increased credibility and legitimacy when dealing with political leaders and the public.

e) Articulate the benefits and cost reductions that will eventually accrue following incorporation of a Quality System within the Agency structure.

The Subcommittee finds that the Quality System cannot be successfully implemented without buy-in and demonstrated commitment from senior management and agrees with the following quote from the ANSI/ASQC E4-1994 consensus, standard upon which the Agency's Quality System is based - "While management delegates quality management functions to staff, they cannot abrogate the ultimate responsibility for the success of the quality system."

The Subcommittee also finds the Agency to be the national and international leader for quality assurance activities within the environmental community. To maintain this leadership position, and to continue to improve the data upon which environmental decisions are made, the Agency's senior management must be prepared to assign resources to the implementation of the Quality System. The Subcommittee believes that the Agency's senior management and Congress must recognize that initially, as the Quality System is implemented, there is the potential that the quality of products and services will improve at the expense of the total amount of work performed. The benefits of a Quality System have been argued to be free of costs, but the validity of this assumption is based on the amortization of costs over the longer term. While the Subcommittee can not estimate the incorporation costs, the Subcommittee is confident that the return on investment will be realized.

In conclusion, the Subcommittee commends the NCERQA and QAD management for their request for a review of the Agency-wide Quality Management Program and the openness and assistance of the QAD staff who faithfully responded to our continued requests for information and answered our numerous questions.

As stated in the earlier report, the above findings and recommendations attest to the significant contributions made towards the quality of data collection activities while recognizing that the job is not nearly done. Continued and increased attention from senior management is needed to implement the Quality System uniformly to all activities on an Agency-wide basis.

We look forward to your response to the advice contained in this report.

Sincerely,

/signed/

Dr. Joan Daisey, Chair
Science Advisory Board

/signed/

Dr. Hilary Inyang, Chair
Environmental Engineering Committee
Science Advisory Board

/signed/

Dr. John Maney, Chair
Quality Management Subcommittee
Environmental Engineering Committee

ENCLOSURE A

CHARGE FOR REVIEW OF THE QUALITY MANAGEMENT SYSTEM

a) To evaluate the relevance, completeness, and practicality of the policy and organizational components of the Agency-wide Quality System.

(1) Relevance: Evaluate whether the policy and organizational components of the Agency's Quality System are applicable to the Agency's mission statement, goals (EPA/190-R-97-002) and those activities identified in Section 1.3 of the Quality Manual.

(2) Completeness: Evaluate whether the policy and organizational components of the Agency's Quality System are structured such that the quality of all Agency activities needed to comply with the Agency's mission statement and goals will be monitored and assessed versus performance measures.

(3) Practicality: Is the present structure of the policy and organizational components of the Agency's Quality System designed for success (i.e., will the policy and organizational levels of the Quality System properly assess and control pertinent activities and facilitate achievement of EPA goals).

b) To evaluate the relevance, completeness, and practicality of the project level of the Agency-wide Quality System.

(1) Relevance: Evaluate whether the project level of the Agency's Quality System is applicable to the Agency's mission statement and goals (EPA/190-R-97-002) and the covered activities identified in Section 1.3 of the Quality manual.

(2) Completeness: Evaluate whether the project level of the Agency's Quality System is structured such that of all project activities needed to comply with the Agency's mission statement and goals will be monitored and assessed versus performance measures. Evaluate whether the project level guidance documents consider all essential aspects necessary to monitor and measure the quality of environmental measurement data.

(3) Practicality: Is the present structure of the project level of the Agency's Quality System designed for success (e.g., are they cost-effective, efficient to implement, understandable by the intended audience, will the project level of the Quality System facilitate achievement of EPA goals).

c) Use available information to evaluate the Agency's success in implementing the Agency-wide Quality System.

ENCLOSURE B

SUMMARY OF ISSUES NOTED TO EPA MANAGEMENT AT THE SUBCOMMITTEE'S INITIAL PUBLIC MEETING

a) The need to include all activities, which have the potential to affect the quality of the Agency's products and services, under the auspices of the Quality System;

b) The need for Agency management to review the appropriateness of the reporting status of the Quality System function within the Agency organization. The Subcommittee recognizes that this recommendation is of a policy nature. However, the Subcommittee believes this recommendation is justified due to the impact of reporting status on the efficacy of the Quality System;

c) The lack of an Agency-wide focal point for quality issues and needed corrective actions;

d) The need for Quality System training to be expanded to include the training of senior management;

e) The need to identify metrics for "bench marking" existing levels of quality and changes over time;

f) The need for guidance appropriate for the development and use of mathematical/computer models and the associated data; and

g) The need to determine if budgeted resources for QAD are sufficient to meet the increased demand for its services.

 ENCLOSURE C

Implementation of the Organization/Program Components of the Quality System

This enclosure addresses the Subcommittee's findings, organized according to the various Quality System components that had been implemented at the time of the review.

Implementation of Quality Management Plans

Finding: The subcommittee felt that the sluggishness in updating QMPs was symptomatic of a more general lack of commitment to the Quality System by senior managers.

Recommendation: The Subcommittee recommends that the Agency provide incentives for states, contractors, and grantees, to complete their QMP.

Recommendation: The Subcommittee recommends that the Agency explore means of making the existence of the QMP and a summary of its contents and its significance more available to its staff.

The Quality Management Plan (QMP) is the first step in creating data of known quality. The QMP provides an overview of responsibilities and lines of authority with regards to quality issues within an organization. EPA managers have described the QMP as similar to the organization's legal agreement for implementation of the Quality System.

The percentage of EPA operations currently covered by QMPs is unknown. There are several Agency organizations that claim exemption from QMPs because they do not collect data as part of their function. Of those EPA organizations identified as organizations that collect data, over 90% have an approved QMP. This extent of coverage does not extend, however to the states, contractors and grantees. None of these groups is required by law or regulation to complete a QMP. EPA officials report that a minority of the groups has elected to complete a QMP. Given the fact that the states are major data sources for the EPA, the absence of a QMP by these groups is a weakness in the quality system. The Subcommittee recommends that the Agency provide incentives for states to complete their QMPs.

Due to the relevance of the QMP to operations, the QMP should be a document of great interest to senior managers and a document in whose design and development they would likely want to participate. For example the QMP is the natural document for each organization to identify success criteria for the Quality System. The QMP can be the vehicle for generally linking policy goals to quality practices in data collection. Senior Managers can also use the QMP for articulating how the Quality System will be used to measure achievement of objectives, a tool, which will be helpful for management oversight and compliance with GPRA.

Managers have described the QMP as one of the lesser-known documents within EPA. When quizzed, few managers had seen their QMP, much less have a familiarity with the content. More generally, the subcommittee finds that there needs to be a greater awareness of the existence and content of the QMP amongst EPA managers and staff. The Subcommittee recommends that the Agency explore means of making the existence of the QMP and a summary of its contents and its significance more available to its staff. Several EPA officials suggested the use of a web site to facilitate easy access.

Finally, a barrier to successful implementation of the Quality System is the relative slowness observed in updating QMPs. This is particularly acute following reorganizations by EPA offices. The Subcommittee felt that the sluggishness in updating QMPs was symptomatic of a more general lack of commitment to the Quality System by senior managers.

2. The Implementation of Management System Reviews (MSRs)

Finding: The Subcommittee found that the MSRs are a necessary component of the overall Quality System and serve the purpose of identifying problems that need to be addressed to improve the Quality System.

Finding: The Subcommittee found that the delayed completion of the final MSR report decreased the overall effectiveness of the MSR.

Recommendation: The Subcommittee recommends that implementation of MSRs and completion of MSR reports should be given high priority.

Management System Reviews are conducted periodically with a goal of 3-4 years as defined in EPA Order 5360.1. The MSR process consists of a site visit; a draft report that details findings and recommended corrective actions, consideration of the reviewed organization's formal response to the draft report and the authoring of a final report. There are 40 divisions and regions which are required to undergo a MSR, either as part of their own QMP (as is the case for regions) or in response to another QMP (for example, the Office of Science and Technology under the Office of Water). According to the information the subcommittee received, for the most part MSRs are now conducted on the correct schedule (approximately every 3-4 years) and the ones which have not been conducted appear to be a result of the QMP being under review or just approved.

The Subcommittee found that the MSRs are a necessary component of the overall Quality System and serve the purpose of identifying problems that need to be addressed to improve the Quality System. Informal interviews by the Subcommittee and a survey taken by the Office of the Inspector General (OIG), supports this claim since it was concluded that QA managers (QAM) found MSRs as a useful means to bring attention to areas of needed improvement and a useful mechanism for communication between QAD and the office/region being reviewed. In addition,  the QAMs found that the MSR comments could function as a change agent by bringing Quality System issues to the attention of management.

A major problem with the MSR process has been the timeliness of the final report. During discussions with QAMs, some had expressed the belief that MSRs would be more effective if they were more timely. In some cases it is the slow turnaround of the initial draft report; however, in others it is the fact that comments from the reviewed organization were not received in a timely fashion. While QAD has limited resources to complete the MSRs, they should be given high priority since they are the only tool available for formalizing corrective actions and giving the office/region a bench mark for improvement (following several MSRs). The Subcommittee found that the delayed completion of the final MSR report decreased the overall effectiveness of the MSR.

As mentioned in an earlier report prepared by this Subcommittee (EPA-SAB-EEC-LTR- 98-003), the Subcommittee recommends that the Agency investigate whether more frequent use of MSRs would facilitate broader acceptance of the Quality System.

3. Implementation of Training

Finding: The Subcommittee found the Agency's overall strategy or formal program for training in the Quality System is inadequate.

Recommendation: The Subcommittee recommends that QAD provide general guidance on how to effectively identify those staff needing training, and on how to design appropriate learning activities.

Recommendation: The Subcommittee recommend that QAD develop record keeping procedures and systems, so the Agency can effectively focus its resources and training capacity.

Recommendation: The Subcommittee applauds the above positive efforts and urges the Agency to identify means for supporting a comprehensive training strategy and program which will ensure effective implementation of the Quality System throughout EPA.

The Subcommittee found that the Agency's overall strategy or formal program for training in the Quality System is lacking. The Subcommittee's first report on the Quality System (EPASAB- EEC-LTR-98-003) recommends that the QAD develop a comprehensive strategic plan for training. Factors that are hindering EPA's ability to effectively implement the Quality System at all levels include: the lack of a long-term vision to meet the Agency's Quality training needs, the lack of clarity about QAD's and others' roles, the inconsistency of training requirements and opportunities, the infrequent tailoring of course content for the audience, the lack of wide support for course attendance, and the lack of resources allocated to training. C-4

The Quality training strategy should clearly identify: Agency-wide goals and objectives for the program; the target audiences; the type and level of training needed by job responsibilities; approaches to design and deliver effective learning experiences (e.g., courses, workshops, conferences, etc.); means to evaluate the short- and long-term effectiveness of training opportunities; the approach for consistent record keeping; and the means to obtain the ongoing resources necessary to support training.

Currently, few programs, offices or regions have identified which staff and managers need Quality training or what types and levels of training would be appropriate for them. During the review, the Subcommittee was not able to locate an inventory of personnel's Quality System learning needs by job responsibilities. The Subcommittee recommends that QAD provide general guidance on how to effectively identify those staff needing training, and how to design appropriate learning activities. Not only should training be tailored to the audience's technical needs and work context, but the format of the activities should also be suitable for the audience's method of learning. For example, most adults learn more effectively when their work experiences are incorporated into the training process, and when they are actively engaged during the course.

Although the Subcommittee found evidence that some trainers use exit evaluation forms to assess the audience's reactions to the training, no long-term (e.g., 3 or more months after training) evaluation processes were found. Resources could be focused on determining the longterm effectiveness of training for selected courses for different levels of staff and managers. Over time, evaluating whether attendees use the knowledge and skills they learned would provide valuable information to improve future training activities.

There is very little evidence that personnel are aware of what training is available. There appears to be no systematic record keeping of who has been trained or which courses were offered, even when specific courses are required of specific staff. Throughout most of the Agency, records are insufficient to assure that training needs are either inventoried or met. The Subcommittee recommend that QAD develop record keeping procedures and systems so the Agency can effectively focus its resources and training capacity.

Where Quality System responsibilities have been decentralized, the number of personnel involved in Quality and requiring training has increased. This situation presents a particular challenge for the Agency when resources allocated for training activities are very limited. It is important to consider the extent of resources required to support decentralized Quality systems. Strategies need to be developed to obtain the essential resources for training to implement a successful Quality System.

The Subcommittee noted some exemplary training initiatives, which were exceptions to the above concerns. In one region, technical staff recognized specific training needs and, above and beyond their routine duties, committed their time, expertise and energies to meet the needs. Some regions reach out to state, tribe and contract staff to invite them to training activities. C-5 Others conduct monthly conference calls or meetings to ensure that lessons learned are shared broadly.

The Subcommittee recognizes that QAD staff understand the importance of tailoring courses to the audience's responsibilities and work context. They have begun to design training guidance, modules and materials more targeted to the audience's routine work. In collaboration with staff from one office, QAD is developing a new approach to address training concerns.

The Subcommittee applauds the above positive efforts and urges the Agency to identify means for supporting a comprehensive training strategy and program which will ensure effective implementation of the Quality System throughout EPA.

4. Implementation of the Quality Assurance Annual Report and Work Plan

Recommendation: The Subcommittee recommends that the Agency encourage all organizations to complete QAARWPs in a timely manner and that the content of these QAARWPs be sufficient for the bench marking of quality, the measurement of changes in quality and the planning of the Quality System.

The purpose of the QAARWP is to provide information on the previous year's QA/QC activities and those planned for the current year. The QAARWP details current and planned resources for the management and implementation of QA/QC activities, training, accomplishments, assessments and facilitates communication between QA staff and management. The QAARWP functions as an important management tool at the organizational level as well as at the Agency-wide level when QAARWP supplied information is compiled across organizations.

The Subcommittee had limited access to QAARWPs, however additional information was gleaned from interviews with QAD personnel and organizational QAMS. The Subcommittee found the QAARWPs to be:

(1) an excellent strategic planning tool for organizational and Agency-wide QA activities;

(2) a mechanism for raising the visibility of QA issues within an organization and on an Agency-wide level;

(3) a mechanism for bench marking quality and changes over time; and

(4) a platform for recognizing accomplishments related to QA.

While the EPA Quality Manual emphasizes the importance of the QAARWP to the Agency's Quality System, interviews indicated that compliance with its annual requirement was  variable across organizations. In fact, information presented to the Subcommittee indicates that 1997 was the first year that all 10 regions submitted a QAARWP and of the regions, research labs, centers and offices that complied, the content and usefulness of the QAARWPs varied. In response to QAD's encouragement and critiques, organizations have, over time, improved the content and value of the average QAARWP. However, the Subcommittee recommends that the Agency encourage all organizations to complete QAARWPs in a timely manner and that the content of these QAARWPs be sufficient for the bench marking of quality, the measurement of changes in quality and the planning of the Quality System.

ENCLOSURE D

Implementation of the Project Level Components of the Quality System

1. Implementation of the Data Quality Objective Process

Finding: the Subcommittee found that adoption of the DQO process by EPA's regions, program offices, the states and tribes has been sporadic to nonexistent.

Finding: The Subcommittee found that there are no demonstrable consequences for managers who apply, or do not apply the DQO process (or its equivalent) in the environmental data collection planning process.

Finding: The Subcommittee found that a lack of statistical expertise was a barrier to implementation of the DQO process.

Recommendation: The Subcommittee recommends that the Agency develop training in, and guidance for, stakeholder identification and participation.

Systematic planning is required by Order 5360.1 to develop acceptance or performance criteria for all operations generating environmental data. The elements of a systematic plan are listed in Section 3.3.8 of the EPA Quality Manual. The EPA Quality Manual also recommends that this required planning be conducted using EPA's Data Quality Objectives (DQO) Process. Detailed guidance in the use of the DQO process has been provided in Guidance for the Data Quality Objectives Process (QA/G-4), (EPA/600/R-96/055) and supplemental guidance in Data Quality Objectives Process Decision Error Feasibility Trials, DQO/DEFT (QA/G-4D), (EPA/600/R- 96/056). The Subcommittee found the DQO process to be relevant, nearly complete and certainly practical as indicated in the Subcommittee's letter report (EPA-SAB-EEC-LTR-98-003). However, the Subcommittee found adoption of the DQO process by EPA's regions, program offices, the states and tribes has been sporadic to nonexistent.

The DQO process apparently has been adopted by the Department of Energy (DOE) for planning the clean up of DOE facilities, indicating that a large government agency can incorporate a structured planning process into their environmental program. However, acceptance and use of the DQO process by the Agency's regions and program offices varies greatly even though training has been provided on the basics of the process. Reasons frequently given for not using the DQO process can be classified as one of three principal categories: administrative, psychological and technical. With regard to the administrative barriers, the Subcommittee found that there are no demonstrable rewards or punishments for respectively applying or not applying the DQO process (or its equivalent) in the environmental data collection planning process. The lack of a requirement to employ the DQO planning process, and the lack of demonstrable consequences for those who do not, may have led to indifference on the part of some project managers.

With regard to psychological barriers, anecdotal information indicates that a fear of admitting that decision errors are possible or a desire to avoid the responsibility associated with defining a tolerable decision error are reasons for avoiding the DQO process. Although the risk of making a wrong decision based on environmental sampling data has always been present, the DQO process requires that this risk be explicitly stated and quantified. For many project managers, this approach represents a paradigm shift in project planning. Most project managers would readily agree that there are errors associated with both analytical and sampling procedures, but admitting that these errors may lead to wrong decisions may be disquieting to them. The psychological barriers to quantification of the tolerable potential decision error may be overcome through involvement of all stakeholders in the planning process and consensus-based decision-making. The Subcommittee recommends that the Agency develop training in, and guidance for, stakeholder identification and participation. In addition, the Subcommittee's recommends that in future revisions to the applicable guidance documents (G-4 and G-4D) QAD more clearly emphasize that correct decisions will be made most frequently when systematic planning such as the DQO process is employed.

Attempts at using the DQO process often stop short of implementing the complete process. Technical barriers are often a major impediment to the implementation of the DQO process. For example, project managers perceive the DQO process to be inflexible and frequently fail to comprehend and apply Step 6 "Specify Tolerable Limits on Decision Errors." Part of the cause for not readily employing Step 6 is that some project managers feel uncomfortable with the statistical nature of this step. To properly specify limits on decision errors requires recognition on the part of the project manager that the probability of making an incorrect decision can be statistically controlled. However, a general lack of appreciation for the consequences of ignoring potential decision errors and a lack of awareness of the added value of the DQO process still persist. For those project managers who are uncomfortable with statistical analysis, the DQO training should be tailored to include a statistical component together with practical examples that illustrate the impact of establishing decision error probability limits on sampling design and costs.

2. Implementation of the Quality Assurance Project Plans (QAPPs)

Finding: The Subcommittee found that while QAPPs are not uniformly employed, the frequency of QAPP usage is significant; yet anecdotal information indicates that QAPPs frequently lack the project specific details necessary for successful implementation.

Recommendation: The Subcommittee recommends that the Agency consider approaches that will encourage the increased incorporation of project-specific details by Agency staff.

The Quality Assurance Project Plan (QAPP) is described as a key component of the Quality System. The QAPP is the principal output of the DQO process and is the project-specific blueprint for obtaining data appropriate for decision-making. Agency policy (Order 5360.1) D-3 requires use of an approved QAPP for any environmental data collection operation in which data are collected for or on behalf of the EPA.

Although, Agency requirements mandate that 100% of environmental data collection operations in behalf of the EPA require an approved QAPP, in practice this is not always the case. For example, the Agency's Inspector General (Report # E1SKB6-09-0041-7100132, March 1987) uncovered data collection activities, including activities in support of a risk assessment, that were carried out without a QAPP and of the 19 QAPPs that were reviewed 14 did not specify DQOs and 11 lacked data assessment requirements. Management System reviews have uncovered Program QMPs and practices that did not require approval of QAPPs prior to data collection.

Due to the intricacies and individual nature of projects, QAPPs should be project specific. However, in many instances (e.g., federal facilities), a generic or off-the-shelf QAPP may be prepared and applied for the entire facility regardless of the number and type of environmental data collection activities that exist. Clearly, this approach to the utilization of a QAPP does not conform to the QAPP preparation guidance provided in EPA QA/G-5, and typically results in a document that lacks the specific technical and quality assurance details necessary for successful implementation of the project. Since the extent of the specific details contained in a QAPP is dependent on the type of project, the use of a generic QAPP would be practical in only limited circumstances.

The EPA QA/G-5 guidance document states that QAPPs must identify and characterize measurement quality objectives pertaining to specific applicable action levels and data quality criteria. These measurement quality objectives may be developed through the DQO process or a similar systematic planning process. The QAPP translates the DQOs into performance specifications and QA/QC procedures for the data collectors. Thus the usefulness of the QAPP will be a function of how well the preceding planning process has identified and defined the DQOs, a previously cited weakness. An improper or incomplete systematic planning will result in an improperly developed or incomplete QAPP.

Because of the potential misuse of generic QAPPs as well as improperly prepared QAPPs, the EPA document QA/G-5 provides a checklist to assist quality assurance (QA) managers in their review of submitted QAPPs. It is unclear to what extent QA managers use this checklist in screening the accuracy and completeness of QAPPs. However, if the checklist were used routinely, improperly prepared or incomplete QAPPs should be readily identified and corrected.

In summary, the Subcommittee found that while QAPPs are not uniformly employed, the frequency of QAPP usage is significant; yet anecdotal information indicates that QAPPs frequently lack the project specific details necessary for successful implementation.

3. Implementation of Data Quality Assessment Guidance

Finding: The Subcommittee found a general lack of awareness by regional, state, and grantee personnel for the G-9 document, its content, and application.

Finding: The Subcommittee found that a lack of statistical expertise was often a barrier to implementation of G-9.

Recommendation: The Subcommittee recommends that the Agency determine whether there are better approaches for increasing awareness of its requirements and guidance documents.

The logical last step in the data quality assurance system is the usability assessment of the acquired data to determine whether they meet the assumptions and objectives of the systematic planning process, which resulted in their collection. In other words, determine whether the data are usable because they are of the quantity and quality required to support Agency decisions. The Agency guidance that addresses the overall data usability assessment process is presently incomplete. Technical Assessments for Environmental Data Operations (EPA QA/G-7) and Environmental Data Verification and Validation (EPA QA/G-8) guidance exists only in working draft form and are not ready for release. Since the Agency's Quality System is presently lacking these key components, this review focused on the implementation of the Data Quality Assessment (EPA QA/G-9) component of the data usability assessment process.

The conduct of a DQA is often sufficiently complex that consultation with a professional statistician is required. QAD has rightly recognized that professional statisticians are not in sufficient supply to meet the demand. Therefore, QAD has prepared Guidance for Data Quality Assessment: Practical Methods for Data Analysis (EPA QA/G-9), (EPA/600/R-96/084) and a supporting statistical software package Data Quality Evaluation Statistical Toolbox (DataQuest) EPA QA/G-9D, (EPA/600/R-96/085). These documents provide a compendium of statistical tools for environmental data evaluation and associated guidance for their use.

While it must be recognized that there is a risk of misuse of these statistical techniques when employed by those who are not statistical professionals, G-9 provides the project team with the tools to evaluate data claims by contractors and other stakeholders. The benefits of having these DQA tools available may well outweigh the risk of their misuse in situations where the DQO process, and/or other systematic planning process, has not been employed.

Regarding how well the G-9 guidance has been implemented; there is limited anecdotal and factual evidence to evaluate its application to the Agency's data collection operations. As a result of interviews however, the Subcommittee found a general lack of awareness by regional, state and grantee personnel for the G-9 document, its content and application; and that a lack of statistical expertise, as for Step 6 of the DQO process, was a barrier to implementation.

NOTICE

This report has been written as part of the activities of the Science Advisory Board, a public advisory group providing extramural scientific information and advice to the Administrator and other officials of the Environmental Protection Agency. The Board is structured to provide balanced, expert assessment of scientific matters related to problems facing the Agency. This report has not been reviewed for approval by the Agency and, hence, the contents of this report do not necessarily represent the views and policies of the Environmental Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor does mention of trade names or commercial products constitute a recommendation for use.

U.S. ENVIRONMENTAL PROTECTION AGENCY

Science Advisory Board

Environmental Engineering Committee (FY99)

CHAIR

Dr. Hilary I. Inyang, Director, Center for Environmental Engineering and Science Technologies (CEEST), University of Massachusetts, Lowell, MA

MEMBERS

Dr. Edgar Berkey, Vice President and Chief Science Officer, Concurrent Technologies Corporation, Pittsburgh, PA

Dr. Calvin C. Chien, Senior, Environmental Fellow, E. I. Du Pont Company, Wilmington, DE

Mr. Terry Foecke, President, Waste Reduction Institute, St. Paul, MN

Dr. Nina Bergan French, President, SKY+, Oakland, CA]

Dr. Domenico Grasso, Head of Department of Civil and Environmental Engineering, Environmental Research Institute, University of Connecticut, Storrs, CT

Dr. JoAnn Slama Lighty, Associate Dean for Academic Affairs, Associate Professor of Chemical Engineering, University of Utah, Salt Lake City, UT

Dr. John P. Maney, President, Environmental Measurements Assessment, Hamilton, MA

Dr. Michael J. McFarland, Associate Professor, Utah State University, River Heights, UT

Ms. Lynne M. Preslo, Senior Vice President, Technical Programs, Earth Tech, Long Beach, CA

SCIENCE ADVISORY BOARD STAFF

Mrs. Kathleen W. Conway, Designated Federal Officer, Science Advisory Board, U.S. EPA, Washington, DC

Mrs. Dorothy M. Clark, Management Assistant, Science Advisory Board, U.S. EPA, Washington, DC 1Provided editorial support for this report, but did not participate in the review.

U.S. ENVIRONMENTAL PROTECTION AGENCY

Science Advisory Board - Environmental Engineering Committee

Quality Management Subcommittee

CHAIR

Dr. John P. Maney, President, Environmental Measurement Assessment, Hamilton, MA

MEMBERS

Dr. Edgar Berkey, Vice President and Chief Science Officer, Concurrent Technologies, Corporation, Pittsburgh, PA

Dr. Hilary I. Inyang, University Professor and Director, Center for Environmental Engineering, Science and Technology (CEEST), University of Massachusetts Lowell, Lowell, MA

Dr. JoAnn Slama Lighty, Associate Dean for Academic Affairs, Associate Professor of Chemical Engineering, University of Utah, College of Engineering, Salt Lake City, UT

MEMBER OF OTHER SAB COMMITTEES

Dr. William J. Adams, Director, Environmental Science, Kennecott Utah Copper Corp., Magna, UT (Ecological Processes and Effects Committee)

CONSULTANTS

Dr. Mohammad A. Ansari, President, American Institute for Pollution Prevention, Chester, VA

Dr. Gordon Kingsley, School of Public Policy, Georgia Institute of Technology, Atlanta, GA

Dr. Michael J. McFarland, Associate Professor, Utah State University, River Heights, UT

Dr. Rebecca Parkin, Director, Scientific, Professional, and Section Affairs, American Public Health Association, Washington, DC

Mr. Douglas Splitstone, Principal, Splitstone & Associates, Murrysville, PA

SCIENCE ADVISORY BOARD STAFF

Kathleen W. Conway, Designated Federal Officer, U.S. EPA, Science Advisory Board, Washington, DC

Samuel R. Rondberg1, Designated Federal Officer, U.S. EPA, Science Advisory Board, Washington, DC

Dorothy M. Clark, Management Assistant, U.S. EPA, Science Advisory Board, Washington, DC

DISTRIBUTION LIST

Administrator
Deputy Administrator
Assistant Administrators
EPA Regional Administrators
Director, Office of Science Policy, ORD
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
Library of Congress
National Technical Information Service
Congressional Research Service