|
Paul Mills, Mentorprises Corporation
July 1999
The development and implementation of performance-based measurement systems (PBMS) was the main focus at the Waste Testing and Quality Assurance Symposium '99 (WTQA) held in July. As promised in the regularly appearing column, PBMS Spotlight, published in the January/February 2000 issue of Environmental Testing & Analysis magazine (Vol. 9, No. 1, p. 10), the following is a full summary of WTQA PMBS sessions that offers a more complete list of the distinguished speakers who presented papers over a two-day period on the status of PBMS, related implementation issues, and varying approaches used for a variety of government, private and third-party programs, as well as detailed abstracts.
WTQA 99 PBMS SESSIONS SUMMARY
Tuesday, July 20 AM-"PBMS Status and Issues"
Chair: Gail Hansen (U.S. EPA)
PBMS implementation progress and issues were presented for U.S. Environmental Protection Agency (EPA) programs and the National Environmental Laboratory Accreditation Conference (NELAC). The Environmental Laboratory Advisory Board (ELAB) recommendations for PBMS, and the results of regional PBMS workshops were reviewed. The status of an American Chemical society (ACS)/EPA PBMS comparison study was described. A position paper on PBMS by the Interagency Methods and Data Comparability Board (MDCB) was given. Key points are summarized below.
• Llewellyn Williams (U.S. EPA-Las Vegas, co-chair, PBMS Workgroup-EMMC)-"EPA Status". Dr. Williams presented a brief summary of the status of PBMS implementation progress at EPA's Office of Solid Waste and Emergency Response (OSWER), the Office of Air & Radiation (OAR), and the Office of Water (OW) programs. He said that state and local governments' concerns about training, legal standing, accountability and enforceability for PBMS are being addressed at OSWER through significant changes to the Resource Conservation and Recovery Act (RCRA) monitoring regulations and cleanup standards. OSWER has developed PBMS seminars for permit writers, regulation writers, and upper management to train them in the use of risk- or technology-based data quality objective (DQO) performance criteria, with "acceptable decision uncertainty" (ADU) expressed as DQOs and minimum quality objectives (MQOs), which will be developed and included in regulations. Appendix VIII will be restructured to meet real-world needs. For new rules, the PBMS approach will be used up front. To reduce industry concerns over lack of opportunities to review and comment on proposed methods, they will be posted for review at the OSW's website at the Methods Team homepage. SW-846 methods are guidance, a fact to be reinforced by dropping any unnecessary regulatory requirements to use SW-846 methods, and by issuing new SW-846 methods through Notices of Data Availability (NODAs) rather than by method promulgation. The Office of Emergency Response and Remediation (OERR) encourages the use of PBMS for field methods, where appropriate. OERR has drafted interim final ADUs for the Superfund program. The publication (EPA 540-R-93-071)-"DQOs for Superfund-Interim Final"-supports the PBMS approach.
The Office of Air and Radiation has dropped method-defined requirements from its PBMS considerations, such as sampling. The OAR retains an up-front approval process for PBMS. OAR will implement PBMS in the Stationary Source, the Fuel, and the Radiation programs. MQOs will be specified for stationary sources as a PBMS option to certain methods, measuring the same analytes, using the same format for methods, applying method-specific quality control (QC), performing peer review of the method and data, analyzing audit samples, and producing and maintaining documentation and certification statements for the lab. The Office of Radiation will publish DQOs/MQOs. The Indoor Air, the Ambient Air Monitoring, the Acid Rain, and the Engines and Vehicles programs do not plan to implement PBMS programs. Indoor Air methods are "guidance," and therefore, PBMS by definition. Methods 111, 114, 115 will have performance criteria established (MQOs).
The Office of Water will implement a "hybrid" reference method/performance criteria approach to PBMS that provides controlled flexibility. No prior approval will be needed for new methods. New methods/modifications will be considered acceptable if the new method is documented and validated according to OW's specified procedures, and if regulators are pre-notified of the intention of a facility to use it. 40 CFR Part 136 contains 26 method-defined parameters, while Part 141 contains eight. CWA 304(h) and SDWA 1401(1)(d) require EPA to publish methods and criteria. OW is working with other EPA offices to finalize a draft PBMS rule package (12/31/99).
• Dr. Jim Pearson (NELAC)-was not present, and Silky Labie (Florida Department of Environmental Regulation) presented material on his behalf. Major changes in Chapter 5 were accepted at the NELAC V meeting in Saratoga Springs, NY. Calibration and minimum detection limit (MDL) requirements that were prescriptive in 1998 were taken out of the 1999 Chapter and replaced with a set of performance standards. Appendix C, "Description of Capability," has been made less prescriptive, to show individuals or team (workgroup) capability. Ethics information and training were added. To date, 11 states have been approved as Accrediting Authorities.
• J. Wilson Hershey (Lancaster Laboratories)-"ELAB Findings--Recommendations for the Implementation of PBMS". Dr. Hershey presented summary details of an ELAB report provided to NELAC and to EPA. These recommendations were previously reported in the article, "ELAB's Recommendations for Implementation of PBMS," by Paul Mills, in Environmental Testing & Analysis, March/April 1999, Volume 8, Number 2. These include a set of recommended "Essential Elements of an Ideal PBMS" (EEIP) that should be considered requirements for a successful PBMS program. Each EPA program office is requested to address the EEIPs in any EPA PBMS implementation plans, guidance, policies or regulations. The "Essential Elements of the Successful PBMS" are: legal defensibility; cost effectiveness; scientifically sound and relevant validation process; good performance criteria; regulatory development; and documentation. Additional "Important Elements of the Successful PBMS" were also recommended in the ELAB report: flexibility; EPA optional approval process; regulatory compliance; consistency; simplicity; clarity of intent; careful implementation; and widely available reference materials. In closing, Dr. Hershey opined that data comparability will be an issue with PBMS, because it is possible that "independently validated methods may give statistically different results."
• Larry Keith (Waste Policy Institute)-"ACS Lessons Learned". Goals of a joint ACS/EPA study were to examine three different environmental monitoring approaches to compare their relative ease of implementation in producing cost-effective answers for regulatory questions, as well as their ability to meet DQOs. Barriers to development and use of innovative technologies were also examined. Initially, six labs were sought to analyze 15 representative samples using real-world matrices, to make two decisions: Did any regulated analytes exceed action levels? What are the levels of analytes of concern for purposes of risk decision-making? In the fall of 1998, the "streamlining" approach was dropped from the study, leaving only a comparison between PBMS and prescriptive methods. DQOs were changed to MQOs. Paper mills' effluent was dropped from the study. Some labs changed owners, and other problems reduced the number of participating labs to three, with one provisional. Samples should be analyzed by the selected labs in Fall 1999.
• Jerry Parr (Catalyst Information Resources)-"GIES PBMS Workshops". With the help of an EPA grant, the research foundation GIES sponsored five regional PBMS workshops in the spring and summer of 1999, with 14 different guest speakers. The regional workshop presentations can be viewed at the following website:
http://www.acil.org/pdffiles/GIES.pdf
. The genesis of the regional workshops was the ELAB recommendation report, previously described by Wilson Hershey. Key findings:
• the regulated entity is responsible for compliance determination;
• legal standing of PBMS is ensured by Daubert principles of law;
• method verification and an effective QC sample program are essential;
• accreditation should be based on data quality needed; and
• new approaches to regulations are needed.
New concepts regarding responsibilities for method validation were presented. The developing lab has initial method development, validation, and documentation responsibility. The establishing lab uses validated methods and has the responsibility to show criteria are met. Without checks, PBMS may be abused. Accreditation provides a rules framework, and helps support PBMS' legal standing. PBMS and NELAC complement each other.
• Andrew Eaton (Montgomery Watson Laboratories)-"MDCB Efforts". (Previously reported in "Reservoir Dogs and Performance Based Systems," by Andrew Eaton and Jerry Diamond in May/June, 1999 Environmental Testing & Analysis). The MDCB consists of 15 delegates, five each from federal organizations, state organizations and private entities, plus a number of other individuals who participate in workgroups and serve as alternate delegates. The MDCB developed a position paper on performance based systems in early 1999 that may be adopted by the National Water Quality Monitoring Council. Key areas of agreement included four essential issues for a working performance based system:
the need for well defined measurement quality objectives (MQOs) or data quality objectives (DQOs) for any program;
• the need for an adequate supply of reference materials to validate a given methodological approach;
• the need for validated methods shown to meet specific MQOs, so that MQOs were not developed in a vacuum;
• the need for adequate training of both chemists and regulators in development of MQOs and validation of methods.
The Board Workgroup is coordinating efforts with NELAC/ELAB, ACS, and several consensus method organizations to develop and validate a pilot program for performance based systems which would clearly identify the criteria needed to be met by laboratories desiring to increase flexibility. The difficulties encountered during the development of this pilot program suggest some of the issues the water industry will face as performance based system processes are implemented in compliance monitoring. Dr. Eaton noted a lack of adequate reference materials as a bottleneck to progress. The Board's efforts are focused presently on nutrient parameters. Check progress and information at
http://srvdwimdn.er.usgs.gov/pmethods/.
Tuesday, July 20, PM--"Environmental Business in the PBMS Paradigm"
Chair: Tony Pagliaro (ACIL)
The first part of this session offered different contracting approaches for analytical services using PBMS considerations, from the client/data user viewpoint. The second part presented laboratory PBMS implementation issues.
• Cheryl Groenjes (Chemist, U.S. Army Corps of Engineers, Omaha, NE)-"A Framework for Contracting Under PBMS". The USACE has developed and issued a standardized template entitled, "The USACE Shell for Analytical Chemistry Requirements for USACE Projects." Project-specific data quality objectives must be established for USACE projects for field and laboratory operations. When no project-specific DQOs exist, the Shell is used. The Shell was not written in response to PBMS, but it fits with PBMS. The Shell establishes the basic approach when performance-based methods, especially the SW-846 methods, are used. It lays out a technical approach to PBMS and establishes default performance objectives; allows flexibility for project DQOs, or potential method limitations; and bases method performance evaluation on batch QC results.
It is the lab's and the data user's responsibility to prove the measurement system works, for the analytes of concern, in a matrix of concern, at the levels of concern. Labs must establish method capabilities of selectivity, sensitivity, range of detection, precision, bias. Data users must ensure the lab applies PBMS within the project matrix to demonstrate capability at acceptable levels of performance.
The Shell has prescriptive requirements: target analytes are identified; preparation method defaults are given, lab quality system requirements are specified; corrective action scenarios, (with decision logic) must be presented, flagging conventions, data review requirements, and notification/documentation requirements are stated, MQOs are established for calibration, batch and matrix QC elements for SW-846 Methods 6010, 7000, 8260, 8270, 8021, 8081, 8082, and 8330. Method performance/control is based on batch QC samples. The 'Shell' describes the requirements for instrument calibration, verification, and continuing calibration, while maintaining a level of flexibility, which may be exercised based on analyst judgment. Each preparation batch is to contain a method blank and a laboratory control sample containing all of the project-specific analytes of concern spiked at the levels of concern to monitor laboratory performance. Each preparation batch would typically contain additional QC samples to monitor the effect of the matrix on the method. Corrective actions are carefully detailed and involve interaction with project managers to avoid the generation of a significant amount of flagged or unusable data.
The Shell also has allowances for flexibility: project-specific DQOs can override Shell requirements, sporadic marginal failures and poorly performing compounds are understood and tolerated. A key performance criterion is the analyst proficiency test. But the Shell's default approach may limit the use of technical people. More guidance is needed for field analytical technologies or unique PBMS systems, and refinement of MQOs/QC limits will be useful. This paper is posted at: http://www.usace.army.mil/inet/usace-docs/.
• Dana Tulis (OERR AOC, new Director of Superfund Analytical Services)- "New Directions in Superfund Analytical Services". Dana presented the current status of the Contract Laboratory Program (CLP) for Superfund analytical support. The CLP has been providing centralized, standardized analytical services to the EPA regions through fixed-price contracts for almost 20 years. Under new leadership, the Analytical Operations Center (AOC) has broadened its mission and customer base by: Providing the appropriate analytical services to meet Superfund's needs of state and inter-Agency clients; becoming customer service driven; embracing continuous improvement; focusing on QA for all analytical needs, and establishing electronic data assessment/ information tools. As an example, the AOC provides a Data Assessment Tool (DAT) for automated data evaluation. Regional customers receive customized electronic files via E-mail within 24 to 48 hours of receipt of data from the laboratory. Regional data validation can be faster without interfacing with EPA's mainframe computer, and manual data entry or re-keying of data is eliminated. DAT is available to all CLP users (e.g., Regions, States, Brownfields). A FASTAC workgroup will investigate expanding DAT (or similar concepts) to non-CLP data. The flexible aspects of PBMS fit in well with the broadened customer-service orientation of the CLP. Dana described the upcoming inorganic analytical contract Statement of Work (expected in Summer 2000) that will allow more flexibility with analyte selection and media, using PBMS for classical wet chemistry parameters. The new SOW will require faster turnaround times and lower detection limits, and specify electronic deliverables for metal analysis and data tape audits.
• Robert Harris and Jim Ealy (Westinghouse/Savannah River)-"Model Agreement for Contracting Laboratory Services". A Department of Energy (DOE) Program was described that modeled corporate procurements, using an Integrated Contractor Purchasing Team (ICPT) from all nine DOE sites, ACIL, and industry labs to establish performance-based contracts under a national services agreement. The team followed commercial procedures to set strategies for continuous improvement, with the goal to become more cost-effective by: partnering with industry. They followed national standards; were non-prescriptive; encouraged reciprocity of QA audits; required a single General Electronic Data Deliverable; eliminated contradictory technical and administrative requirements; and established the basis for PBMS implementation. Analytical services are now available on a GSA pricing schedule-lab providers get on the schedule, and government buyers can purchase from them directly, without issuing their own RFPs. These are indefinite quantity, fixed unit price agreements, not quite contracts, but "agreements to agree." More than 20 agreements have been made so far. DOE will identify site-specific analytical needs, select a lab from the pool, negotiate prices, issue a project-specific contract, send samples, and monitor performance. The mechanism provides opportunities to develop PBMS for all sites. Limited analytical services are available at this point.
The labs are evaluated using Performance Indicator Factors (PIFs) covering the most recent 90-day period (one year for performance evaluation [PE] samples). PIF weights the holding times, calibration and PE sample results and turnaround times using a formula: PIF = 0.6 TAT + 0.2 HT + 0.2 CCV + 0.1 PE. Each lab's PIF is on a DOE website, where all labs can view their own results and get details, even challenge the scoring. During the process of lab selection, a DOE buyer considers the PIF of the labs, and applies it to the labs' quoted "spot prices" and sample shipping costs to calculate a "Best Value Added." In Oak Ridge, the performance measures since May, 1998 improved by 10%. Customers are delighted. Labs like it because they are rewarded and recognized for excellent performance and quality. Future benefits will include reduced technical time spent; reduced travel for contractor audits. Labs will save costs due to: reduced audits (reciprocity is accepted), using one EDD, and standardizing methods. Incumbents will not have to submit extensive proposals. Field sites are able to negotiate for improved pricing.
• Robert Pullano (General Engineering Labs)-"Managing a Lab Under PBMS". The goal of PBMS is to produce data that is defensible, scientifically sound, and cost effective, that meets the DQOs. To make PBMS work, labs will need people who know the customer's project objectives, that understand production lab operations, and are able to present methods to lab operations, project teams, and regulators. PBMS requires more of almost everything, but the rewards are greater, too. He listed four lab elements and their sub-elements, contrasting them and showing how they change from prescriptive to PBMS scenarios:
Lab Facilities-dedicated work areas, instrumentation, and computers for development
Instrumentation and Computers-newer, faster, more flexible. LIMS will need the ability to disaggregate costs, track schedules for unique projects
Personnel-highly motivated senior level scientists with knowledge of methods, projects, and sites; will train, mentor, and perform R&D. Contracts/Sales force with technical background and good communication skills. Human Resources staff will need more time to find qualified people. Marketing must sell technical talents, not just price and performance.
Infrastructure and Administration-People need a project management and technical background, with product/project/site knowledge and good communication skills. Accounting will be more project-specific. Quality Systems will have to manage documentation and records, training, and demonstrated capabilities.
• Harry Gearhart (DuPont Engineering)-"Establishing Laboratory Performance Expectations". PBMS guidelines are needed, with specifications for determination of DQOs and MQOs, and Agency guidance for method verification and modification. EPA must determine what constitutes an acceptable demonstration of compliance. The regulated entity is responsible for demonstration of compliance with DQOs and MQOs. It selects methods to meet application-specific requirements. PBMS will be client-specific, possibly site-specific. New approaches will be needed for third-party data review, and for marketing a lab's creativity, flexibility and technical depth rather than its ability to perform specific methods. There will be a transition period when clients will request labs continue to use EPA-approved methods. When all stakeholders have a common understanding of PBMS, they will more readily accept it.
• Gary Fallick (Waters Corp.)-"Implementing New Technology-One Instrument Vendor's View". Mr. Fallick presented a "validation time-line" for new instrumentation that considers actions that should be taken before offering them for purchase. The manufacturers must properly validate their products and provide thorough documentation that they can produce legally defensible data. The purchasing lab must similarly demonstrate that it can use the product effectively, by validating methods that are developed, and documenting performance.
• Larry Jackson (Environmental Quality Management)-"The Role of Consensus Standards". PBMS is new to EPA, but is an old concept (it is essentially the Scientific Method). All parties must work together under PBMS. Technical defensibility, not legal defensibility, is crucial. Lab liability is much greater under PBMS than with prescriptive methods. The liability can be controlled using communication and documentation. Federal Register 2/19/98, Vol. 63, No. 33, describes the use of consensus standards. Once designated as a consensus standard, it can be used to determine compliance, unless the method specifically conflicts or is inconsistent with a regulation. ASTM methods are a good starting place for PBMS, with some very usable methods. They have legal standing and are technically defensible. There are no QC acceptance criteria embedded in ASTM methods-this is the user's responsibility. The site http://www.astm.org lists all the ASTM methods available.
Wednesday, July 21 AM--"PBMS Implementation"
Chair: Joan Fisk (U.S. EPA OERR)
This two-part session was divided between "Ensuring Scientific and Legal Defensibility" of environmental data collected under a PBMS scenario, and the actual technical issues of "Field and Laboratory Implementation" to be considered in carrying out PBMS data collection.
David Friedman (U.S. EPA Office of Research and Development)-"The EPA Approach". David said there is no consistent, defined regulatory structure for PBMS in terms of standards, accreditation, or assessment. David asked, "How do we improve quality?" "What is the minimum amount of data needed to be scientifically and legally defensible?" "How should regulations be constructed to demonstrate compliance?" One way to determine measurement system applicability is to "try it and see if it fits," using method verification to show "it works" and method validation to show that "it works for a specific application." This will help show if the method is free of unacceptable bias, sensitive, and repeatable. PBMS allows us to take advantage of technology changes, but we have to make sure that a method is not laboratory-specific. We must demonstrate that the method is appropriate, is working properly, and that the lab is doing things correctly. Regulations should be written so that compliance can be demonstrated. This may change from specifying a "flat level" concentration to a "% confidence" level. Does the method work for specific applications? Is the method free of unacceptable bias, sensitive, and repeatable? Bias can be measured/controlled using a variety of techniques: analyze standard reference materials (SRMs); compare two different analytical techniques; conduct spiking studies; compare results to data produced by an independent standards organization; or subject the results to peer reviews and independent method verification. Precision also can be demonstrated by analyzing spiked or unspiked replicate samples.
• Larry Gonzalez (U.S. EPA)-"PBMS and the Comparable Fuels Rule: A Model for Changing the Regulatory Process". Larry provided an example of a program that addresses the questions raised by David. EPA's Comparable Fuels Rule has been changed to reflect a PBMS approach, allowing Alternate Test Procedures (ATPs) that meet 40 CFR Part 261.38 (c)(8) if the regulated entity can:
• demonstrate there are no constituents at specification levels;
• ensure that sampling and analysis is "precise, unbiased, and representative of the waste," and;
• show constituents are below the specification level, at the upper 95% confidence level around the mean.
• Deborah Loring (Corporate QA Manager, Severn-Trent Labs)-"QA Perspective on EPA's Approach to PBMS". She asked, "If PBMS works, what will the future look like?" "What is a 'modification' of an existing test method, and when is it 'PBMS'?" "Is a complete validation necessary for a method verification?" She is concerned that "compliant data could be scientifically invalid." She proposed a regulatory structure for oversight and assessment that would foster legally defensible and scientifically credible data and a straightforward accreditation and assessment of a lab's ability to perform under PBMS. To implement PBMS, three key elements are necessary for regulatory and quality assurance (QA) oversight:
1. General standards should exist for PBMS.
2. Method validation guidelines should be incorporated as part of those standards. An example that could be used is the 1994 U.S. Food and Drug Administration (FDA) Elements of Method Validation.
3. Accreditation should exist for PBMS, and "field of testing" should be included in an accreditation scheme.
The ability to perform PBMS would be reviewed against the general standards with PBMS and method validation checklists for the on-site assessment of labs. PBMS method summaries and evidence of method validation would be reviewed. Selected PBMS methods would be fully assessed. A general PBMS accreditation would be granted. Then the lab must submit updates for each new PBMS method brought on-line. Follow-up assessments would include reviews of general accreditation and of selected new PBMS methods.
• Richard Burrows (Quanterra)-"Technical Perspective on EPA Approach". He noted that PBMS represents a major change for environmental laboratories. The days of using a prescriptive method, whether it worked for the particular application or not, are drawing to a close. The replacement for a prescriptive method should be a process where experienced chemists can use any method, providing that there is proof that it works for the particular application. EPA can assist the industry by providing these protocols-they need to be sufficient to the task, but not so arduous that they represent an insurmountable financial barrier to the use of PBMS. Richard stressed the operative principles for PBMS to produce legally and scientifically defensible data: "Validate the method for the analytes of concern, in the matrix of concern, at the levels of concern." The lab must use real or simulated reference materials for validation, and many more such materials must be developed and made available. Use an EPA method performance as a current baseline. A consistent validation protocol to be applied to all methods is needed.
A common question from regulators is "How can we keep the labs honest?" One answer is to have a strong accreditation program, such as NELAC offers. Another is to use double-blind evaluation samples, at the levels of concern in the matrix of concern. This is an empirical way to know how well the lab is performing. But he listed several PBMS 'traps,' including unnecessarily long analyte lists, unreasonable MQOs, and believing it will be easy to get better quality, cheaply. How can labs be productive under PBMS? For example, GC/MS semivolatiles analyses may include up to 200 analytes--how good is the lab's performance for each analyte? There are many extraction and analysis method variables to consider. Richard suggests optimizing the preparatory and analytical conditions for the approximately 30 analytes commonly found, while demonstrating that the others aren't present in the samples. Labs and regulated entities will have to take more responsibility for data quality. They need more reference methods, and different QC samples and criteria for common versus. uncommon analytes. Fortunately, much work is underway to provide the basis for implementation of PBMS:
• NELAC will provide a consistent basis for laboratory accreditation;
• An ELAB workgroup has provided a summary of requirements for effective PBMS;
• An EPA/industry workgroup is developing a validation guide;
• A Department of Defense initiative will provide a good basis for understanding the real performance of current prescriptive methods; and
• A wider variety of reference materials is becoming available.
The legal defensibility of PBMS was discussed by Alan Horowitz and Rick Colbert.
• Alan Horowitz, (Zenica)-"Private Sector Perspective". Enforcement predictability is desired. Industry needs constant compliance targets. There should be an ability to demonstrate compliance, with confidence in the methods and their results. Key principles include
"PBMS equals compliance"
Criteria are derived from existing methods and standards
There must be reliability and predictability of evidence
To explain the concept of "credible evidence," he used an analogy of automobile speed indicators that allow the use of different detection devices to detect speeders. PBMS would define a reliable speedometer, and let others provide them. From the Comparable Fuels Rule, "in enforcement action, the burden of proof is on the one claiming exclusion." So, what defines 'compliance'?
• Identifiable and certain compliance targets
• Method and PBMS criteria and standards/limits will yield compliance (e.g., the Comparable Fuels Rule)
• From ELAB, "Any regulated entity meeting the PBMS requirements and whose lab results demonstrate compliance, should be judged compliant." EPA and citizens' suits should not challenge compliance using an alternative PBMS method. But while method reliability is subject to challenge ("burden of proof," and "rebuttable presumption" concepts), PBMS should not be used to change underlying performance standards.
• Rick Colbert (EPA OECA)-"Enforcement Perspective". PBMS methods must meet the test of legal defensibility. This is currently the "Daubert Rule." The Supreme Court in the case of Daubert v. Merrill Dow Pharmaceuticals ruled that judges must act as gatekeepers to ensure the admissability of relevant and reliable scientific evidence. Courts must ensure that the reasoning or methodology underlying scientific testimony is scientifically valid and whether it can be properly applied to the facts at issue. All scientific evidence must meet this test. The Supreme Court gave several factors that courts are to use in determining scientific validity:
• Is the science/technique capable of testing/has it been tested?
• Has it been peer reviewed (e.g., publication in a peer-reviewed journal)?
• Is the error rate known?
• Are there standards for operation of a technique?
• Is the science or technique accepted by the scientific community?
PBMS methods will need to pass the Daubert test. If EPA is not going to approve PBMS methods, peer review becomes a big issue. The courts will likely be looking to see whether the individual methods have been peer reviewed or, perhaps, that there is some validation protocol for PBMS methods that has itself been peer- reviewed.
"PBMS Implementation"
Chair: Joan Fisk (U.S. EPA OERR)
The second portion of the two-part session addressed the actual technical issues of "Field and Laboratory Implementation" to be considered in carrying out PBMS data collection.
• Duane Geuder (USEPA-OERR)-"Overview of Field and Laboratory Implementation Issues" Mr. Geuder reminded the audience that PBMS includes sampling, and all the attendant issues that are more easily attacked in the laboratory are still outstanding in the field. Designing a sampling plan is a critical part of project planning, and these plans must clearly state how background samples are to be collected and used. He emphasized the importance of proper sample handling procedures, since data are more often challenged on chain-of-custody rather than lab performance issues. He presented the following reminders:
• P stands for Performance and Planning
• B represents Background, Boundaries, Boondocks
• M means Measurement
• S is for Semper qualitatis (quality forever), and "Sell the concept of PBMS"!
Two of the presentations (Bertoni and Bryan) covered planning for data collection for PBMS. The first related to the actual development of the QAPP and how to develop project-specific measurement quality objectives, and the second and complementary one, discussing error sources and the importance of balancing and controlling them.
• Malcolm Bertoni (RTI)-"Developing Project Specific MQOs from DQOs".
The goals for this presentation were to show how systematic planning can help implement PBMS, and to raise some issues about specifying measurement performance criteria, balancing error sources and managing uncertainty. Quality begins by understanding the user's needs, continues through a systematic planning process (like the DQO process) that specifies performance criteria for data collection, then assesses the data against the user's needs. The DQO process is a proven method for systematic planning that supports defensible decision-making. DQOs supply decision performance criteria, linked to the end user's needs. The Data Quality Objective (DQO) process was explained and how it leads to decision performance criteria for both sample design and QA/QC design. MQOs have measurement performance criteria, linked to the technical user's needs. The MQOs flow from the QA/QC design and become part of the QAPP. The important MQOs (or indicators of data quality) are selectivity, sensitivity, detection limit, precision, and bias. Sampling design quality indicators are representativeness, completeness, and comparability. The relationship between DQOs and MQOs is dependent on the specific project (i.e., where tradeoffs can occur to meet DQOs). The optimal level of measurement precision can be determined by looking at sources of variability, and their effects on performance and cost. The DQO Process generates information that drives the selection of key measurement performance criteria:
• Sampling Design-Sampling Plan, specifications
• QA/QC Design-Measurement Performance Criteria (Selectivity, Sensitivity, Detection Limit, Precision, and Bias), MQOs, QA/QC Protocols, etc.
These outputs are combined into the QA Project Plan. Measurement performance criteria and the sampling design yield a statement of representativeness, completeness, and comparability.
DQO decision error rates relate to total variability. It may be difficult to estimate or measure subcomponents of variability. Usually the most effective strategy is to reduce the largest component of variance (if possible). Generally speaking:
Total Variability = Laboratory Variability + Field Variability
The move to PBMS is driving a closer look at systematic planning (more people will be developing DQOs). PBMS is a natural fit with DQOs. Translating DQOs into measurement quality objectives requires multidisciplinary technical skills. More work needs to be done to develop case studies and practical examples.
• Rex Bryan (DynCorp)-"Balancing Error Sources for Project Planning". Rex used cartoon characters and a Wild West story line to continue the discussion of sources and treatment of variability. He noted that PBMS has given us the opportunity to re-emphasize the importance of total uncertainty. Undue emphasis on the analytical laboratory/ analytical methods has been the modus operandi in the past - with the serious consequence of ignoring the far larger source of error-sampling/field variance. Various sources of uncertainty were discussed as well as types of uncertainty or error. When a "total project variance" approach is taken, specialists in analytical procedures can help determine if the greater expense from additional analyses is cost effective, given the likely number of samples proposed. Instead of promoting only the BEST method at all times the laboratory can act as a consultant in proposing lower quality, but still acceptable, methods at lower cost. This is PBMS at its best.
Applying the statistical technique of ANOVA (analysis of variance), management should allocate more resources towards the highest variability. You might be willing to have many lesser quality samples to know where the contaminated soils are, rather than knowing the precise levels from a few high-quality samples at a few locations. But how will we know if the lesser quality samples are good enough to mean anything?
Plotting the variance of errors on the y-axis, and error separation distance on the x-axis, produces a graph called a variogram. Variograms are Practical Representations of Field Variability. Spatial considerations emphasize how many samples should be taken, and their location. This graph can be used to calibrate the location and number of samples. Laboratory variability is also incorporated in a variogram. A variogram can only be obtained from an initial sampling. When a sample is estimated to be below a threshold and it truly is not, a false-negative
(F-) has occurred, and the site may not be remediated. The cost of this error, which may include illness, cancer, death, etc. is difficult to calculate. When a sample is estimated to be above a threshold and it truly is not, a false-positive (F+) has occurred. Unnecessary remediation may occur at a large expense. By varying the required threshold of concern, a trade off of F(-) to F(+) can be done. With no further sampling, a F(-) rate can be fixed, while the F(+) area may increase to a very costly size.
A major implication of spatial sampling is that not all samples are alike. Different sampling patterns may be more efficient in reducing both F(+) and F(-). An optimum sampling strategy would be to trade-off costs of sampling with remediation. The optimum number of samples is when one addition dollar of sampling reduces one dollar of unnecessary remediation. Spatial considerations can have a great impact on limiting expensive decision errors-thus showing the need to focus more on field/sampling uncertainty.
• Mike Shepherd (Radian)-"Moving MQOs into Laboratory Operations".
Radian has been using a PBMS system for years as a consequence of the widely varying requirements found in the industry. A key to the approach is a system of companion documents: standard operating procedures (SOPs) and protocol specifications. SOPs are used to establish a controlled, approved process, or "HOW" work is performed. The Protocol Specification describes the performance characteristics, or "HOW GOOD." The protocol specification is useful in project planning to document standard laboratory capabilities, communicate measurement objectives to the laboratory staff in a context that they can use to perform the work, and bridge the gap between laboratory capabilities and overall project data quality objectives. While the mechanics of the analytical procedure (How to?) remained the same, the performance specifications (How good?) could vary.
Protocol specifications define the analytical needs and MQOs for the project into terms meaningful to the lab. They comprise:
• Technical specifications -represent those most readily measured and compared against defined performance criteria to measure the quality of the result
• Procedural specifications-describe those things that must be done as part of the analytical process, and includes requirements for calibration and quality control checks, equations to be used to calculate results, and other procedure-oriented specifications.
• Operational specifications-describe systems and conditions under which the analyses must be performed, and includes operational definitions of terminology, training requirements, health and safety requirements, and other system-oriented specifications.
These specifications are issued as controlled documents on a lab Intranet, with default or minimum standards. Lab MQOs can be linked to project DQOs, to better communicate lab performance capabilities at the start of projects. This approach allows flexibility and continuous improvement.
Wednesday, July 21 PM--"Laboratory Auditing and Accreditation"
Chair: Rock Vitale, (Technical Director of Chemistry of Environmental Standards, Inc.)
• Gary Ward of ITS presented a lively and controversial paper, "The Role of Auditing and Accreditation." He described an extensive examination and evaluation of a lab's data set that included results alleged to have been "altered" and reported to clients.
• Ann Rosecrance (Core Laboratories). Her presentation "The Role of a Compliance Program and Data Quality Review Procedures under PBMS," showed two innovative ways to assist in PBMS adoption and acceptance: 1) implementation of a compliance program to ensure ethical work performance; and 2) expansion of traditional data review procedures to include closer evaluation of data quality by analysts, supervisors and data reviewers. Key elements to include in a compliance program are an ethics policy, management support and reinforcement, ethics related procedures, a zero-tolerance policy, ethics assistance and reporting, program management with a compliance officer, and compliance training and audits. An ethics program should focus on compliance, and prevention. The lab should conduct both the traditional quality audits and data quality reviews. The QA program focuses on group issues, while the compliance program focuses on individual ethics issues. Unacceptable activities and practices should be defined. Immediate action is necessary if ethics violations occur. Procedures should be established for handling non-compliant data and out-of-control events. In looking at the data, check the MDLs, initial calibrations, proficiency studies, LCS, method blanks, MS/duplicates, and sample data. The combination of an effective compliance program and data quality review procedures will help ensure ethical work performance and acceptable data quality under PBMS.
The following papers discussed the "paradigm shift" for auditors under PBMS. They showed how auditors conduct audits, and how much the audits will cost due to changes in the scope of the assessments. Auditing procedures and skills will also need to evolve for PBMS.
• Sylvia Labie (Quality Assurance Section, Florida Department of Environmental Protection). In her talk, "Auditing - What The Future Holds," Ms. Labie indicated that Florida is approaching the "PBMS future" with both anticipation and apprehension. On the one hand, the National Environmental Laboratory Accreditation Program (NELAP) should decrease the confusion laboratories have over multiple program and agency requirements by consolidating those requirements and the audits by various agency programs. On the other hand, there are many concerns over auditing under a performance-based measurement system.
Many of the concerns center on the difference between auditing "by the book" and auditing against a set of performance standards. The biggest challenge will be to overcome the traditional mindset of auditing by the method checklist. Auditing under PBMS represents a paradigm shift, with distinct responsibilities that require knowledge of the mechanics as well as the theory of analytical methods. Trained and experienced auditors will have to conduct lab, field, and data audits. In the on-site audits, the auditor is verifying equipment and personnel, as well as PE results to determine: "Does the lab do it by the book?" "Is the method consistently meeting permit requirements?" They must check data quality, QC measures and documentation. The auditors must not only be well-versed in NELAC standards, but in the environmental programs for which the data are to be used. They must have an understanding of the program needs and a sound technical knowledge of the methods to be audited. Above all, the auditors and the data users must understand that laboratory audit does not produce or guarantee useable data. The data must be clearly related to the program needs and data quality objectives. The bottom line is quality, and quality data means that there is a better likelihood that environmental decisions based on such data are sound and in the best interests of public health and natural resource management. (Ms. Labie's colleague from the State of Florida's DEP, Dr. Carl Kircher, presented a detailed description of the PBMS challenges facing states in a feature article in the November/December, 1999 issue of Environmental Testing & Analysis).
• Lynn Bradley (Director of Environmental Health, Association of Public Health Laboratories) presented "What Government Laboratories Expect from Accreditation Programs." The National Cooperation for Laboratory Accreditation (NACLA) was established to accredit the accrediting bodies. In deciding whether to seek voluntary accreditation, labs have to weigh the cost against the market value of being accredited. This equation may balance differently for private and public laboratories, since public labs have less profit motive. Government labs have little profit motive, and low-bidder contracts offer no quality incentives. In June 1999, the Association of Food and Drug Officials endorsed a vision for lab accreditation, using Draft ISO Standard 17025 as the foundation for all laboratory accreditation, with a single audit to those standards, supplemented by field-of-testing standards. She emphasized that NELAC needs strong federal support to sustain the exceptional inter-state cooperation that it has enjoyed thus far, and to reach full implementation, it will need additional resources in fiscal year 2000 and beyond. NELAC has been chronically under-funded for its mission, and will need multiple databases for storing information on labs, Proficiency Test providers, accrediting authorities, etc. While laboratory certification is desirable, proficiency credentials for lab staff may be more meaningful.
• Roxanne Robinson (Vice President of American Association for Laboratory Accreditation, A2LA) presented "Challenges in Assessing Performance Based Methods." Roxanne described the auditor training program content used by A2LA as a third-party accreditation body, and how it is changing for PBMS. The format for the final product, the Scope of Accreditation, will have to accurately reflect the laboratory's PBMS capabilities. She also pointed out additional costs for such audits, compared to traditional audits. Assessor preparation time may increase to determine the laboratories' levels of PBMS and the appropriateness of their use. Evaluating the training and qualifications of the decision-makers, validators and technicians could require closer scrutiny. The contract review process becomes more critical when the lab is determining the needs of the client. PBMS increases the need to assess initial method validation processes, to determine if DQOs were met. Were they tied to regulated methods? Were written protocols used and available? Assessment cost implications will depend upon the extent of modifications. There may be additional prep time to become familiar with PB methods. Additional time during the on-site assessment may be needed to ascertain the competency of staff to validate and perform the PB methods. Assessment team size may be increased to ensure appropriate technical expertise is available. Accreditation bodies will have to ensure that technically competent assessors are available. A2LA needs to recruit more technical assessors with subject expertise in technical areas. Reference methods defining data quality objectives and performance criteria, laboratory cooperation in providing assessors with the necessary information, and means for assessors to exchange information will all help in the efforts to assess PBMS.
The session concluded with a paper presented by Paul Mills (DynCorp), "Documentation Requirements for Laboratories." The interrelated and complementary nature of the National Environmental Laboratory Accreditation Conference (NELAC), the International Standards Organization (ISO) Guide 25 and Draft Standard 17025, and PBMS were discussed. NELAC accreditation of a laboratory provides credibility; while the ISO guide/draft standard provides a quality system framework that can incorporate PBMS with the lab. PBMS provides method flexibility for specific purposes, but the laboratory performing the tests must provide evidence that the desired quality is met. Method validation activities should be documented by the method developer. The Environmental Laboratory Advisory Board (ELAB) Checklist Workgroup recommended to NELAC and EPA that analytical methods be written in laboratory SOPs (in EMMC format). Analyst competence should be documented in an initial demonstration of performance by the method user. The lab should document the results from quality control samples to continue to show acceptable on-going performance. All the other quality system elements described in ISO Guide 25 and elaborated upon in ISO Draft Standard 17025 must also be present. Mr. Mills stressed that legally acceptable documentation is accountable, traceable, secure, retrievable and sufficiently complete to allow reconstruction of the work. It is identified to permit reference to the items or activities that produced the documents. It is permanently and legibly signed and dated by those responsible for the work.
|
