< Back

The PBMS concept opens wide the door for changes as labs, vendors, and data users seek improvements in environmental analysis. Because measurement systems can be complex, changes may affect sensitivity, ruggedness, specificity, throughput, comparability, compatibility, and even competitiveness across instruments, labs, and programs. This article is part of a PBMS series in Environmental Testing &Analysis that will present implementation issues and how they are being addressed.

There are at least three parties involved with an audit--the auditor, the audited organization, and the client requesting the audit who receives the report. Each has certain interests in the audit, and a stake in the outcome. What auditing issues will be faced in evaluating a laboratory operating under a Performance Based Measurement System? EPA requested and received public input after the intent to implement PBMS was announced. Interested parties and industry experts have listed several issues of concern to these parties, in surveys and in presentations at the USEPA Waste Testing and QA Symposium in July (reported in the previous issue of ET&A). This article presents some of these issues, and describes how they are being addressed by NELAC and others in the industry.

From the Federal Register announcement of EPA's intent to implement PBMS, October 6, 1997, Volume 62, Number 193, "Commenters (on PBMS) may wish to address: The potential effect of PBMS on compliance monitoring and enforcement of regulatory and statutory requirements. For example: …The level of expertise necessary for EPA and state inspectors to successfully determine the adequacy of a PBMS method; the resource and training implications of PBMS, especially for state environmental programs; the advantages and disadvantages of using method performance criteria and documentation requirements for establishing that methods achieve required performance levels."

Dr. Llewellyn R. Williams, USEPA-Las Vegas, provided a summary of the collected public comments in response to the Federal Register announcement. These were presented to EPA Regional offices and at stakeholder meetings, from March through June, 1998. The issues listed by states, labs, and industry respondents that are pertinent to audits can be summarized as follow:

Implementation should be coordinated with NELAC to reduce variability in state interpretations, achieve reciprocity among states, and establish a consistent approach across EPA programs. Training is needed for inspectors, and audit checklists need improvement. It will be more difficult and time consuming to audit PBMS. Laboratories are especially concerned about disclosure of proprietary information during audits

How are these concerns turned into specific questions for the three parties, and how are they being answered? What can we expect? These and other issues are discussed below, with a description of responses.

Auditor Issues:

(1) Auditor Qualifications and Training: What expertise is required to review the methods/modifications, and to assess a lab's performance? Comments from industry experts included "The days of non-chemists auditing labs are over. The audit community already has a problem with too many rogue auditors out to find problems." Others see the emerging National Laboratory Accreditation Conference/Program (NELAC/NELAP) as being the logical vehicle for the needed laboratory inspections.  

Survey respondents summarized auditor qualifications in words consistent with NELAC July 2, 1998 Revision 10, Section 3.2.1, "On-Site Assessment Personnel, Basic Qualifications," including: "a) Be familiar with the relevant legal regulations, accreditation procedures, and accreditation requirements; b) Have a thorough knowledge of the relevant assessment methods and assessment documents; c) Be thoroughly familiar with the various forms of records described in Section 3.5.3, Records Review; d) Be thoroughly cognizant of data reporting, analysis, and reduction techniques and procedures; e) Be technically knowledgeable and conversant with the specific tests or types of tests for which the accreditation is sought and, where relevant, with the associated sampling and preservation procedures; and f) Be able to communicate effectively, both orally and in writing."

"Whether the States that will be implementing NELAC will have the needed expertise to do a credible job is still a nagging question." Shortages of qualified assessors are expected, as well as shortfalls in funding training. From the July 1, 1998 meeting minutes of the Environmental Laboratory Advisory Board, a third-party assessor workgroup was approved to evaluate minimum credentials for third-party assessors, both individuals and organizations.

Training should also reduce variability between assessors and prevent inconsistent audits. From NELAC, Section 3, Revision 10, July 2, 1998, Section 3.2.1, On-Site Assessment Personnel Basic Qualifications: The assessors must meet the 5-day initial assessor training course requirements within 2 years of the NELAP's recognition of the first Accrediting Authority. Annual refresher training would be required. Four years would be allowed for meeting the technical course requirements. The technical training requirement is to ensure the technical competence of each assessor and to assure the reciprocity of each audit, they are to train the assessors how to review data and audit each subject. Seven technical training courses are envisioned, either 2.5 days (for microbiology, biological, inorganic nonmetals/miscellaneous, inorganic metals, asbestos, and radiochemistry) or 5 days (organics, including GC, GC/MS, and HPLC). The NELAP Laboratory On-Site Assessment Training Manual is available as a guidance document, on NELAC's Bulletin Board on the Website.

(2) Checklists and Criteria: What should the audit team examine at the lab? What records are needed in audit planning stages to become familiar with the method(s)/modifications(s)? What criteria should be applied?

From Analytical Chemistry News and Features, 12/1/96, "EPA Plans New Approach to Analytical Methods": "Some states worry that by abandoning prescriptive methods with their convenient checklists, the auditor’s job will get even more difficult. That problem would get even worse under the present NELAC plan, which gives states a significant role in accrediting laboratories for all EPA programs….It is unreasonable to expect that any inspector would be well versed in methods ranging from wet chemistries to ICP-MS…."

Industry experts commented: "Audits should entail more review of raw data and electronic data along, with the quality systems review. Verification of continued acceptable performance is a must, including results of proficiency test samples for each method and analyst. Look at training records. Check calibration and QC data. Project or program-specific DQOs should be the driver, plus EPA-specified categories of information (MDLs, accuracy, precision, etc.). Examine PE/PT samples, double-blind and real matrixes if possible. Examine data management systems, to follow the trail of method changes. Many labs are not currently able to manage the data for methods, standards, training, etc. that will be required to prove performance." "Perhaps the only viable way (to audit the lab) will be to take the approach of the ISO 9000 audits where the SYSTEM is audited and certified. Then an appropriate list of fields would need to be developed to evaluate completeness and adequacy for the PBMS SYSTEM at a laboratory."

A key question is how audit checklists can be developed. The NELAC On-Site Assessment Committee has taken a method-specific approach to the development of NELAC parameter checklists, based on current EPA programs. EPA has begun movement toward a performance-based measurement sytem, but there still exist some programs that specify certification by method. Section 3.6.3 of NELAC says "Standardized checklists, as documented in the NELAP Assessor Training Manual, must be used for the on-site assessment. The use of checklists does not replace the need for assessor observations and staff interviews, but is another tool which assists in conducting a thorough and efficient assessment. A checklist is not a substitute for assessor training and experience."

Appendix E of NELAC Quality Systems, Revision 9, July 2, 1998, presents checklists to be used in initial and continuing demonstrations of performance. These checklists are completed by the lab for each matrix within each medium for which performance is demonstrated. These remain on file at the lab, and must be submitted to regulatory authorities upon request. These are not the same checklists that assessors will use. At the same meeting, the ELAB Laboratory Assessment workgroup recommendations on checklists was forwarded to the NELAC On-Site Assessment Committee for consideration A special ad hoc committee under ELAB developed recommendations about checklists that were reported to NELAC in June. (SEE ELAB WORKGROUP SIDEBAR).

(3) How often should the lab be audited? Responses were consistent with the policy expressed in NELAC, 3.3.1, Frequency of On-Site Assessments: "Accrediting authorities must require a comprehensive on-site assessment of each facility that is accredited at least every two years. Assessments may be conducted more frequently for cause, at the option of the accrediting authority."

(4) What records do auditors need in planning the audit to become familiar with the method(s)/modifications(s)? "Lots of data are desired up front. This should be reviewed by a team of chemists evaluating performance data before and during visits."

Auditee Issues:

(1) Disclosure of confidential information to assessors: NELAC Section 3.2.2, Assessor Qualification, states: "Before an assessor can conduct on-site assessments, the individual must be qualified by an accrediting authority. Each assessor must sign a statement before conducting an assessment certifying that no conflict of interest exists and provide any supporting information as required by the accrediting authority."

(2) How much documentation will the lab have to provide the assessment team before their on-site visit? What will be necessary to show EPA-established performance criteria are met? "The lab should be able to show that it is following its own SOP and meeting method-specific criteria. As part of an initial certification process, NELAC will require the lab to provide records of personnel qualifications, proficiency test results, and quality systems, and undergo an on-site assessment. Chapters 2, 3, and 5 of the NELAC standards discuss these requirements." These records should be made available to the assessment team, either by the accrediting authority or by the lab. 

(3) How long will an assessment take? How many staff will have to act as on-site lab guides and technical contacts to demonstrate methods/modifications, produce additional records, and answer further questions? Experts estimated that "Depending upon its size, clientele, programs, methods, and instrumentation, the lab can expect to host a team of auditors (3-6 people) for at least one week."

Regulator/Client Issues: 

(1) If the auditing organization doesn’t have the internal resources (numbers of qualified and available assessors) to review the data or conduct on-site visits, how will states evaluate labs? Will an independent 3rd-party auditor become the default model, approved/certified by ASTM, ASQ, A2LA, NELAC, etc.? 

From "EPA Shifts into Performance Gear" by Andrea Kinney and Birgit Caliandro, in Today's Chemist At Work, April, 1998: "Implementation of PBMS means that DQOs will need to be established for all monitoring activities under RCRA (and under regulated programs)…EPA will have to establish and provide training, because the technical expertise required by state and federal parties to review DQOs and their method counterparts, method quality objectives, for each method may not exist at the regional and local levels."

Opinions differ: Some say experienced chemists can become assessors. Most recommended experienced, trained third party auditors (A2LA, NELAC, etc.). A dissenter said: "Much of the private sector has no confidence or respect for A2LA or current state certifications. I fear that this lack of confidence will not change with one certified under NELAC." Another respondent said: "Government at all levels (Federal, State, local) has little or no capability to efficiently set criteria and make judgements about the myriad of different techniques that will be introduced by competing commercial interests. The complexity of analytical methods for environmental contaminants at very low concentrations is so great and the available talent so limited that it will be very expensive and time-consuming to even attempt this. Third party auditing will likely be attempted and prove a failure, because of the lack of talent and the high cost."

"I think PBMS will push the analytical community and users towards 3rd party certification and audits, because the agencies won’t be able to bear the costs of maintaining current expertise in a wide variety of fields. Labs will have to buy their lab and method approvals from whatever organization the EPA and states can agree upon. On the bright side, it may be that one approval and audit is good for all the various regulatory bodies, and labs won’t have "audits du jour" to schedule and work around. Audit and method performance data should become shared."

From minutes of the NELAC On-Site Assessment Committee Meeting, 6/30/98: : The committee explained that they have received word that NELAC will not have the resources to provide oversight for assessor training programs and it is investigating alternatives. It was also reported that NELAC will be removed from the funding issue, and that funding would be decided "by the relationship between whomever provided oversight, and the training providers. The laboratories will fund their own certifications through their relationships with the training providers."

(2) Assessment Records Maintenance and Access: How will the history of method changes be stored? Will the local, state, and/or federal agencies maintain records of methods/modifications and audits, by single lab and multi-lab users? Will this be accessible to other agencies as well as labs, so the advances of any lab can be shared with all? Or must this information be protected as competitive advantages?

Confidentiality agreements, proprietary information stamps, and non-disclosure agreements, were all cited as essential to protect the laboratories and clients who invest in PBMS improvements. Each lab/client would decide who would receive the information. "The information should only be shared if written permission is specifically granted by the laboratory." "If a client pays a lab to develop something, it is the client's property and not the lab's. Proprietary changes and developments won’t be shared, so progress may be slowed. Labs will have to protect confidentiality, stamp SOPs as confidential or proprietary business information. Labs will require employees to sign confidentiality agreements, too, to prevent job changers from taking the methods with them. This will discourage sharing information on method improvements between labs. Vendors will be the ones to spread changes, by default. Standards-setting bodies like ASTM will assume a "provider of reference methods" role. Vendors may become brokers of such changes and improvements, if labs sign agreements to sell new technology. Big labs with R&D capability will add to their advantage, while little labs won’t keep up overall. Clients will have a more permanent bond to the lab if the lab develops a method tailored to their waste and a method that is more cost effective. Chemists and/analysts may change jobs less often due to signing non-disclosure and non compete agreements."

(3) How can the data user be assured the lab’s results will be comparable, consistent, cost-competitive, reproducible over time as further changes are made, and readily accessible?

"The history of method changes is documented by indicating the SOP in use at the time of analysis. The real issue is being able to identify a substantive change in a method that requires a distinct separation in performance assessments." To link performance with changes over time, a relational database structure may provide the tools for searching correlations between method changes, matrixes, time, instruments, etc.

(4) Will clients hesitate to authorize the use of non-prescriptive methods, such as PBMS? If clear advantages can be offered, plus assurances that data will still meet regulatory approval, the data requesters may feel more secure with these methods.

Other Issues: From "Is PBMS in Your Future?" by Larry Keith, in EnvirofACS, the newsletter of the Environmental Chemistry Division of the American Chemical Society: " First, we're still in the early stages of developing the PBMS approach. There are many identified issues that still need to be resolved and other issues that we may not even be aware of yet. Second, realistic DQOs are needed and many people will need help and training to be able to develop them for their individual situations. Third, state and federal regulators and other "data users" will have to understand some of the basic problems inherent with many environmental analyses (e.g., variability in the data, confidence levels of analyte identifications and concentrations, method selectivity and sensitivity, interferences, false positives and false negatives, etc.). Again, training may be needed to help understand these variables. Fourth, comparative tests of PBMS versus conventional consensus methods are not known to have been made yet. Nevertheless, PBMS is scheduled to begin as accepted methodology probably as early as 1998."

SUMMARY: Other opinions were that most laboratories will continue to use or modify EPA reference methods, and changes will come slowly. Regulatory agencies will be faced with less information because of proprietary methods. Regulators will be forced to take the end result and verify it by another "independent" lab that may use a different method. NELAC is setting up the rules under which labs' confidentiality can be protected, and PBMS approaches of labs can be sheltered. But if EPA is no longer the focus of new method development and distribution, there is likely to develop a two-tiered lab industry. One set will be small labs, who lack the resources to develop new and better ways for analyzing samples, and will continue to use standard reference methods. Their clients will be those who are required by states' laws to follow those methods. The second set of labs will be the large or chain labs, who can set devote people and equipment to R&D. They can work with vendors to produce better measurement technologies. These labs will reap the rewards of their investments, but will have no incentive to share them with the smaller labs. New measurement systems may also be driven by clients with specialized needs (overcoming industry or plant-specific interferences, etc.). But if the client pays for it, it won't be shared for free with others. New technology may be licensed or purchased, but it won't be free anymore. It may remain secret for as long as confidentiality agreements are in force, or until industrial espionage spills the secrets. 

ELAB WORKGROUP FOR LABORATORY ASSESSMENT

In June, 1998, the NELAC On-Site Assessment Committee reported comments from the contributing membership regarding audit checklists indicated a split of opinions between: quality systems approach to assessing laboratory capabilities, without the use of detailed method specific checklist, and a dual system that includes an audit of the quality system as well as a detailed review of methods using a method specific checklist. 

The Committee stated that for the NELAC to enjoy initial success, there must be a melding of the "method specific" and "quality systems" approaches to conducting the on-site assessments and in the development of the checklists. A Quality Systems/PBMS approach might be devised which could accomplish the same goal as a method specific approach to laboratory accreditation.

In response to NELAC's request for input, the Environmental Laboratory Advisory Board recruited 15 individuals from industry and government to form an ad hoc subcommittee to make recommendations to ELAB regarding the use of method checklists for laboratory assessments. These recommendations, reached by consensus after exchanging ideas by teleconferences and e-mail, are summarized below, and were submitted to the NELAC for consideration in July. 

ELAB CHECKLIST WORKGROUP RECOMMENDATIONS:

Checklists are a tool to use during an on-site assessment, but should not be the primary driver of the process. Checklists should be used in the context of a logical flow to get to the level of detail to allow determinations of acceptability (e.g., does the lab have an adequate quality system, and is it being followed in all aspects deemed important for the project/program being audited?) If the quality system documentation is lacking, there is no need to go to the level of a technical systems review. If the documentation is there, most labs do things the same way in each lab area--e.g., standards traceability, data review, calculation checks, etc.--so if these common items were answered at a higher level in a hierarchy of questions, any technical system review could be much shorter. 

Characteristics of a Technical Systems review

• Grouped by analytical technology/instrumentation
• Sample preparation and analysis are separate categories
• Data collection
• Data reduction
• Data verification
• Verification of Quality Systems implemented at bench level

Characteristics of Checklists

• detailed to be useful reminder, but not constricting
• assume auditor knowledge of details
• flows in order of lab review process
• simple
• logical
• not restrictive
• able to document findings
• able to accumulate information to build complete picture
• useful for lab to understand logic of deficiency
• need reference to NELAC standard
• should be widely available
• not to exceed 25 pages

An example checklist was provided.  Other recommendations included: Assessment should include actions taken by the lab on previous findings. The Quality Systems committee should define the major areas which should result in loss of accreditation if not corrected. NELAC should develop a process for feedback and continuous improvement to improve Quality Systems, On-Site Assessment, and the tools (checklists, manuals) used to assist in implementing.

CONFIDENTIALITY ISSUE AUDITORS WILL LIKELY HAVE TO SIGN NON-DISCLOSURES. EMPHASIS ON RESOURCES NEEDED, PROCESS, DOCUMENTATION, USE OF AUDIT RESULTS. 

PBMS may push the analytical community and users towards 3rd party certification and audits, because the agencies won’t be able to bear the costs of maintaining current expertise in a wide variety of fields. Labs may have to buy their lab audits and method approvals from certifying organizations. But maybe one approval and audit will suffice for all regulators, and labs won’t have to schedule "audits du jour." Audit and method performance data should be shared, but who can best maintain a database to collect and provide information for searches? What restrictions will be necessary?

Copyright (c) 1997-2009 Mentorprises Corporation