< Back

Dr. Llewellyn R. Williams, USEPA-Las Vegas, provided a summary of the collected public comments in response to the Federal Register announcement. These were scheduled for presentation to EPA Regional offices and stakeholder meetings, from March through June, 1998. He categorized them by responder class. The issues pertinent to audits are noted below:

Public Comments -- States

Implementation should be coordinated with NELAC

Training needed for inspectors

More difficult and time consuming to audit PBMS

Need to achieve reciprocity among states

Requires a strengthened proficiency program

Labs/permitees should provide states with written notice of intent to use a PBMS method

Public Comments -- Lab Industry

Could result in substantial cost savings

Will improve quality of inspections, data reliability

Will focus on data quality versus method compliance

Concerned about variability in state interpretations

Concerned about proprietary information

Checklists need improvement

Public Comments -- Industry

Want consistent approach among EPA programs

Fear states might use PBMS to lower permit limits

Implications for enforcement litigation

Inspectors need considerable training

Implementation should be coordinated with NELAC

From the e-mail survey of colleagues , issues (in bold text) and responses (in plain text) included the following:

For auditors:

From a related article in Analytical Chemistry News and Features, 12/1/96, "EPA Plans New Approach to Analytical Methods": "Some states worry that by abandoning prescriptive methods with their convenient checklists, the auditor’s job will get even more difficult. That problem would get even worse under the present NELAC plan, which gives states a significant role in accrediting laboratories for all EPA programs. It is unreasonable to expect that any inspector would be well versed in methods ranging from wet chemistries to ICP-MS.

What expertise is required to review the methods/modifications, and to visit the lab?

"The days of non-chemists auditing labs are over. The audit community already has a problem with too many rogue auditors out to find problems." " I see the emerging National Laboratory Accreditation Conference/Program (NELAC/NELAP) as being the logical vehicle for the needed laboratory inspections. Whether the States that will be implementing NELAC will have the needed expertise to do a credible job is still a nagging question."

Respondents independently summarized auditor qualifications in words that were consistent with the NELAC MAY, 1998 PROPOSED CHANGES, 3.2.1, "On-Site Assessment Personnel, Basic Qualifications," including

"Have a thorough knowledge of the relevant assessment methods and assessment documents;

Be technically knowledgeable and conversant with the specific tests or types of tests for which the accreditation is sought and, where relevant, with the associated sampling and preservation procedures."

3.2.2, Assessor Qualification: "Each assessor must sign a statement before conducting an assessment certifying that no conflict of interest exists and provide any supporting information as required by the accrediting authority."

What records do auditors need in planning the audit to become familiar with the method(s)/modifications(s)?

"Lots of data are desired up front. This should be reviewed by a team of chemists evaluating performance data before and during visits."

What should the audit team examine at the lab? What criteria should be applied?

"Audits should entail more review of raw data and electronic data along, with the quality systems review. Verification of continued acceptable performance is a must, including results of proficiency test samples for each method and analyst. Look at training records. Check calibration and QC data. Project or program-specific DQOs should be the driver, plus EPA-specified categories of information (MDLs, accuracy, precision, etc.). Recommend examination of PE samples, double-blind and real matrixes if possible. Examine data management systems, to follow the trail of method changes. Many labs are not currently able to manage the data for methods, standards, training, etc. that will be required to prove performance. "A key question is how audit checklists can be developed. Perhaps the only viable way will be to take the approach of the ISO 9000 audits where the SYSTEM is audited and certified. Then an appropriate list of "fields" would need to be developed to evaluate for completeness and adequacy for the PBMS SYSTEM at a laboratory." (See sidebar on ELAB Methods Checklist Work Group)

How often should the lab be audited?

Responses were consistent with the policy expressed in NELAC, 3.3.1, Frequency of On-Site Assessments: "Accrediting authorities must require a comprehensive on-site assessment of each facility that is accredited at least every two years. Assessments may be conducted more frequently for cause, at the option of the accrediting authority."

For the lab being audited, the issues include:

Providing requested up-front documentation to show EPA-established performance criteria are met, in an easily-understood format and with accompanying narrative. How much is enough?

"The lab should be able to show that it is following its own SOP and meeting method-specific criteria. As part of an initial certification process, NELAC would require the lab to provide records of personnel qualifications, proficiency test results, and quality systems, and undergo an on-site assessment. Chapters 2, 3, and 5 of the NELAC standards discuss these requirements."

Will satisfactory regular PE sample (WS, WP, etc.) results be required/accepted?

"Adequate procedures and test methods may not be in place by the time a performance-based approach is adopted. It is not enough to assess the performance of the analytical method because great variability exists in earlier stages of the measurement process. How the samples are prepared and subsampled for analysis can produce significant errors that often go unreported. A bank of contaminated material that could be characterized and inserted periodically into the sample stream as blind performance evaluation materials would go a long way in assessing performance of more of the measurement process." NELAC establishes a Proficiency Testing program (PT) which will provide standardized test samples with results verified against NIST SRMs, or an independently prepared calibration material. These will allow comparison across laboratories. Successful analysis of two sets of PT samples at least 30 days apart would be necessary for initial application for certification, then analysis of one PT sample twice per year in each field of testing would be required."

How much time will be required to provide on-site lab guides and technical contacts to demonstrate methods/modifications, produce additional documentation on performance (instruments, analysts, matrixes, etc.), and answer further questions?

"Depending upon the size of the lab, probably one week on site, hosting a team of auditors (3-6 people)."

How will the history of method changes be stored? To link performance with changes over time, a relational database structure may provide the tools for searching correlations between method changes, matrixes, time, instruments, etc.?

The history of method changes is documented by indicating the SOP in use at the time of analysis. The real issue is being able to identify a substantative change in a method that requires a distinct separation in performance assessments."

How much information about changes should be shared, and how much retained privately as a competitive advantage (faster, better, cheaper) for the lab?

The information should only be shared if written permission is specifically granted by the laboratory.

If there is a competitive advantage for the development and application of new methods/changes, who controls it, the client who paid for development or the lab who developed it?

"If a client pays a lab to develop something..its the clients' property and not the lab's. Proprietary changes and developments won’t be shared, so progress may be slowed. Labs will have to protect confidentiality, stamp SOPs as confidential or proprietary business information. Labs will require employees to sign confidentiality agreements, too, to prevent job changers from taking the methods with them. This will discourage sharing information on method improvements between labs. Vendors will be the ones to spread changes, by default. ASTM will assume a "provider of reference methods" role. Vendors may become brokers of such changes and improvements, if labs sign agreements to sell new technology. Big labs with R&D capability will add to their advantage, while little labs won’t keep up overall. Clients will have a more permanent bond to the lab if the lab develops a method tailored to their waste and a method that is more cost effective. Chemists and/analysts may change jobs less often due to signing non-disclosure and non compete agreements.

Will clients hesitate to authorize the use of non-prescriptive methods, such as PBMS?

Depends if clear advantages can be offered, plus assurances that data will still meet regulatory approval.

For the regulators, the issues include, but aren't limited to:

From "EPA Shifts into Performance Gear" by Andrea Kinney and Birgit Caliandro, in Today's Chemist At Work, April, 1998: "Implementation of PBMS means that DQOs will need to be established for all monitoring activities under RCRA (and under regulated programs)EPA will have to establish and provide training, because the technical expertise required by state and federal parties to review DQOs and their method counterparts, method quality objectives, for each method may not exist at the regional and local levels.

Will the local, state, and/or federal agencies maintain records of methods/modifications and audits, accessible to other agencies as well as labs, so the advances of any lab can be shared with all? Or must this information be protected under confidentiality agreements?

Confidentiality agreements, proprietary information stamps, non-disclosure agreements, etc. were all cited as essential to protect the laboratories and clients who invest in PBMS improvements. Each lab/client would decide who would receive the information?

If the auditing organization doesn't have the internal resources to review the data or conduct the on-site visit, who should help (e.g., states may not employ chemists familiar with state-of-the-art methods development and applications, so how will states evaluate such labs)?

Opinions differ: some say experienced chemists, some say trained and accredited auditors. Most recommended experienced, trained third party auditors(A2LA, NELAC, etc.) NELAC requires the use of trained and certified auditors, based on the NELAC, 3.2.1 and 3.2.2. A dissenter said: "Much of the private sector has no confidence or respect for A2LA or current state certifications. I fear that this lack of confidence will not change with one certified under NELAC." Another respondent said: "Government at all levels (Federal, State, local) has little or no capability to efficiently set criteria and make judgements about the myriad of different techniques that will be introduced by competing commercial interests. The complexity of analytical methods for environmental contaminants at very low concentrations is so great and the available talent so limited that it will be very expensive and time-consuming to even attempt this. Third party auditing which will likely be attempted and also prove a failure because of the lack of talent and the high cost."

How can the data user be assured the lab's results will be comparable, consistent, cost-competitive, reproducible over time as further changes are made, and readily accessible?

From "Is PBMS in Your Future?" by Larry Keith, in EnvirofACS, the newsletter of the Environmental Chemistry Division of the American Chemical Society: " First, we're still in the early stages of developing the PBMS approach. There are many identified issues that still need to be resolved and other issues that we may not even be aware of yet. Second, realistic DQOs are needed and many people will need help and training to be able to develop them for their individual situations. Third, state and federal regulators and other "data users" will have to understand some of the basic problems inherent with many environmental analyses (e.g., variability in the data, confidence levels of analyte identifications and concentrations, method selectivity and sensitivity, interferences, false positives and false negatives, etc.). Again, training may be needed to help understand these variables. Fourth, comparative tests of PBMS versus conventional consensus methods are not known to have been made yet. Nevertheless, PBMS is scheduled to begin as accepted methodology probably as early as 1998. (Footnote: Dr. Keith heads an American Chemical Society task force, formed to help EPA evaluate what information was needed as EPA moved toward implementation of PBMS. It concluded that before new approaches are used they should be compared to currently used prescriptive type methods. A study was proposed to EPA to evaluate several approaches for environmental monitoring using real world samples. The goals of the proposed study are to quantify the advantages, disadvantages, and relative costs of environmental monitoring using three approaches (i.e., prescriptive methods, streamlined methods, and PBMS). The study will compare the three approaches and determine:

The relative cost of monitoring under each approach.

The quality of data generated with each approach.

The degree to which data generated with each approach satisfactorily answers regulatory question for which monitoring is being conducted; and ease of implementation by the regulated community, the laboratory community, EPA, and state and local regulatory agencies." (Dr. Keith can be contacted at larry_keith@wpi.org)

Other opinions were that most laboratories will continue to use or modify EPA reference methods, and changes will come slowly. Regulatory agencies will be faced with not having as much information because of proprietary methods. Regulators will be forced to take the end result and verify it by another "independent" lab that may use a different method. More negotiation will be done on the chemistry issues as opposed to defaulting to SW 846. "This can be seen as an open door to welcome new and/or alternate technologies and an opportunity to encourage thinking and planning tobuy the right thing." "It is a natural outcome of the DQO process." "I think PBMS will push the analytical community and users towards 3rd party certification and audits, because the agencies won’t be able to bear the costs of maintaining current expertise in a wide variety of fields. Labs will have to buy their lab and method approvals from whatever organization the EPA and states can agree upon. On the bright side, it may be that one approval and audit is good for all the various regulatory bodies, and labs won’t have "audits du jour" to schedule and work around. Audit and method performance data should become shared." 

Copyright (c) 1997-2009 Mentorprises Corporation