Peter Jenks
The Jenks Partnership, Salisbury, UK. [email protected]
DOI: https://doi.org/10.1255/sew.2021.a4
© 2021 The Author
Published under a Creative Commons BY-NC-ND licence
Every now and then a new analytical challenge appears and, for the most part, the ISO/IEC 17000 series of standards adapts and evolves to include the new challenge. The latest challenge, testing for the presence, or absence, of the virus that causes COVID-19 seems to be a particularly difficult challenge.
How can this be? It is worth first looking at how chemical and clinical metrology has evolved over the last 25 years since I became part of this never-ending journey. On my journey I have learnt some basic truths about analytical science and how quality, reliable data can be developed, mainly through the application of ISO/IEC Standards.
The publication of ISO/IEC 17025 in 1999 was the genesis of robust quality management in analytical science. The standard requires, amongst many other things, validation of the method used, in some cases including sample collection and preparation where this part of the analytical process has a significant impact on the final result. Instrument calibration follows, using wherever possible Certified Reference Materials (CRMs) produced by an ISO/IEC 17034 accredited producer and on-going quality control using Reference Materials (RMs). Periodic proficiency testing is also required, if available, from a provider accredited to ISO/IEC 17043.
The application of the 17000 series of ISO/IEC Standards to chemical metrology differentiates it from many other areas of scientific measurement, in that unlike most ISO standards for management systems, assessment of the laboratory is normally carried out by a third party, normally the national organisation responsible for accreditation. Laboratories are, therefore, “accredited” under ISO/IEC 17025, rather than “certified” or “registered” by a third-party service as is the case with ISO/IEC 9000 quality standard. The Accreditation Body attests annually the technical competence within a laboratory, in addition to its adherence and operation under a documented quality system, specific to a Scope of Accreditation.
The result of this is that the quality of analytical data produced by ISO/IEC 17025 accredited testing laboratories in the environmental, food and beverage, pharmaceutical, minerology and metallurgy and construction materials sectors has reached a level of excellence which means the original concept behind ISO/IEC 17025, “Tested Once, Accepted Everywhere”, is slowly becoming a reality in many sectors. There are concerns: many respected scientists would never employ any analytical lab without independently reassuring themselves of the lab’s competence. Experience shows it is possible to send the same sample to three accredited labs and get three significantly different results.
One of the features of the ISO/IEC 17000 quality management umbrella is that the product of testing, analytical data, should wherever possible be accompanied by an estimate of total uncertainty and traceability to the SI system, through an unbroken chain of comparisons.
In clinical chemistry the basic tenets of the ISO/IEC 17000 series approach to quality management have been included in ISO/IEC 15189, which is, in effect, the Accreditation Standard for medical and diagnostic testing. In validating a diagnostic or clinical test, the Sample Preparation and Extraction or Sample Preparation stage would always be also validated, since recovery from the bio matrix varies significantly from method to method.
Also, as I understand it, ISO/IEC 15189 applies to all clinical tests, “where appropriate” including small molecule, drug or endogenous small molecules used as bio markers. The main reason that large molecules are not included, be they proteins or oligonucleotides, was that CRMs or RMs have not been routinely available and that there is difficulty in linking chemical amount of substance with biological activity. So, it is very difficult, or impossible, to demonstrate commutability, which is defined as “the equivalence of the mathematical relationships between the results of different measurement procedures for a reference material and for representative samples from healthy and diseased individuals”.
This brings me to the tests used to detect infection from the COVID-19 virus during the present pandemic. Polymerase Chain Reaction (PCR) tests are nothing new; they were first described in 1983 by the American biochemist Dr Kary Mullis at Cetus Corporation, and for which he won a half share in the 1993 Nobel Prize for inventing the PCR process.
Since then, PCR tests have been used to measure the presence of DNA, especially in forensic analysis and the study of DNA in research applications. Most PCR methods utilise thermal cycling; this exposes reactants to repeated cycles of heating and cooling to allow different temperature-dependent reactions, primarily DNA melting and enzyme-driven DNA replication. PCR employs two main reagents—primers (which are short, single-strand DNA fragments known as oligonucleotides that are a complementary sequence to the target DNA region) and a DNA polymerase. In the first step of PCR, the two strands of the DNA double helix are physically separated at a high temperature in a process called nucleic acid denaturation. In the second step, the temperature is reduced so the primers bind to the complementary sequences of DNA. The two DNA strands then become templates for DNA polymerase to enzymatically assemble a new DNA strand from free nucleotides, the building blocks of DNA. As PCR progresses, the DNA generated is itself used as a template for replication, setting in motion a chain reaction in which the original DNA template is exponentially amplified. The formula used to calculate the number of DNA copies formed after a given number of cycles is 2n, where n is the number of cycles. Cycling continues until a fluorescent indicator included in the reagents binds to the synthetic DNA and can be detected. Thus, a reaction set for 30 cycles results in 230, or 1,073,741,824, copies of the original double-stranded DNA target region.
It must not be forgotten that the higher the number of cycles needed to reach a detectable signal for the viral genome, the lower the amount of viral load that is in the sample; the lower the number of cycles, the more prevalent the virus was in the original sample. It would be helpful if the number of cycle thresholds needed could be used as an important metric by which patients, the public and policy-makers can make more informed decisions about how infectious and/or sick an individual with a positive COVID-19 test might be, but there is absolutely no evidence to support such an assertion.
A recent study from the Infectious Diseases Society of America,1 suggested that at 25 cycles of amplification, 70 % of PCR test “positives” are not “cases”, since the virus cannot be cultured because it is dead, and by 35 cycles it is reported that 97 % of the positives are non-clinical. The study has not been validated by others and it is worth considering that without evidence linking the data to clinical histories it is possible that the “false positives” may be because the virus has been cleared from the body by the patient’s immune system, leaving unviable RNA fragments.
As stated above, rt-PCR is not testing for disease, it is testing for a specific RNA and this is the key pivot. At 25 thermal cycles, 70 % of the positive results are not really “positives” in any clinical sense, since fragments of RNA cannot make anyone ill. Further, it is the functioning of complete viral RNA inside the infected cell that does lead directly to making a person ill, so this diagnostic assay is not detecting infectious virus but solely the viral genome.
Despite not being intended as a diagnostic tool, developments in PCR have made possible the rapid and highly specific diagnosis of certain infectious diseases, including those caused by bacteria or viruses. Cai et al.’s paper2 explains this in more detail, from a medical perspective.
Viruses are unique in having a genome composed of either DNA or RNA and unlike every other “living thing”, never both. There are, however, many viruses whose genome is composed of DNA, good examples are Varicella zoster (chickenpox), herpes simplex etc. There are also viruses, e.g. HIV, that have an RNA genome in the virus, but which generate a DNA copy of their genome as part of their replication process. The SARS viruses are all RNA and the first successful use of Reverse Transcriptase PCR (rt-PCR) was with the development and validation of a method to detect the SARS-CoV-2 viral genome.3
It is hardly surprising that rt-PCR was the method of choice to detect the COVID-19 as it is a member of the same SARS-CoV-2 family of viruses, along with at least four common cold viruses. The development of a COVID-19 diagnostic test was carried out at great speed by a multitude of teams, all based on the genome released by the Chinese researchers in January 2020. An article by Vandenburg et al.4 noted that more than 260 different rt-PCR tests for COVID-19 had been developed.
Vandenberg et al. also pointed out that the new, novel rt-PCR tests are not yet fully validated and should be considered as simply qualitative tests. There are most certainly grounds to believe the tests produce false negative results, as Kucirka et al.5 make clear.
Applying an ISO/IEC perspective to all of this suggests that, whilst further optimisation of tests and more extensive clinical and epidemiological validation, including formal FDA/EMEA approval, are still needed, an ISO/IEC Standard that can be used when developing future new novel test methods is also needed. The lack of rigorous method validation and appropriate Quality Assurance/Quality Control (QA/QC) leads me to three real concerns.
Sampling
Taking a swab from specific areas at the back of the throat and upper nose is not simple: to take samples effectively and reproducibly requires the sampling procedure to be standardised and carried out by trained personnel, yet many samples are taken at home by the patient or a patient’s family member. This adds a significant potential for error through incomplete sampling.
False positives
All the PCR tests are based on detecting a specific fragment of viral RNA. So, they all amplify sequences from two viral genes (ORF1a, or viral spike protein, that is the gene used for vaccine development and the N, or nucleocapsid, gene) and a positive PCR signal is required for both genes before the test is scored as positive.
Without a standardised viral fragment to be used for both target and as a reference sample and without real agreement on the number of thermal cycles, effectively setting a detection limit, the rt-PCR test remains but a qualitative screening test and not a diagnostic tool.
Commutability
A fragment of RNA found in a nasal swab can have three possible sources. Infected cells shedding active viruses, remains of dead viruses killed by the host’s immune system (these can linger for weeks or months after the patient has recovered from the infection) and dead RNA fragments inhaled from the environment. Only infected cells shedding active viruses are of medical or epidemiological concern, dead viruses are background noise. Jaafar et al.1 have demonstrated that many positive tests are from non-viable RNA, and so cannot be grown using classical virology methods: in a way this is a demonstration of negative commutability.
In mid-December 2020 the WHO issued guidance6 on the use of rt-PCR as a diagnostic tool for COVID-19, warning users that any positive result (SARS-CoV-2 detected) or negative results (SARS-CoV-2 not detected) should not be used in isolation but in combination with specimen type, clinical observations, patient history and epidemiological information.
There are research efforts to find other ways to detect viruses: Professor Christelle Prinz and Dr Edouard Berrocal working at Lund University in Sweden7 have been funded by the European Research Council to further develop a novel spectrophotometric method for the measurement of viruses. The article is interesting, and could be a significant new development, but the question of commutability still remains. The lateral flow test has attracted much attention; unlike PCR, it does not test for viral RNA but particular proteins made by the virus. Instead of taking two days to get a result, they take half an hour or so.
Taken together these issues show that the present testing regime in many countries fails to meet the requirements of ISO/IEC 17025 or ISO/IEC 15189 and Good Laboratory Practice. I do not believe the rt-PCR tests as presently used for mass testing can be considered to be any more than a qualitative screen with a high level of uncertainty, and should only be used along with other methods and a classical review of demonstrated symptoms to confirm infection of symptomatic patients. Despite this, various Governments believe their response to the pandemic has been science led, when in fact the opinions of a few notable high-profile scientists interpreting a test that is not proven to be fit for purpose as a diagnostic tool has been the main driver.
Why do I hold this view? Back in the early 70s, during my second year at Brunel University, we had one-term course entitled “Fundamentals of Science”, given by Dr John Dore, a botanist. The takeaway message from the course was that as a scientist you have to question everything you read or are told and trust nothing that has not been third-party verified, if possible, by someone you know and trust. I think this is a principle that applies to everything that is described as “science” and made public in any vehicle. I have stuck to this approach throughout my career.
Although I started out my postgraduate career as a medical biochemist, studying at the Woolfson Centre at Birmingham University under Professor T.P. Whitehead, one of the “founding fathers” of external Quality Assessment of Clinical Laboratories, I migrated quite quickly into the commercial side of analytical chemistry and biochemistry, including microbiology. So, I do have the background to be able to take an objective view of what is happening.
I am profoundly unhappy that many Governments are making changes to society that impact on the life and health of many using data from a qualitative test that has a high level of uncertainty, caused by sampling error, lack of method validation and no real QA. It is simply bad science and undermines public trust in “science”.
References
- R. Jaafar, S. Aherfi, N. Wurtz, C. Grimaldier, T. Van Hoang, P. Colson, D. Raoult and B. La Scola, “Correlation between 3790 quantitative polymerase chain reaction–positives samples and positive cell cultures, including 1941 severe acute respiratory syndrome Coronavirus 2 isolates”, Clin. Infect. Dis. ciaa1491 (2020). https://doi.org/10.1093/cid/ciaa1491
- H.Y. Cai, J.L. Caswell and J.F. Prescott, “Nonculture molecular techniques for diagnosis of bacterial disease in animals: a diagnostic laboratory perspective”, Vet. Pathol. 51(2), 341–350 (2014). https://doi.org/10.1177/0300985813511132
- Coronavirus: il Viaggio dei Test. Istituto Superiore di Sanità (2020). https://www.iss.it/primo-piano/-/asset_publisher/o4oGR9qmvUz9/content/id/5269706
- O. Vandenberg, D. Martiny, O. Rochas, A. van Belkum and Z. Kozlakidis, “Considerations for diagnostic COVID-19 tests”, Nat. Rev. Microbiol. (2020). https://doi.org/10.1038/s41579-020-00461-z
- L.M. Kucirka, S.A. Lauer, O. Laeyendecker, D. Boon and J. Lessler, “Variation in false-negative rate of reverse transcriptase polymerase chain reaction-based SARS-CoV-2 tests by time since exposure”, Ann. Intern. Med. 173, 262–267 (2020). https://doi.org/10.7326/M20-1495
- “Nucleic acid testing (NAT) technologies that use real-time polymerase chain reaction (RT-PCR) for detection of SARS-CoV-2”, WHO Information Notice for IVD Users (2020). https://www.who.int/news/item/14-12-2020-who-information-notice-for-ivd-users
- Prestigious ERC Grants for New Covid Test and Spectrometer (2021). https://www.lunduniversity.lu.se/article/prestigious-erc-grants-new-covid-test-and-spectrometer
Peter Jenks
Since graduating from Brunel University London in the mid-1970s, I have been closely involved in the building of Quality Business in Analytical Science around the world. It has been a fascinating and enjoyable journey through the ever-changing analytical chemistry world. I’ve had to constantly seek out new developments and opportunities, technical, commercial and geographical. In 2013, I was invited to become a Fellow of the Royal Society of Chemistry, in recognition of my contribution to increasing awareness of metrology in chemical analysis. I have held senior roles in business development and licensing for a number of manufacturers.
