Completion of Recommended Tests and Referrals in Telehealth vs In-Person Visits (2024)

Key Points

Question Is there a difference in the rate of completion for diagnostic tests and referrals ordered during a telehealth visit compared with those ordered during an in-person visit?

Findings In this cohort study of 4133 diagnostic tests and referrals (colonoscopies, cardiac stress tests, and dermatology referrals) ordered between March 1, 2020, and December 31, 2021, at 2 affiliated clinical primary care sites, 58% of those ordered during in-person visits were completed within the designated time frame compared with 43% of those ordered during telehealth visits. The rate of completion was between 40% and 65% for all test types, regardless of visit modality.

Meaning The findings of this study suggest that rates of completion for diagnostic tests and referrals were low for all visit types but worse when ordered during telehealth visits.

Importance Use of telehealth has increased substantially in recent years. However, little is known about whether the likelihood of completing recommended tests and specialty referrals—termed diagnostic loop closure—is associated with visit modality.

Objectives To examine the prevalence of diagnostic loop closure for tests and referrals ordered at telehealth visits vs in-person visits and identify associated factors.

Design, Setting, and Participants In a retrospective cohort study, all patient visits from March 1, 2020, to December 31, 2021, at 1 large urban hospital-based primary care practice and 1 affiliated community health center in Boston, Massachusetts, were evaluated.

Main Measures Prevalence of diagnostic loop closure for (1) colonoscopy referrals (screening and diagnostic), (2) dermatology referrals for suspicious skin lesions, and (3) cardiac stress tests.

Results The study included test and referral orders for 4133 patients (mean [SD] age, 59.3 [11.7] years; 2163 [52.3%] women; 203 [4.9%] Asian, 1146 [27.7%] Black, 2362 [57.1%] White, and 422 [10.2%] unknown or other race). A total of 1151 of the 4133 orders (27.8%) were placed during a telehealth visit. Of the telehealth orders, 42.6% were completed within the designated time frame vs 58.4% of those ordered during in-person visits and 57.4% of those ordered without a visit. In an adjusted analysis, patients with telehealth visits were less likely to close the loop for all test types compared with those with in-person visits (odds ratio, 0.55; 95% CI, 0.47-0.64).

Conclusions The findings of this study suggest that rates of loop closure were low for all test types across all visit modalities but worse for telehealth. Failure to close diagnostic loops presents a patient safety challenge in primary care that may be of particular concern during telehealth encounters.

The use of telehealth has increased substantially in recent years, by more than 60-fold in 2020.¹ Although telehealth made up less than 1% of medical visits before the COVID-19 pandemic, it became ubiquitous with the onset of the pandemic, before tapering to still-unforeseen levels, with 37% of adults reporting at least 1 telephone or video visit in 2022.² Past studies have examined patient experiences with telehealth, finding a high degree of satisfaction due to the ease of use, improved communication, and reduced wait and travel times.^3-6 Studies have also found high levels of diagnostic concordance (defined as degree of agreement) overall between telehealth and in-person visits.^7,8 However, little is known about how telehealth affects a patient’s likelihood of completing tests and specialty referrals—termed diagnostic loop closure.⁹

Failure to close referral loops is one of the leading causes of diagnostic errors in primary care. This is of particular importance with telehealth, given the finding of reduced diagnostic concordance¹⁰ in primary care due to the breadth of potential diagnoses.^7,9 The eventual outcomes of these oversights can be catastrophic; an estimated 12 million diagnostic errors occur annually in the US, contributing to 64 000 preventable deaths and costing the health care system $100 billion per year.^11,12

In this study, we examined the prevalence of the failure to close diagnostic loops for 3 representative types of tests and referrals ordered during primary care telehealth visits compared with those ordered during in-person visits and those ordered without a visit. We hypothesized that orders placed during telehealth visits were less likely to be completed due to a variety of patient, clinician, and system-level factors, which we sought to explore further.

We conducted a retrospective cohort study at 2 primary care sites in Boston, Massachusetts: 1 large urban hospital-based primary care practice and 1 affiliated community health center. We examined all visits from March 1, 2020, to December 31, 2021. The hospital-based site cares for approximately 42 000 patients while the affiliated community health center cares for approximately 6000 patients. We accessed clinical data for the 2 sites through a data warehouse, which maintains data for both and allows search by billing diagnoses, tests, referrals, and patient demographic characteristics. At both sites, patients have the option of both in-person visit and telehealth visits (telephone or video). Our research protocol was approved by the institutional review boards at Beth Israel Deaconess Medical Center and Harvard Medical School. The institutional review boards at both institutions deemed that informed consent from study participants was not required because only deidentified data were involved. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.¹³

We reviewed referrals and tests resulting from a concerning finding or need for screening, prompting the clinician to order an additional diagnostic workup. The order may have been placed during an in-person visit, a telehealth visit, or on a date when no visit occurred. We extracted data for 3 types of high-risk tests and referrals ordered between March 1, 2020, and December 31, 2021: colonoscopy referrals, dermatology referrals for suspicious skin lesions, and cardiac stress tests (including both exercise and chemical stress tests, with and without imaging). These tests were chosen due to their different characteristics, ease of test completion, high-risk nature with major clinical implications, and institutional prioritization. We then examined all orders for completion between March 1, 2020, to March 30, 2022.

Within all visit modalities, the order for testing or referrals is placed electronically through the electronic medical record. There is little to no proactive outreach for the scheduling of patients for routine orders. During an in-person visit, the patient is given a form with a telephone number to call to schedule the test or referral. For patients with limited English proficiency or complex needs, a member of the support staff team may help the patient schedule this test at checkout. During a telehealth visit, a patient is given the telephone number to call to schedule the test or referral by the clinician during the visit. In all scenarios, the patient receives no communication after the visit reminding them to schedule the test or referral.

Our main independent variable was the modality of the patient visit during which the diagnostic test or referral was ordered (in-person, telehealth [telephone and video], or ordered without a visit). Our primary outcome was failure to close the loop on the test or referral. The loop was considered closed if the orders were completed within 365 days for colonoscopy, 90 days for dermatology, and 45 days for cardiac stress test referrals. Time intervals for loop closure were determined through review of the literature, in conversations with physicians administering the tests based on the urgency of making a diagnosis, and with the goal of ensuring adequate time for patients to schedule despite current systematic delays.

We included several patient factors obtained from the electronic medical record as independent covariates. These included demographic variables such as age, sex assigned at birth (female or male), self-reported race (Asian, Black, White, or unknown or other), self-reported ethnicity (Hispanic, non-Hispanic, or unknown), educational level (high school or less, college, or unknown), primary language (English, other, or unknown), and insurance type (Medicaid, Medicare, commercial, or unknown or other), which we regarded as a surrogate for income level. Race and ethnicity were included due to the importance of identifying disparities in care as a first step toward addressing them. Race was divided into the following categories: American Indian or Alaska Native, Asian, Black or African American, White, declined to answer, other race or mixed race, prefer not to say, unable to obtain, and unknown or not specified. For ease of reporting, it was narrowed down (Table 1). The unknown or other category includes American Indian or Alaska Native, declined to answer, other race or mixed race, prefer not to say, unable to obtain, and unknown or not specified.

We also included patients’ chronic illness burden and psychiatric illness burden as covariates. Chronic illness burden was calculated using the Charlson comorbidity index (0, 1-2, or ≥3).¹⁴ This was determined using the billed diagnosis codes for all medical visits within 2 years of the index referral date. Psychiatric illness burden was based on a billing code for depression (International Statistical Classification of Diseases and Related Health Problems, Tenth Revision code F32) at a visit before or on the index referral date, using the date range (yes or no). Other markers of psychiatric illness, such as anxiety, were not examined.

Other covariates described the clinical site (hospital-based primary care practice and affiliated community health center), type of clinician ordering the test or referral (attending physician, resident physician, or nurse practitioner), and whether the patient had ever used the patient portal to review a visit note, which we used as a surrogate for patient engagement. We additionally included a marker for pause of elective procedures at our institution to represent periods of high COVID-19 infectivity, which occurred from March 13 to June 9, 2020; December 7, 2020, to March 1, 2021; and November 21, 2021, to January 27, 2022.

We compared patient characteristics between the 3 groups: patients with orders placed during in-person visits, telehealth visits, and no visit. We then performed a univariate analysis with loop closure as the outcome. For the primary comparison of loop closures associated with telehealth visits with in-person visits, we adjusted for all the patient factors listed above using multivariable logistic regression. Additionally, we performed a stepwise regression analysis to identify the extent to which groups of covariates contributed to the findings (ie, we sequentially adjusted for grouped covariates to determine which group best explained the univariate result). The stepwise regression analysis first adjusted for the test type, then patient demographic characteristics and comorbidities, then clinical site and clinician type, and finally patient engagement. For tests of statistical significance, analysis of variance was used for continuous variables and χ² analysis was used for all other variables. We conducted all statistical analyses using Stata, version 16.1 (StataCorp LLC), with statistical significance set at a 2-sided α value of .05.

Among 4133 patients with test and referral orders, the mean (SD) age was 59.3 (11.7) years; 2163 (52.3%) were women and 1970 (47.7%) were men; 203 (4.9%) were Asian, 1146 (27.7%) were Black, 2362 (57.1%) were White, and 422 (10.2%) had an unknown or other race (Table 1). The 4133 orders included colonoscopy referrals (n = 3254), dermatology referrals for suspicious skin lesions (n = 455), and cardiac stress tests (n = 424). Of these, 2167 (52.4%) were ordered during an in-person visit, 1151 (27.8%) were ordered during a telehealth visit, and 815 (19.7%) were ordered without a visit. Compared with patients with in-person visits, patients with telehealth visits were more likely to be White, have higher educational attainment, have English as their primary language, have commercial insurance, and to have been seen by an attending physician (Table 1).

Of the 1151 orders placed during a telehealth visit, 490 (42.6%) were completed within the designated time frame vs 58.4% (1265 of 2167) placed during an in-person visit and 57.4% (468 of 815) placed without a visit (Figure 1). In the univariate analysis, patients with telehealth visits were less likely to close the loop on high-risk tests and referrals overall compared with in-person visits (odds ratio [OR], 0.53; 95% CI, 0.46-0.61) (Table 2). Patients with orders placed without an in-person or telehealth visit were equally likely as patients with an in-person visit to close the loop on ordered tests and referrals. A stepwise multivariable regression that adjusted for groups of covariates sequentially did not result in significant changes in rates of loop closure when comparing telehealth visits with in-person visits (Table 3). In the adjusted analysis, patients with telehealth visits were still less likely to close the loop on all tests and referral types overall (OR, 0.55; 95% CI, 0.47-0.64). When adjusting for periods of high COVID-19 infectivity, the overall odds of loop closure were 0.58 (95% CI, 0.51-0.67) across all referrals and test orders. The rate of loop closure plateaued before the allotted time frame expired for all test types (colonoscopies, 365 days; dermatology referrals, 90 days; cardiac stress tests, 45 days) and the time to loop closure did not appear to differ between orders originating from telehealth vs in-person visits (Figure 2).

We identified a total of 3254 colonoscopy referrals, which made up 78.7% of all orders; 998 (30.7%) were ordered during a telehealth visit, 1570 (48.2%) were ordered during an in-person visit, and 686 (21.1%) were ordered without a visit. Of the colonoscopy referrals ordered at a telehealth visit, 39.8% (n = 397) were completed within 365 days, compared with 56.9% (n = 893) ordered at an in-person visit and 56.7% (n = 389) ordered without a visit. Before adjustment for the covariates, patients with telehealth visits were less likely to close the loop on colonoscopy referrals compared with in-person visits (OR, 0.50; 95% CI, 0.43-0.59). This remained consistent in the adjusted analysis (OR, 0.49; 95% CI, 0.42-0.58) (Table 2).

We identified a total of 455 dermatology referrals for suspicious skin lesions, which made up 11.0% of all orders; 65 (14.3%) were ordered during a telehealth visit, 301 (66.2%) were ordered during an in-person visit, and 89 (19.6%) were ordered without a visit. Of the dermatology referrals ordered at a telehealth visit, 63.1% (n = 41) were completed within 90 days, compared with 61.5% (n = 185) ordered at an in-person visit and 62.9% (n = 56) ordered without a visit. There was no significant difference between telehealth and in-person visits for dermatology referral loop closure in the unadjusted (OR, 1.07; 95% CI, 0.61-1.87) or adjusted (OR, 1.07; 95% CI, 0.60-1.95) analyses (Table 2).

We identified a total of 424 cardiac stress tests, which made up 10.3% of all orders; 88 (20.8%) were ordered during a telehealth visit, 296 (69.8%) were ordered during an in-person visit, and 40 (9.4%) were ordered without a visit. Of the stress tests ordered at a telehealth visit, 59.1% (n = 52) were completed within 45 days, compared with 63.2% (n = 187) ordered at an in-person visit and 57.5% (n = 23) ordered without a visit. There was no significant difference between telehealth and in-person visits for stress test loop closure in the unadjusted (OR, 0.84; 95% CI, 0.52-1.37) or adjusted (OR, 0.84; 95% CI, 0.49-1.43) analyses (Table 2).

Loop closure rates for selected diagnostic tests and specialty referrals were unacceptably low across all visit modalities. Furthermore, in a multivariable analysis, patients with telehealth visits were substantially less likely to close open loops compared with patients with in-person visits. In subgroup analyses based on test type, patients with telehealth visits were less likely to close the loop on colonoscopies, but statistically significant differences were not detected for urgent dermatology referrals or stress test orders. Sequentially adjusting for test type, patient demographic characteristics, comorbidities, clinical site, clinician type, and patient engagement did not markedly change the odds of loop closure in the stepwise regression analysis.

As reported, the rate of loop closure was low (between 40% and 65%) for all test types, regardless of visit modality, compared with an acceptable threshold of 80% in population health metrics and an ideal goal of 95% in models based on expert consensus.¹⁵ This is consistent with research that has found suboptimal completion rates for tests ordered in urgent care scenarios and suggests that common system factors may play a more important role in loop closure than visit modality.¹⁶ Thus, while the differences in loop closure between telehealth and in-person visits may be concerning, system-level changes are needed to improve test completion rates across all modalities. These might include automated tracking for outstanding tests within electronic medical records and interventions such as telephone outreach to patients, automated text and email reminders, and the use of referral managers.¹⁷ These considerations may be particularly important for patients who rely heavily on telehealth, such as those in remote rural areas and disadvantaged patients with limited health access and literacy.

When investigating notable differences in loop closure for orders placed during telehealth visits, our findings suggest that differences in loop closure may be inherent to telehealth as a modality. One potential mechanism to explain this may be the lack of systems in place to help patients complete test and referral orders. During in-person visits, members of the support staff team sometimes help patients schedule their tests at checkout; however, this support is absent during telehealth visits. After the visit, patients do not receive any communication reminding them to schedule the test or referral, which may further limit loop closure. Other potential explanations include the possibility that it may be more difficult to remember information provided during telehealth visits, that telehealth may present unique communication barriers, or that it may be more difficult to engage patients in shared decision-making during virtual visits, thus decreasing patient engagement with test and referral orders.^18-21

These barriers may be more pronounced for colonoscopies compared with other types of orders given the increased difficulty in preparing for and completing the invasive procedure, thus explaining the disparities in loop closure compared with the other orders included in our analysis.²² Furthermore, given the notable difference in loop closure for colonoscopies, telehealth visits may serve as a marker for patients not participating in a high-complexity test. They may also highlight an important association between test indication and test completion, with screening indication yielding lower priority for both the patient and physician. This builds on past work that has identified lower rates of patient adherence to hemoglobin A_1c testing recommendations during telehealth encounters.²³ With this association in mind, an order for a routine test placed during a telehealth visit might serve as a process marker to trigger additional assistance or alternative ways to complete the indicated test. Our observation that telehealth does not seem to worsen loop closure for tests prompted by patient symptoms or concerns (suspicious skin lesions or chest pain) suggests that patient activation may be a key factor in closing loops and may overcome some of the challenges inherent to telehealth visits.

The widespread failure to close diagnostic test and referral loops identified in this study may also provide a mechanism to explain past findings that primary care practices with higher telehealth use are associated with increased rates of acute care visits and that primary care telehealth visits are associated with increased emergency department encounters,^24,25 possibly due, at least in part, to a failure to close the loop on ordered tests. These findings should be contextualized within the literature, which has identified significant advantages to telehealth vis-à-vis convenience, access, and diagnostic concordance.^5-8 Nonetheless, our findings highlight a potential limitation to telehealth that, to our knowledge, had not been previously identified.

Strengths of this study include the novelty of the inquiry, variety of tests evaluated, adjustment for confounding, and patient characteristics that are consistent with prior reports evaluating the composition of telehealth patient populations.^1,2 This study has limitations. First, the study lacks data on the urgency of orders, scheduled tests and referrals that were canceled, and clinical outcomes resulting from the failure to close diagnostic loops. The CIs for estimates of loop closures when comparing telehealth visits with in-person visits for dermatology referrals and stress tests were wide, suggesting that our smaller sample sizes may have reduced our opportunity to identify important differences. Similarly, although the study used data from 2 different primary care settings, there were too few visits at the community site to detect significant differences. We lack data to fully explain why orders without a visit had completion rates that were similar to those of face-to-face visits. Whether the orders resulted from a patient telephone call or message was not available in the data set. In addition, the multivariable stepwise analysis suggests possible issues with residual confounding.

In this cohort study, completion rates for diagnostic tests and referrals were low for all treatment modalities at 2 primary care sites, but worse for telehealth. These results are among the first to document how telehealth affects patients’ likelihood of completing recommended tests and specialty referrals. Telehealth visits present novel opportunities to reduce diagnostic errors and may play an increasingly important role in clinical practice given the continued expansion of telehealth following the COVID-19 pandemic.

Accepted for Publication: October 3, 2023.

Published: November 15, 2023. doi:10.1001/jamanetworkopen.2023.43417

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2023 Zhong A et al. JAMA Network Open.

Corresponding Author: Maelys J. Amat, MD, MBA, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Boston, MA 02215 (mamat@bidmc.harvard.edu).

Author Contributions: Dr Amat had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Mr Zhong and Dr Amat shared first authorship.

Concept and design: Zhong, Amat, Anderson, Sternberg, Fernandez, Schiff, Benneyan, Phillips.

Acquisition, analysis, or interpretation of data: Anderson, Shafiq, Sternberg, Salant, Fernandez, Schiff, Aronson, Benneyan, Singer, Phillips.

Drafting of the manuscript: Zhong, Amat, Phillips.

Critical review of the manuscript for important intellectual content: Anderson, Shafiq, Sternberg, Salant, Fernandez, Schiff, Aronson, Benneyan, Singer.

Statistical analysis: Anderson, Shafiq, Fernandez, Benneyan.

Obtained funding: Schiff, Benneyan, Singer, Phillips.

Administrative, technical, or material support: Amat, Sternberg, Schiff, Aronson, Benneyan.

Supervision: Amat, Fernandez, Schiff, Phillips.

Conflict of Interest Disclosures: Dr Anderson reported receiving grants from the American Heart Association, US Deprescribing Research Network, and American College of Cardiology outside the submitted work. Dr Phillips reported holding small equity interest in Bicycle Health and Grow Therapy outside the submitted work, serving as an expert witness for Morrison Mahoney LLP, and being an unpaid member of the scientific advisory board of Maven Project, a nonprofit organization. No other disclosures were reported.

Funding/Support: This work was funded by Agency for Healthcare Research and Quality (AHRQ) grant R18 5R18HS027282. Dr Amat was supported by the Arnold Tofias and Leo Condakes Quality Scholarship Program. Dr Anderson was supported by a grant from the National Institute on Aging (K76AG074878).

Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ or the National Institutes of Health.

Data Sharing Statement: See the Supplement 1.

Additional Contributions: Dru Ricci, BA (Harvard Medical School Center for Primary Care), assisted with the preparation of the manuscript. No financial compensation was provided.