Table of Contents
Background and aim
Respiratory Rate-Oxygenation (ROX) and modified ROX (mROX) indexes have been proposed to detect early high-flow nasal cannula (HFNC) therapy failure. We evaluated the utility and relationship of ROX and mROX indexes in COVID-19 patients started on HFNC oxygen therapy.
This pilot study collected data from adult COVID-19 patients requiring HFNC oxygenation from 29 Jan - 29 Jun 2021. The patients were divided into two cohorts based on HFNC therapy success. ROX and mROX were compared using statistical diagnostic testing, including receiver operating characteristics and area under the curve (AUC) using online Epitools (epitools.ausvet.com.au/) and MedCalc software (MedCalc Software Ltd, Ostend, Belgium, www.medcalc.org/); p<0.05 was considered significant.
Twenty-seven patients fulfilled the inclusion criteria; 48.15% of therapy failed. The cohort's mean ± standard deviation age was 53.93 ± 10.67 years; 74.1% were male. The accuracy of predicting failure for mean ROX versus mROX at baseline and six-hour values was 59.81 versus 70.68 and 67.42 versus 74.88, respectively (all p>0.05). The AUC for ROX and mROX at baseline and at six hours were statistically indifferent. Only an mROX of 4.05 (mean value) and 3.34 (Youden’s J cut-off) had a sensitivity plus specificity at 156% and 163%, respectively.
Both ROX and mROX at baseline and six hours had fair-to-good accuracies and AUC; the differences were insignificant. Both ROX and mROX had better accuracies at six hours. However, only mROX < 4.05 at six hours fulfilled the sensitivity plus specificity criteria to be a clinically valuable screener.
Severe acute respiratory syndrome following the Coronavirus disease of 2019 (COVID-19) has been a global pandemic for nearly two and a half years. These patients are often hypoxemic and need oxygen supplementation. The high-flow nasal cannula (HFNC) oxygen (O2) therapy is newer yet commonly used in critical care during acute hypoxemic respiratory failure (AHRF) arising from postoperative pulmonary complication, pulmonary edema, heart failure, and even in mild hypercapnia respiratory failure. The World Health Organization and other scientific societies list HFNC O2 among the options for ventilatory support. A recent systemic review indicates that HFNC O2 therapy may reduce the need for invasive ventilation and escalation of therapy compared to conventional oxygen therapy in COVID-19 patients with AHRF . Also, in their recent guidelines on managing critically ill adults with COVID-19, the Surviving Sepsis Campaign recommends using HFNC over non-invasive positive pressure ventilation (NIPPV) .
However, HFNC oxygenation might delay identifying the need for intubation that would have become necessary. This delay is associated with a poorer prognosis . Roca et al. have formulated and subsequently validated the diagnostic accuracy of an index termed Respiraratory Rate-Oxygenation (ROX) and defined as the ratio of peripheral oxyhemoglobin saturation (SpO2) and the fraction of inspired O2 (FiO2) to respiratory rate (RR) for determining HFNC failure. They demonstrated that the ROX index could assist the clinician in deciding whether or not to intubate patients on HFNC O2 therapy for AHRF .
It has been seen that the respiratory rate of patients with COVID-19-related AHRF is sometimes lower than those with other AHRF due to other common causes like pneumonia, and acute pulmonary edema, which can be attributed to the severe level of hypoxia associated with it. As the ROX index uses respiratory rate as the denominator, the ROX index threshold for predicting HFNC O2 therapy failure may be different in the case of COVID-19-associated AHRF. Further, in some patients with COVID-19, the phenomenon of "Happy Hypoxia," or more correctly, "Silent Hypoxia," has been seen . Such patients exhibit shallow oxygen levels (demonstrated by pulse oximetry values) but do not demonstrate dyspnoea, irritability, or other signs of hypoxia. Furthermore, while SpO2/FiO2 has a good correlation with a partial pressure of O2 (PaO2) and FiO2 with low concentrations of supplemental oxygen , whether this relationship stands well in patients receiving FiO2 of 100% is not well established. Nevertheless, the correlation between SpO2/FiO2 with PaO2/FiO2 and the fall of SpO2 with PaO2 is not linear [7, 8].
As SpO2 is also used as one of the parameters for calculating the ROX index, the accuracy of the ROX index thresholds in predicting HFNC failure in COVID-19 patients might have to be validated again. A relevant modification of the said index has been proposed by Karim and Esquinas, termed the "Modified ROX (mROX) index” . The mROX has proposed using PaO2 in place of SpO2 to calculate the index. The present prospective, observational pilot study was aimed to evaluate the relationship of ROX and mROX index for predicting failure of HFNC O2 therapy in COVID-19 patients with AHRF.
The article's preprint was submitted to the Research Square platform on May 23, 2022, which is available from https://assets.researchsquare.com/files/rs-1680854/v1/ececdb62-5bd8-4648-a922-ecf5d12c423b.pdf?c=1653317976, last accessed Dec 24, 2022.
Materials & Methods
The present observational pilot study was conducted in a tertiary care teaching and research institute in India. The data collection was done from 29 Jan 2021 to 29 Jun 2021. The research protocol was reviewed by the research cell, approved by the institutional ethical committee, and subsequently registered in the clinical trial registry of India prospectively (CTRI/2021/01/030431). The study was initially approved to enrol up to 65 years of age; however, it was later amended to include up to 70 years age groups. The study follows the Declaration of Helsinki and the Good Clinical Practice Guideline for biomedical research. The study result is reported per the Strengthening the Reporting of Observational Studies in Epidemiology guideline.
Patients admitted with COVID-19 infection requiring intensive care management and on HFNC oxygen therapy for oxygen supplementation were screened for enrolment. The inclusion criteria were patients aged 18-70 years on HFNC oxygen therapy with continuous SpO2 and RR monitoring. Patients with known pre-existing respiratory diseases like chronic obstructive pulmonary disease (COPD), bronchial asthma, idiopathic pulmonary fibrosis, active cardiac diseases, and chronic renal failure with pulmonary edema were excluded. The sample size was calculated using an online calculator: www.crutzen.net/n.htm . We hypothesized a 10% probability of finding a difference between the ROX and mROX and considered a 95% confidence, resulting in a sample size of 29 patients.
Data were collected by observing the patients' parameters from the multiparameter monitor, files, and reports. The investigating team did not intervene in any phase of the treatment of the patients. All patients were on continuous peripheral oxyhemoglobin saturation (SpO2) monitoring; arterial blood gas (ABG) analysis was often done to assess the progress of the artificial respiratory support and escalation-de-escalation of support. All patients put on HFNC O2 therapy were approached for recruitment, and only consenting patients were enrolled. Over and above demographic and history of illness-related data, respiratory rate (RR), SpO2, and the fraction of inspired O2 (FiO2) data were collected every six hours till the patient was either weaned from HFNC or therapy was converted to NIPPV (Bi-level) or invasive mechanical ventilation (IMV). RR, SpO2, and FiO2 data were collected whenever the treating team collected an ABG sample. Further, when the treating team decided to intubate or escalate from HFNC, considering it a failure, RR, SpO2, FiO2, and ABG data were also collected at that time point. ROX and mROX index values were calculated from these data. HFNC failure was defined as the need for either NIV (Bi-level) or IMV, as decided by the attending physician. Usually, decreased level of consciousness (Glasgow Coma Score <12), cardiac arrest/arrhythmias, and severe hemodynamic instability requiring norepinephrine (>0.1μg/kg/min) or persisting or worsening respiratory conditions are defined as at least two of the following criteria: failure to achieve adequate oxygenation (PaO2 <60 mmHg or SpO2 <90% despite HFNC flow ≥60L/min and FIO2 of 100%), respiratory acidosis (PaCO2 >55 mmHg or with pH <7.25), RR >30/min or inability to clear secretions were the factors considered for determining the failure. However, a strict protocol for adherence was absent, but the investigating team noted these criteria-related data at the failure time point. The disease severity was assessed by acute physiology, chronic health evaluation II (APACHE II) score at 24 hours, and organ failure by sequential organ failure assessment (SOFA).
Continuous variables are reported as mean ± standard deviation (SD) or median [interquartile range (IQR)] when appropriate. The differences between the two groups were analyzed by Student’s t-test or Mann-Whitney U test. Categorical variables are reported as numbers and percentages and analyzed using Fisher's exact or Chi-square test. INSTAT software (Graphpad Prism Software Inc., La Jolla, USA) was used for this analysis. The receiver operating characteristic (ROC) curves, area under the ROC curves (AUC), and Youden’s J point analysis were done online using Epitools-Epidemiological Calculators (epitools.ausvet.com.au/). Diagnostic test evaluation statistical tests for accuracies were done using MedCalc (www.medcalc.org/) free statistical calculator online. A two-tailed p-value < 0.05 was considered significant.
A total of 32 patients were screened for eligibility; one was excluded per the age criteria, and four patients' HFNC oxygen therapy was deemed ineffective within a few minutes of admission. Data from the rest of the 27 patients were included for final analysis. Three-fourths of the data were from male patients. The mean ± SD age of the patients was 53.93 ± 10.67 years. The clinicodemographic data are presented in Table 1.
Thirteen (48.15%) of the 27 patients' HFNC therapy failed, mostly within 6-24 hours (range 6-96 hours). In the univariate analysis, SOFA, blood urea nitrogen level at admission, ROX at six hours, and mROX at admission and six hours significantly differed among the success and failure cases (Table 2).
The mean values of the ROX and mROX at the baseline and six hours were higher for success cases than failure; the difference was significant except for ROX at baseline (Figure 1).
The sensitivity, specificity, accuracy, predictive values, and likelihood ratio analysis showed slightly better values for mROX at six hours. At six hours, the mean mROX value was 4.05, with the highest sensitivity of 87.5%, a positive likelihood ratio of 2.37, and a sensitivity plus specificity of 156%. However, the differences were insignificant statistically (Table 3).
Similarly, the area under the ROC curve was also highest for mROX at six hours, i.e., 0.81 (95% CI 0.636 - 0.985), while the ROX at six hours had an AUC of 0.717 (95%CI 0.512 - 0.92). The same for the ROX baseline was 0.712 (95% CI 0.512 - 0.911), while the mROX baseline was 0.747 (95% CI 0.558 - 0.937). The ROC curves for ROX and mROX at baseline and six hours are presented in Figure 2.
The Youden's J cut-off point for ROX and mROX at baseline was 4.59, 4.1, and at six hours was 4.47 and 3.34, respectively. The sensitivity and specificity at Youden's J cut-off points for the ROX and mROX and their efficacy at p=0.01 are presented in Table 4. While the Youden’s J cut-off point value for mROX at baseline (i.e., 4.1) had the highest specificity of 84.6%, the Youden’s J cut-off point value for mROX at six hours (i.e., 3.34) had the highest sensitivity of 85.7%, and sensitivity plus specificity of 162.6%.
In the present observational pilot study, the mROX showed slightly better but statistically insignificant predictive accuracy for HFNC failure than the ROX index of respective time points. These findings are in line with Roca et al.’s analysis, where they also found that the diagnostic accuracies of ROX index with SpO2/FiO2 and PaO2/FiO2 (mROX index) were statistically indifferent at two, six, and 12-hours . Unfortunately, the efficacy of both ROX and mROX was only fair to good. None of the indexes at baseline or six-hour could achieve a sensitivity and specificity near 90%. The maximum sensitivity was achieved by mROX at six hours (i.e., 87.55%) at a cut-off value of 4.05, but the specificity was only 63%. While a screening tool does not require high specificity, high sensitivity is desirable to detect most true positive cases. In that context, ROX at six hours and mROX at baseline and six hours appear to have the potential, as they all have a sensitivity of >70%. Nonetheless, for a screening test to be clinically helpful, both sensitivity and specificity are essential, and a criterion based on sensitivity + specificity of 1.5 and above is considered . The only screening test that qualified our pilot cohort's criteria was mROX at six hours. The sensitivity and specificity of the ROX index in our study were like the other contemporary studies and meta-analyses [13, 14].
The present study hypothesis was based on the clinicopathological features of COVID-19 pneumonia and acute respiratory distress syndrome (ARDS). As respiratory rate and oxygenation are altered in COVID-19 AHRF patients compared to non-COVID-19 AHRF patients, we assumed that the ROX and mROX index may have altered efficacy in predicting HFNC O2 therapy in COVID-19 patients. The probable explanation for failing mROX to find better predictors in the present study might be the need for higher FiO2 during HFNC therapy. These patients are mostly moderate-to-severely ill and usually do not require 100% FiO2. At the relatively lower FiO2, SpO2/FiO2, and PaO2/FiO2 correlate well even in COVID-19 patients . As the need for FiO2 increases toward 100% in patients receiving HFNC O2 therapy, the chances of HFNC O2 therapy failure also increase.
In their landmark study, Roca et al. reported the ROX index at 12 hours as having the best prediction accuracy (area under ROC 0.74 [95% confidence interval, 0.64-0.84]) with the best cut-off point for ROX index estimated to be 4.88 . Vega et al. validated the utility of the ROX index as a predictor of HFNC failure for COVID-19 patients  and reported similar results to Roca et al. The 12-hour ROX index was the best predictor of intubation with an AUC of 0.7916 [CI 95% 0.6905-0.8927], and the best threshold was 5.99 [Specificity 96%, Sensitivity 62%]. Even the multivariate analysis to determine the predictive factors for HFNC failure by Lun et al. found the ROX index at 12 hours as one of the significant entities . However, the time point for the best prediction is different. Ferrer et al. found the ROX index a good predictor in their observational study . They reported the ROX index at 24 hours as the best predictor of success (AUC 0.826, 95%CI 0.593-1.00) with a cut-off point of 5.35 (Sensitivity 0.91, Specificity 0.79, positive predictive value [PPV] 0.92, negative predictive value [NPP] 0.79). On the other hand, Suliman et al. , enrolling COVID-19 patients, found that ROX on day one was a significant predictor of intubation through regression analysis.
In their systematic review and meta-analysis, Prakash et al. assessed the ROX index as a predictor of HFNC failure in COVID-19 patients with AHRF; eight retrospective studies (n=1301 patients) were considered for analysis . The meta-analysis yielded a summary area under the curve (sAUC) of 0.81 (95% CI, 0.77-0.84) with a sensitivity of 0.70 (95% CI, 0.59-0.80) and specificity of 0.79 (95% CI, 0.67-0.88) of ROX index for predicting HNFC failure. The positive and negative likelihood ratio was 3.0 (95% CI, 2.2-5.3) and 0.37 (95% CI, 0.28-0.50), respectively, and was strongly associated with good predictive accuracy (diagnostic odds ratio 9, 95% CI, 5-16). Similarly, the systematic review and meta-analysis by Zhou et al. for ROX index as a predictor of HFNC outcome in pneumonia patients with AHRF found good predictive performance for successful HFNC weaning in patients with an area under ROC of 0.81 (95% CI 0.77-0.84), a pooled sensitivity and specificity of 0.71 (95% CI 0.64-0.78) and 0.78 (95% CI 0.70-0.84), respectively . The analyses also suggested that the ROX index was a reliable predictor of HFNC success in patients with COVID-19 pneumonia. The AUC analysis of our cohort for both ROX and mROX index at admission and six hours was also in line with these studies.
Although most studies using ROX have stressed HFNC failure, Suliman et al. studied the ROX and mROX for predicting the intubation of COVID-19 patients . At the time of intubation, the ROX and mROX median (min-max) values were 3.88 (3.33-6.09) and 5 (3.14-5.52), respectively. In contrast to Suliman et al.’s study , 11 out of 13 failure patients were initially tried on Bilevel pressure support non-invasive ventilation before intubation and invasive mechanical ventilation. Nevertheless, all but one patient ultimately required IVM. The mROX and ROX values at six hours and 12 hours in our failure cohort were within the range of the study by Suliman et al. , which required intubation, indicating the efficiency of the indexes for not only predicting HFNC failure but also predicting intubation. Rochwerg et al. performed a systematic review and meta-analysis to evaluate the safety and efficacy of HFNC in patients with acute hypoxemic respiratory failure. Nine randomized controlled trials were included (n=2093 patients) and analyzed. They reported that HFNC usage reduced the need for invasive mechanical ventilation compared to conventional oxygen therapy (RR 0.85 95% CI 0.74-0.99 with low certainty) .
Further, the use of HFNC reduced the need for escalation of therapy, although it did not affect ICU and hospital length of stay. Notably, delaying intubation leads to increased mortality. In this context, using either the ROX or mROX indexes to predict failure and intubation might be an excellent clinical practice despite needing more sensitivity, specificity, and other limitations .
The other intriguing observational finding is the ROX or mROX values trend. Persistent lower values and decreasing trends over 12 hours or 24 hours were the characteristics of the cohort that failed the HFNC. The findings also corroborate with the data presented by Suliman et al.  where the cohort requiring intubation also showed a decreasing ROX value over 72 hours.
In the present study, mROX and ROX were almost similar, except for a marginally higher predictive accuracy of mROX at six hours. A recent study taking the readings obtained four hourlies over 24 hours and using the average value also showed that mROX might be better than ROX . However, our study is limited because it was an observational study conducted in a single center and a pilot study. The decision to convert artificial respiratory support from HFNC to NIV (bilevel positive airway pressure - BiPAP) or IMV was at the treating physician's discretion, and no written or strict protocol was followed. We intended to identify the better screener at the earliest and therefore stressed primarily within six hours of the HFNC application. Although the patients were followed until they were either discharged or in heavenly abode, we needed more data to analyze at delayed time points. Sedation is another factor that impacts the tolerance and respiratory rate during non-invasive artificial respiratory support. It bears importance in the context that the sedation requirements of COVID-19 patients are very high and variable . We did not monitor the sedation levels and drugs used for the same, which might have had a minor impact on our results.
Both ROX and mROX at baseline and six hours had fair-to-good sensitivity, specificity, positive and negative predictive values, and area under the ROC curve; the differences were statistically insignificant. The accuracies of the indices were better at six hours than the baseline. Although both the indices can be used, only mROX at six hours, with a mean value of 4.05 and a cut-off value of 3.34 had a sensitivity plus specificity at 1.56 and 1.63 (i.e., 156% and 163%), respectively, can be considered a clinically valuable screener. A falling trend or failure to improve the index values over time and progressively increasing need for FiO2 is a vital observation to predict progression towards HFNC oxygenation failure.