By Sarena L. McLean MSc. (Epidemiology & Biostatistics), Health Sciences Researcher & Member of Doctors for Covid Ethics (2021)
The study “A Critical Analysis of All-Cause Deaths during COVID-19 Vaccination in an Italian Province” by Alessandria et al. aimed to evaluate the impact of COVID-19 vaccination on all-cause mortality while correcting for immortal time bias (ITB). Conducted in the Piedmont region of Italy, the study utilized a retrospective cohort design using a data set from the Italian National Healthcare System, residents aged 10 and older, from January 1, 2021, to April 30, 2022 (Alessandria et al., p. 2). The authors aligned follow-up periods to ensure comparability between vaccinated and unvaccinated groups, dividing the cohort into groups based on vaccination status: unvaccinated, one dose, two doses, and three or four doses.
The study found an increased risk of death associated with receiving one or two doses of the vaccine, with less clear results for three or four doses. Covariates analyzed included gender, age, hypertension, diabetes, chronic obstructive pulmonary disease (COPD), cardiovascular disease, kidney diseases, cancer, and SARS-CoV-2 infection status. The authors used Cox proportional hazards models to estimate hazard ratios for all-cause mortality and employed Restricted Mean Survival Time (RMST) and Restricted Mean Time Lost (RMTL) for additional measures of survival and life expectancy (Alessandria et al., p. 4).
1. Strengths of the Study
The study used an innovative ITB correction, designed for accurate vaccination impact assessment. Additionally, the use of a large dataset from the Italian National Healthcare System provided a comprehensive dataset for analysis. Importantly the authors chose to complete an all-cause mortality analysis versus examining COVID-19 death statistics, which are rife with misclassification errors. The all-cause mortality is not only more robust but also captures the impact from other indirect effects of COVID-19 such as delayed medical treatments.
2. Areas for Improvement
2.1. Overall Presentation
The article requires clarity, and a few items are missing. For example, when beginning to read the abstract, the authors lay out the analysis problem in general, which is an excellent consideration. However, the authors would best help the reader orient themselves first by providing an overview or context. We know nothing about the work, so help us come along with you. There is no clear research question, and all reviewers want to see very clear hypotheses, objectives, and purpose. I struggled at first to simply understand the unstratified sample size. Do not make work for the reader.
There are many indications that the authors were innovative and thorough in their approach and of diligence to their process. Unfortunately, there were details and explanations missing, which created a lot of questions during the review. It is challenging to discern if there are methodological or statistical analysis issues or are simply missing pieces of information creating a conversation about items that are already addressed and valid. The comments below are intended to demonstrate what arose in this review.
2.2. Descriptive Statistics
More detailed descriptive statistics are needed to provide context for the study population and their baseline characteristics. These statistics are vital for both lay readers and academics to understand the foundation of the investigation and provide essential information to assess both the statistical approach and interpret the findings (Alessandria et al., p. 3). As a health science researcher, I am always looking for the measures of central tendency as I am interested in the distribution of these variables. Additionally, this item relates to the above point about unclear or missing details. The entire scientific method is predicated on being able to replicate a study. We are unable to do that in this instance simply because information is unclear. For example, I am assuming the authors decided not to provide details about vaccine manufacturers or types of vaccines for a substantial reason. However, there is no discussion about this. We are all challenged by word counts in publishing our work; yet some background and explanatory information is crucial.
2.3. Covariates/Confounders
Multicollinearity
The study includes several covariates that vary in their potential to cause multicollinearity issues. Age and sex are unlikely to present problems. Cancer and infection are only moderately correlated with other conditions, while hypertension and COPD, though more correlated with other comorbidities, are manageable within the model. However, cardiovascular disease is highly correlated with multiple conditions, notably hypertension and diabetes, which can complicate the analysis. The most significant concerns are diabetes and kidney disease, which are strongly correlated with each other and with other chronic conditions. This high correlation, known as multicollinearity, can severely impact the stability of the regression coefficients, making it difficult to determine the individual effect of each variable. For example, the study’s regression model might show a misleadingly high effect of vaccination on mortality if the true effect is confounded by the combined impact of diabetes and kidney disease, leading to unreliable results (Alessandria et al., p. 5). I would have preferred to read that the author’s had dealt with this issue even in prose or with a table.
Limited Covariates
The covariates were very limited in this study. This is generally problematic because it limits the ability to control for factors that can significantly influence health outcomes. From a social epidemiological perspective, missing covariates highly correlated with poor health outcomes, such as socio-economic status (SES), access to healthcare, and mental health status, can lead to residual confounding where we may miss the opportunity to see what is truly driving the relationship(s) resulting in poor outcomes (Alessandria et al., p. 4). Importantly, these same factors were directly impacted by COVID-19 response measures. For instance, individuals with lower SES often face greater barriers to healthcare access and may have challenging living conditions, which can increase their vulnerability and result in an increase in both the burden of disease and mortality. The study did include clinical covariates such as chronic diseases like hypertension, diabetes, COPD, cardiovascular disease, kidney diseases, cancer, and SARS-CoV-2 infection status, but broader social determinants of health were overlooked (Alessandria et al., p. 4). In reviewing the covariates, it’s important to consider their relationships to each other. For example, diabetes is a leading cause of kidney disease.
Statistical Methods
An early question about the model was the exclusion of deaths in the first 2-week time period after vaccination. For example, we know that cardiac death frequently occurs in that time frame; thus, the omission of this time period could impact the results.
It was unclear from the paper whether the authors examined and can assure readers that the data met the assumption criteria for the Cox model statistical analysis. The assumption required for a Cox model to be robust and valid is that the hazard rations for the covariates must be constant over time (Alessandria et al., p. 5-6). Confirming this would be very helpful for readers, as it would address concerns about the robustness of the statistical analysis. There is some information in the notes for Table 3, which do demonstrate the authors were addressing this issue along with a reference to Schoenfeld’s test (Alessandria et al., p. 4). Hoewever, in my view, it remains unclear if the assumption is sufficiently addressed, particularly with respect to the confounders. More comprehensive evidence should have been provided of how the proportional hazards assumption was tested and addressed for each covariate, especially confounders. Including detailed plots or test statistics and clearly explaining the stratification rationale would strengthen the study’s validity and address concerns effectively.
If the proportional hazards assumption is violated in a Cox regression model, it can cause several problems. The estimates of the hazard ratios may be biased, meaning that they do not accurately show the true relationship between the covariates and the risk of the event occurring. This can lead to incorrect statistical tests and confidence intervals, causing wrong conclusions about the effects of the covariates. In the context of the study by Alessandria et al., if this assumption were violated, it could compromise the validity of their findings about the impact of COVID-19 vaccination on all-cause mortality. The observed effects might be due to changes over time rather than a true relationship, leading to erroneous results.
While Kaplan-Meier survival curves and a simplified Cox model do not directly address immortal time bias as comprehensively as the authors’ approach, these methods can be adjusted to partially mitigate ITB. Incorporating time-dependent covariates can help align risk periods correctly. As Tabachnick and Fidell (2013) explain, using time-dependent covariates in Cox regression can effectively handle violations of the proportional hazards assumption, providing more reliable results. However, in this instance, clarity trumps accuracy.
The advanced and complex nature of the original analysis, combined with missing or unclear foundational information, such as the research question and descriptive statistics, makes it difficult to determine if the authors’ findings are robust. For example, when reviewing the statement “the SARS-CoV-2 infected population shows an HR of 0.58 (CI95 = 0.53–0.63) compared to the population without it” stood out as curious. The finding suggests that the SARS-CoV-2 infected group had as 42% lower hazard or risk compared to the non-infected group. I did not find any outcomes listed in the paper that suggested a lower risk made sense given the outcomes referred to death.
Finally, an important suggestion is to clearly state what you are not going to do. It clears up any confusion and lays a boundary for why we do not venture into areas that are beyond the scope and resources we have in our work. Overall, it takes away a critique of the reviewers.
3. Conclusion
There could be valuable findings in this study, but the presentation is hindered by scant foundational information and very complex data analysis. Simplifying the methods and ensuring clarity in objectives and descriptive statistics would greatly enhance the study’s reliability and accessibility. It would allow the reader to move beyond the initial information without so many questions. In this study, the investigators have been bold to embrace an untraditional statistical approach; thus, to truly add to the body of literature, the detail is very important. The peer review process was painful because it challenged my hopes. Years ago, I asked a beloved professor, Dr. Paul Maxim, what to do about bias; his answer was to get rid of it and if you can’t do that, declare it. By embracing a more rigorous review process, we uphold the integrity of science and contribute to more reliable and impactful research.
I hope that the authors will consider revising the study to address these concerns and recirculating their work. It is innovative yet complex, so clarity is key. I give credit to the authors for stepping out boldly to address gaps in the research.
References
Alessandria, M. et al. (2024) A Critical Analysis of All-Cause Deaths during {COVID}-19 Vaccination in an Italian Province. Microorganisms 12:1343 http://dx.doi.org/10.3390/microorganisms12071343
Tabachnick, B. G., & Fidell, L. S. (2019). Using Multivariate Statistics (7th ed.). Pearson.
Please consider helping!
Awake Freedom TV On Roku is Broken! 🙂
I could no longer maintain the costs on my own, so I moved the off of a dedicated server which was needed for the TV platform to work. On top of that, I want to build an app Awake Freedom TV app, for mobile phones. We need your help to first resurrect our TV channel, and then build the mobile app. In order to do that, I need a minimum of 800/month on subscription payments. Please help anyway you can with the form below.
Leave A Comment