Assessing and Reporting Heterogeneity of Treatment Effect in Randomized Clinical Trials

Specific Aim 1:
Refining the analytic algorithm We will first pilot approaches that employ both internal and external models and perform simulations to test the vulnerability of internal models to over-fitting, and exaggerated HTE from over-fitting. The goal of this pilot phase is to better appreciate the advantages and disadvantages of internal vs external models in actual practice and to develop a standardized approach for conservative, internal models that avoid over-fitting. This 5-month phase will also permit us to develop efficient programming processes for database management and analyses, refine other methodological aspects of the analysis and refine the presentation of the results.
Specific Aims 2 and 3

Based on our experience and the results of specific aim 1 (and incorporating stakeholder input), we will have: 1) fully specified the analytic algorithm for the standardized analysis of outcome risk and risk-based HTE using internal models; 2) fully determine the risk variables for each trial category; 3) refined methods for presenting the results; 4) fully determined the complete set of trials to be analyzed. We will then be prepared to apply the algorithm to trials in a “high throughput” fashion. We conservatively estimate we will perform 2 such analyses per month and anticipate analyzing approximately 30 additional trials in this 15 month phase. In addition to generating the metrics described in Task 2 and aggregated across trials in Task 4, results will be clinically interpreted on a trial-by-trial basis. This will be performed by comparing analysis results with the original report, with particular attention to the overall results of the trial, any subgroup analysis that was reported and by reviewing the discussion. The purpose of this is to identify those trials in which performance of a risk-based analysis may have provided additional clinically important information that was not contained in the original report. This preliminary qualitative assessment will serve as a basis for future research. This analysis leverages the ever-growing availability of publically available databases, especially those available through the NHLBI website and NIDDK repository, as well as other databases available to the applicants (particularly PMR, who has more than two dozen large cardiovascular prevention trials that he has analyzed for other pooled analyses.
Based on these analyses, we will describe results across trials, including: 1. The distribution of EQRRs across trials. It has previously been suggested that EQRRs in the range of 5 to 20 are typical. This will represent the first attempt to quantify the degree of heterogeneity in a set of trials in any systematic fashion. In the absence of HTE on the relative scale, EQRR describes heterogeneity of ARR and NNT across quartiles. 2. The distribution of MMRR. This will provide the first systematic description of the relationship between the average risk (reflected in the summary results of clinical trials) and the median risk. MMRRs that substantially deviate from 1 indicate summary results may not reflect expected risks and benefits for the most “typical” patients. 3. Risk-based HTE. We will report the proportion of trials with statistically significant risk-based HTE in relative effects. 4. The distribution of extreme quartile absolute risk reduction (ARR/NNT). As this information is redundant with EQRR in th   e absence of HTE on the relative scale, we will summarize this information in trials where HTE is detected.

Note that no attempt will be made to formally meta-analyze results across trials (since averaging across these trials is not meaningful). Rather, we will report the distributions of their point estimates (median, range, inter-quartile values). In addition to the above metrics, we will summarize the results of our qualitative assessment comparing the results of our analyses with the original study reports. We will also test a limited number of exploratory hypotheses. These include examining whether the degree of heterogeneity (as reflected in the EQRR) and skewness (MMRR) is sensitive to various trial characteristics (such as the overall outcome rate seen in the trial). We hypothesize that trials with lower outcome rates (e.g. less than 20%) are more likely to yield high EQRRs and more skewed distributions. This may indicate that risk modelling approaches are especially important when outcome rates are lower. Interpretation of results includes both trial-by-trial interpretation and interpretation of the aggregated results and experience. In particular, we seek to understand both the value and the limitations of the approach. The next steps depend largely on the results of the research, but will include refining the framework initially described in Trials and dissemination activities. The process and assets for these activities are discussed in the section on stakeholders.

For each GSK trial which we would like to include in our project (described above in section A4), treatment effect may very well be sensitive to baseline risk characteristics of the patients enrolled. Furthermore, the outcomes for each of the trials we are requesting from GSK (also detailed above in Section A5) have known predictors of risk. We searched current literature and found published predictive risk models that can be used to assess heterogeneity of treatment effect unique to each GSK clinical trial. Listed below are the published predictive models we plan to use to assess treatment effect heterogeneity in the requested GSK trial.

GSK Study ID: AR1108888 Study title: FondaparinUx Trial with Unfractionated Heparin (UFH) during Revascularization in Acute Coronary Syndromes Published model citation: Piper WD, Malenka DJ, Ryan TJ Jr, Shubrooks SJ Jr, O'Connor GT, Robb JF, Farrell KL, Corliss MS, Hearne MJ, Kellett MA Jr, Watkins MW, Bradley WA, Hettleman BD, Silver TM, McGrath PD, O'Mears JR, Wennberg DE; Northern New England Cardiovascular Disease Study Group. Predicting vascular complications in percutaneous coronary interventions. Am Heart J 2003;145(6):1022-9. PMID: 12796758