Detection of uninformative clinics during the trial of a new drug using statistical and machine learning techniques

Proposal
1323

Title of Proposed Research

Lead Researcher

Praveen Deorani

Affiliation

KKT Technology pte. Ltd.

Funding Source

Commercial organization: KKT Technology Pte. Ltd.

Potential Conflicts of Interest

Kevin Craig is a full-time employee and share-holder of Covance Clinical Development Services, a division of Laboratory Corporation Inc.

Data Sharing Agreement Date

21 October 2015

Lay Summary

Major Depressive Disorder (MDD) affects 350 million people worldwide and is now the world's second leading cause of disability [1]. Remission rates with standard therapies are 30-50% [2]. Despite the significant unmet need, no treatments with new mechanisms of action have been approved this century. Over the last ten years the probability of success for new MDD drugs entering human testing has been 7.2% [3]. Historically, for treatments that are known to be effective, up to 50% of clinical trials fail [4]. A key determinant of this failure is the high variability in subjective clinical ratings and large placebo response rates.
Our hypothesis is that poor quality endpoint data at a site level differs systematically from the general trial data, for example in variability, correlation between assessments or implausible placebo response trajectories generated in a given center. These systematic differences would allow detection of sites with poor quality data during the conduct of the trial. The aim of this study would be to develop a quantitative methodology to detect sites contributing poor quality data.
Blinded data from previous MDD trial will be divided into a training set and a test set to assess whether outlier sites can be detected and to determine the accuracy of such predictions. Our aim is to classify each recruitment center on an ongoing basis during patient accrual as informative or non-informative.
The detection of uninformative centers at an early stage could lead to more efficient drug testing as the noise level in the data due to placebo response could be controlled. This would mean more reliable results in a shorter time-frame, and with a smaller number of subjects. It is thus of great significance to find ways to recognize clinical centers which have a high probability of providing inaccurate data about the effect of drug and placebo.

References
[1] Ferrari et al. (2013) Burden of Depressive Disorders by Country, Sex, Age, and Year: Findings from the Global Burden of Disease Study 2010. PLoS Med 10(11): e1001547. doi:10.1371/journal.pmed.1001547
[2] Sinyor M et al. (2010). The Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Trial. Canadian J Psych 55 (3): 126-135.
[3] Hay M et al. (2014) Clinical development success rates for investigational drugs. Nature Biotech. 32(1) p40-51.
[4] Turner EH et al. (2008) Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 17;358(3):252-60.

Study Data Provided

[{ "PostingID": 1949, "Title": "LILLY-F1J-MC-HMAI", "Description": "A Double-Blind, Placebo- and Clomipramine-Controlled Study of Duloxetine in Patients with Major
Depression" },{ "PostingID": 1950, "Title": "LILLY-F1J-MC-HMAQ(A)", "Description": "Duloxetine Versus Placebo in the
Treatment of Major Depression" },{ "PostingID": 1951, "Title": "LILLY-F1J-MC-HMAQ(B)", "Description": "Duloxetine Versus Placebo in the
Treatment of Major Depression" },{ "PostingID": 1956, "Title": "LILLY-F1J-US-HMFS", "Description": "Duloxetine Versus Placebo in Patients With Major Depressive Disorder (MDD): Assessment of Energy and Vitality in MDD" },{ "PostingID": 2086, "Title": "LILLY-F1J-MC-HMAH", "Description": "Duloxetine 20/30 mg vs. Placebo in Major Depression" },{ "PostingID": 3090, "Title": "LILLY-F1J-US-HMGR", "Description": "A Phase 4, 8-week, Double-blind, Randomized, Placebo-controlled Study Evaluating the Efficacy of Duloxetine 60 mg Once Daily in Outpatients with Major Depressive Disorder and Associated Painful Physical Symptoms" },{ "PostingID": 3092, "Title": "LILLY-F1J-US-HMGU", "Description": "Duloxetine Versus Placebo in the Acute Treatment of Patients With Major Depressive Disorder and Associated Painful Physical Symptoms" },{ "PostingID": 3505, "Title": "LILLY-F1J-MC-HMBV", "Description": "Duloxetine Versus Placebo in the Treatment of Elderly Patients With Major Depressive Disorder

Medicine: Duloxetine hydrochloride, Condition: Major Depressive Disorder, Phase: 3, Clinical Study ID: F1J-MC-HMBV , Sponsor: Lilly." },{ "PostingID": 3506, "Title": "LILLY-F1J-US-HMCB", "Description": "Duloxetine Once-Daily Dosing Versus Placebo in Patients With Major Depression and Pain

Medicine: Duloxetine hydrochloride, Condition: Major Depressive Disorder, Phase: 3, Clinical Study ID: F1J-US-HMCB , Sponsor: Lilly." }]

Statistical Analysis Plan

Signal detection
A multivariate logistic analysis to test statistical relationship between two placebo predictor variables (HAMD-17 score at baseline (P) and HAMD-17 score at the end of the study (F)) and the probability of detecting a signal of a clinically relevant treatment effect (the difference between end-of-study HAMD-17 scores of active treatment and placebo).
The posterior probability of the difference (P - F) will be used for ranking the centers according to their performance within each individual trial.
A probabilistic threshold derived from an ROC curve analysis to classify clinical centers as informative or noninformative.

Non linear longitudinal modeling
Estimating treatment effect by associating a weighting factor to the data collected at clinical centers. The weight will be defined by the posterior probability of detecting a clinically relevant difference between active treatment and placebo at that center.

Dealing with missing data
Different strategies such as “Missing Not At Random”, “Missing At Random” And “Missing Completely At Random”) will be employed. The best strategy will be chosen based on statistical measures such as p-values.

Detecting outliers
Details - “It is not possible to detect an outlier without defining what is normal”. The normal may be defined as 11 < final HAMD < 20. There can be other information about what is normal. For example, we know older people should recover slower than younger people. If a clinic shows otherwise results, its data may be questionable. We can assign probabilities based on few more tests like this. If the data fails all the tests, or fails half of them or passes all of them etc.

Machine learning
Details - We can use standard machine learning techniques to identify and learn patterns in data which determine whether the data is informative or not. We can use all the biometric and demographic features of the patients and along with the HAMD data for this
We can also try to classify each centre directly. We can plot the HAMD scores of all patients for each centre. Uninformative and informative centres may exhibit different distributions. Then, given a new centre, we can try to compare its HAMD score distribution against uninformative and informative centres using formulas like KL divergence. At last, assign the centre as informative if it is more similar to informative centres.
Since different data requested are about same drug, all the data can be pooled together for pattern finding

References:
[1] Clin Pharmacol Ther. 2008 Sep;84(3):378-84. doi: 10.1038. Model-based approach and signal detection theory to evaluate the performance of recruitment centers in clinical trials with antidepressant drugs. Merlo-Pich E(1), Gomeni R.
[2] Neuropsychopharmacology. 2015 Apr 21. doi: 10.1038. A Novel Methodology to Estimate the Treatment Effect in Presence of Highly Variable Placebo Response. Gomeni R(1), Goyal N(2), Bressolle F(1), Fava M(3).
[3] Eur J Pharm Sci. 2009 Jan 31;36(1):4-10. doi: 10.1016. Modelling placebo response in depression trials using a longitudinal model with informative dropout. Gomeni R(1), Lavergne A, Merlo-Pich E.

Publication Citation

Craig, Kevin & Liman, Christian & Yan, Yang & Goyal, Shubham & Roy, Nawal. (2017). More of what works: Detection of informative sites during the conduct of clinical trials using machine learning.
10.13140/RG.2.2.11323.62242.