Subgroup analysis is an important problem in clinical trials. For example, when a new treatment is approved for use, there may be concerns that the efficacy is driven by extreme efficacy in a subgroup only. Various methods have been proposed for the examination of subgroup effects. In recent years, researchers often attempt to identify a potential subgroup in which the subjects will have an enhanced treatment effect. In this proposed dissertation, we assume that there exists two potential subgroups in which the subjects react differently to the treatment. We propose a logistic-normal mixture model where the group means as well as the mixing proportions may be covariate-dependent. Testing the existence of subgroups is critical in the mixture model, but requires non-standard statistical tests. We derive a test based on a small number of EM iterations towards the likelihood, and propose the bootstrap approximation for the critical values of the test. When subgroups exist, the mixture model helps us identify the factors that are associated with the group membership. We apply the proposed method to the Aids Clinical Trials Group 320 study, and demonstrate that the patients with higher values of basline CD4 or RNA tend to benefit significantly more by adding a protease inhibitor to two nucleoside analogues. We also extend our results to logistic-normal mixture models with unequal variance across subgroups and/or non-normal component distributions. For the generalizations, we propose similar EM tests and study their properties. For the unequal variance case, a penalty on the variance is added to the log likelihood which is to prevent under-estimation of the variances. For the cases with non-normal components, with certain conditions, constant proportions could be added to the starting value set.
