Estimation of responder cell frequency and binomial three-level nonlinear mixed effects model in limiting dilution assays

Responder cell frequencies (RCF), which describe boosted immune responses in herpes zoster (HZ) prevention studies, are estimated by using limiting dilution assays (LDA). The theoretical linearity assumption between the logarithm of the proportion of nonresponding wells (s) and the cell concentration (N) (or dilution level) in LDA, based on the single-hit Poisson model, is often violated with observed data resulting in biased estimates of RCF. In this dissertation, the Poisson assumption is modified by applying a mixture of Poisson and gamma distributions, resulting in a negative binomial assumption, which presents a better fit between s and N. In LDA for HZ prevention studies, binary responses (responding or no responding wells) are measured repeatedly at different cell concentrations and over time. To account for the correlation between responses to varying dilution levels from individuals, and the correlation between repeated assays of individuals over time simultaneously, a binomial three-level nonlinear mixed effects model is proposed. In this model, the binomial proportion has a nonlinear functional relationship with covariates, cell concentrations and times, which is based on the negative binomial assumption. For parameter estimation, a maximum likelihood method is applied. An adaptive Gaussian quadrature rule is utilized to directly approximate the likelihood integrated over the random effects. There is a lack of non-Gaussian multilevel nonlinear mixed effects software, which can execute such a complicated fit. In this dissertation an algorithm for the three-level nonlinear mixed effects model, which can be inserted into the code in the SAS procedure NLMIXED, is suggested.