## Abstract

We consider a large declining population of cells under an external selection pressure, modeled as a subcritical branching process. This population has genetic variation introduced at a low rate which leads to the production of exponentially expanding mutant populations, enabling population escape from extinction. Here we consider two possible settings for the effects of the mutation: Case (I) a deterministic mutational fitness advance and Case (II) a random mutational fitness advance. We first establish a functional central limit theorem for the renormalized and sped up version of the mutant cell process. We establish that in Case (I) the limiting process is a trivial constant stochastic process, while in Case (II) the limit process is a continuous Gaussian process for which we identify the covariance kernel. Lastly we apply the functional central limit theorem and some other auxiliary results to establish a central limit theorem (in the large initial population limit) of the first time at which the mutant cell population dominates the population. We find that the limiting distribution is Gaussian in both Cases (I) and (II), but a logarithmic correction is needed in the scaling for Case (II). This problem is motivated by the question of optimal timing for switching therapies to effectively control drug resistance in biomedical applications.

Original language | English (US) |
---|---|

Pages (from-to) | 3661-3697 |

Number of pages | 37 |

Journal | Stochastic Processes and their Applications |

Volume | 124 |

Issue number | 11 |

DOIs | |

State | Published - Nov 2014 |

### Bibliographical note

Funding Information:J.F is partially supported by the National Science Foundation ( NSF DMS-1224362 and NSF DMS-1349724 ). K.L is partially supported by National Science Foundation ( NSF DMS-1224362 and NSF CMMI-1362236 ). J.Z is partially supported by the National Science Foundation ( NSF DMS-12-24362) .

## Keywords

- Branching processes
- Escape from extinction
- Population dynamics
- Weak convergence