Markov Processes and Applications: Algorithms, Networks, Genome and Finance Book

Preface xi

1 Simulations and the Monte Carlo method 1

1.1 Description of the method 2

1.2 Convergence theorems 3

1.3 Simulation of random variables 5

1.4 Variance reduction techniques 9

1.5 Exercises 13

2 Markov chains 17

2.1 Definitions and elementary properties 17

2.2 Examples 21

2.2.1 Random walk in E = Zd 21

2.2.2 Bienaymé-Galton-Watson process 21

2.2.3 A discrete time queue 22

2.3 Strong Markov property 22

2.4 Recurrent and transient states 24

2.5 The irreducible and recurrent case 27

2.6 The aperiodic case 32

2.7 Reversible Markov chain 38

2.8 Rate of convergence to equilibrium 39

2.8.1 The reversible finite state case 39

2.8.2 The general case 42

2.9 Statistics of Markov chains 42

2.10 Exercises 43

3 Stochastic algorithms 57

3.1 Markov chain Monte Carlo 57

3.1.1 An application 59

3.1.2 The Ising model 61

3.1.3 Bayesian analysis of images 63

3.1.4 Heated chains 64

3.2 Simulation of the invariant probability 64

3.2.1 Perfect simulation 65

3.2.2 Coupling from the past 68

3.3 Rate of convergence towards the invariant probability 70

3.4 Simulated annealing 73

3.5 Exercises 75

4 Markov chains and the genome 77

4.1 Reading DNA 77

4.1.1 CpG islands 78

4.1.2 Detection of the genes in a prokaryotic genome 79

4.2 The i.i.d. model 79

4.3 The Markov model 80

4.3.1 Application to CpG islands 80

4.3.2 Search for genes in a prokaryotic genome 81

4.3.3 Statistics of Markov chains Mk 82

4.3.4 Phased Markov chains 82

4.3.5 Locally homogeneous Markov chains 82

4.4 Hidden Markov models 84

4.4.1 Computation of the likelihood 85

4.4.2 The Viterbi algorithm 86

4.4.3 Parameter estimation 87

4.5 Hiddensemi-Markov model 92

4.5.1 Limitations of the hidden Markov model 92

4.5.2 What is a semi-Markov chain? 92

4.5.3 The hidden semi-Markov model 93

4.5.4 The semi-Markov Viterbi algorithm 94

4.5.5 Search for genes in a prokaryotic genome 95

4.6 Alignment of two sequences 97

4.6.1 The Needleman - Wunsch algorithm 98

4.6.2 Hidden Markov model alignment algorithm 99

4.6.3 A posteriori probability distribution of the alignment 102

4.6.4 A posteriori probability of a given match 104

4.7 A multiple alignment algorithm 105

4.8 Exercises 107

5 Control and filtering of Markov chains 109

5.1 Deterministic optimal control 109

5.2 Control of Markov chains 111

5.3 Linear quadratic optimal control 111

5.4 Filtering of Markov chains 113

5.5 The Kalman - Bucy filter 115

5.5.1 Motivation 115

5.5.2 Solution of the filtering problem 116

5.6 Linear-quadratic control with partial observation 120

5.7 Exercises 121

6 The Poisson process 123

6.1 Point processes and counting processes 123

6.2 The Poisson process 124

6.3 The Markov property 127

6.4 Large time behaviour 130

6.5 Exercises 132

7 Jump Markov processes 135

7.1 General facts 135

7.2 Infinitesimal generator 139

7.3 The strong Markov property 142

7.4 Embedded Markov chain 144

7.5 Recurrent and transient states 147

7.6 The irreducible recurrent case 148

7.7 Reversibility 153

7.8 Markov models of evolution and phylogeny 154

7.8.1 Models of evolution 156

7.8.2 Likelihood methods in phylogeny 160

7.8.3 The Bayesian approach to phylogeny 163

7.9 Application to discretized partial differential equations 166

7.10 Simulated annealing 167

7.11 Exercises 173

8 Queues and networks 179

8.1 M/M/ 1 queue 179

8.2 M/M/ 1/ K queue 182

8.3 M/M/ s queue 182

8.4 M/M/s/s queue 184

8.5 Repair shop 185

8.6 Queues in series 185

8.7 M/G/&infinity; queue 186

8.8 M/G/ 1 queue 187

8.8.1 An embedded chain 187

8.8.2 The positive recurrent case 188

8.9 Open Jackson network 190

8.10 Closed Jackson network 194

8.11 Telephone network 196

8.12 Kelly networks 199

8.12.1 Single queue 202

8.12.2 Multi-class network 203

8.13 Exercises 203

9 Introduction to mathematical finance 205

9.1 Fundamental concepts 205

9.1.1 Option 206

9.1.2 Arbitrage 206

9.1.3 Viable and complete markets 207

9.2 European options in the discrete model 208

9.2.1 The model 208

9.2.2 Admissible strategy 208

9.2.3 Martingales 210

9.2.4 Viable and complete market 211

9.2.5 Call and put pricing 213

9.2.6 The Black-Scholes formula 214

9.3 The Black-Scholes model and formula 216

9.3.1 Introduction to stochastic calculus 217

9.3.2 Stochastic differential equations 223

9.3.3 The Feynman-Kac formula 225

9.3.4 The Black-Scholes partial differential equation 225

9.3.5 The Black-Scholes formula (2) 228

9.3.6 Generalization of the Black-Scholes model 228

9.3.7 The Black-Scholes formula (3) 229

9.3.8 Girsanov's theorem 232

9.3.9 Markov property and partial differential equation 233

9.3.10 Contingent claim on several underlying stocks 235

9.3.11 Viability and completeness 237

9.3.12 Remarks on effective computation 238

9.3.13 Historical and implicit volatility 239

9.4 American options in the discrete model 239

9.4.1 Snell envelope 240

9.4.2 Doob's decomposition 242

9.4.3 Snell envelope and Markov chain 244

9.4.4 Back to American options 244

9.4.5 American and European options 245

9.4.6 American options and Markov model 245

9.5 American options in the Black-Scholes model 246

9.6 Interest rate and bonds 247

9.6.1 Future interest rate 247

9.6.2 Future interest rate and bonds 248

9.6.3 Option based on a bond 250

9.6.4 An interest rate model 251

9.7 Exercises 252

10 Solutions to selected exercises 257

10.1 Chapter 1 257

10.2 Chapter 2 262

10.3 Chapter 3 275

10.4 Chapter 4 277

10.5 Chapter 5 278

10.6 Chapter 6 279

10.7 Chapter 7 282

10.8 Chapter 8 289

10.9 Chapter 9 291

Reference 295

Index 297