Markov Chains

Abstract ix

Preface xi

Acknowledgments xiii

About the CompanionWebsite xv

1 Historical Notes 1

1.1 Introduction 1

1.2 On theWings of Dependent Variables 2

1.3 From Bernoulli to Markov 5

2 FromObservation to Simulation 9

2.1 Introduction 9

2.2 Stochastic Matrices 9

2.3 Transition Probabilities 11

2.4 The Simulation of a Two-State Markov Chain 14

3 Building the Stochastic Matrix 25

3.1 Introduction 25

3.2 Building a Stochastic Matrix from Events 25

3.3 Building a Stochastic Matrix from Percentages 32

4 Predictions Using Two-State Markov Chains 37

4.1 Introduction 37

4.2 Performing the Predictions by Using the Stochastic Matrix 37

4.3 The Steady State of a Markov Chain 46

4.4 The Long-Run Distribution of a Markov Chain 55

5 Predictions Using n-State Markov Chains 61

5.1 Introduction 61

5.2 Predictions by Using the Three-State Markov Chain 61

5.3 Predictions by Using the Four-State Markov Chain 71

5.4 Predictions by Using n-State Markov Chains 80

5.5 Markov Chain Modeling on Measurements 84

6 AbsorbingMarkov Chains 93

6.1 Introduction 93

6.2 The Absorbing State 93

7 The Average Time Spent in Each State 99

7.1 Introduction 99

7.2 The Proportion of Balls in the System 99

7.3 The Average Time Spent in A Particular State 100

7.4 Exemplification of the Average Time and Proportions 101

8 Discussions on Different Configurations of Chains 107

8.1 Introduction 107

8.2 Examples of Two-State Diagrams 113

8.3 Examples of Three-State Diagrams 115

8.4 Examples of Four-State Diagrams 117

8.5 Examples of State Diagrams Divided into Classes 123

8.6 Examples of State Diagrams with Absorbing States 127

8.7 The Gambler's Ruin 128

9 The Simulation of an n-State Markov Chain 131

9.1 Introduction 131

9.2 The Simulation of Behavior 131

9.3 Simulation of Different Chain Configurations 145

A Supporting Algorithms in PHP 165

B Supporting Algorithms in Javascript 193

C Syntax Equivalence between Languages 223

Glossary 225

References 227

Index 231