Preface ......................................................... V
Outline of Contents .......................................... XVII
Notation and Symbols .......................................... XXI
1. Introductory Measure Theory .................................. 1
1. Probability Theory: An Introduction .......................... 1
2. Basics from Measure Theory ................................... 2
2.1. Sets .................................................... 3
2.2. Collections of Sets ..................................... 5
2.3. Generators .............................................. 7
2.4. A Metatheorem and Some Consequences ..................... 9
3. The Probability Space ....................................... 10
3.1. Limits and Completeness ................................ 11
3.2. An Approximation Lemma ................................. 13
3.3. The Borel Sets on  .................................... 14
3.4. The Borel Sets on n ................................... 16
4. Independence; Conditional Probabilities ..................... 16
4.1. The Law of Total Probability; Bayes' Formula ........... 17
4.2. Independence of Collections of Events .................. 18
4.3. Pair-wise Independence ................................. 19
5. The Kolmogorov Zero-one Law ................................. 20
6. Problems .................................................... 22
2. Random Variables ............................................ 25
1. Definition and Basic Properties ............................. 25
1.1. Functions of Random Variables .......................... 28
2. Distributions ............................................... 30
2.1. Distribution Functions ................................. 30
2.2. Integration: A Preview ................................. 32
2.3. Decomposition of Distributions ......................... 36
2.4. Some Standard Discrete Distributions ................... 39
2.5. Some Standard Absolutely Continuous Distributions ...... 40
2.6. The Cantor Distribution ................................ 40
2.7. Two Perverse Examples .................................. 42
3. Random Vectors; Random Elements ............................. 43
3.1. Random Vectors ......................................... 43
3.2. Random Elements ........................................ 45
4. Expectation; Definitions and Basics ......................... 46
4.1. Definitions ............................................ 46
4.2. Basic Properties ....................................... 48
5. Expectation; Convergence .................................... 54
6. Indefinite Expectations ..................................... 58
7. A Change of Variables Formula ............................... 60
8. Moments, Mean, Variance ..................................... 62
9. Product Spaces; Fubini's Theorem ............................ 64
9.1. Finite-dimensional Product Measures .................... 64
9.2. Fubini's Theorem ....................................... 65
9.3. Partial Integration .................................... 66
9.4. The Convolution Formula ................................ 67
10.Independence ................................................ 68
10.1.Independence of Functions of Random Variables .......... 71
10.2.Independence of σ-Algebras ............................. 71
10.3.Pair-wise Independence ................................. 71
10.4.The Kolmogorov Zero-one Law Revisited .................. 72
11.The Cantor Distribution ..................................... 73
12.Tail Probabilities and Moments .............................. 74
13.Conditional Distributions ................................... 79
14.Distributions with Random Parameters ........................ 81
15.Sums of a Random Number of Random Variables ................. 83
15.1.Applications ........................................... 85
16.Random Walks; Renewal Theory ................................ 88
16.1.Random Walks ........................................... 88
16.2.Renewal Theory ......................................... 89
16.3.Renewal Theory for Random Walks ........................ 90
16.4.The Likelihood Ratio Test .............................. 91
16.5.Sequential Analysis .................................... 91
16.6.Replacement Based on Age ............................... 92
17.Extremes; Records ........................................... 93
17.1.Extremes ............................................... 93
17.2.Records ................................................ 93
18.Borel-Cantelli Lemmas ....................................... 96
18.1.The Borel-Cantelli Lemmas 1 and 2 ...................... 96
18.2.Some (Very) Elementary Examples ........................ 98
18.3.Records ............................................... 101
18.4.Recurrence and Transience of Simple Random Walks ...... 102
18.5.∑∞n=1 = ∞ and P(An i.o.) = 0 ........................... 104
18.6.Pair-wise Independence ................................ 104
18.7.Generalizations Without Independence .................. 105
18.8.Extremes .............................................. 107
18.9.Further Generalizations ............................... 109
19.A Convolution Table ........................................ 113
20.Problems ................................................... 114
3.Inequalities ................................................ 119
1. Tail Probabilities Estimated via Moments ................... 119
2. Moment Inequalities ........................................ 127
3. Covariance; Correlation .................................... 130
4. Interlude on Lp-spaces ..................................... 131
5. Convexity .................................................. 132
6. Symmetrization ............................................. 133
7. Probability Inequalities for Maxima ........................ 138
8. The Marcinkiewics-Zygmund Inequalities ..................... 146
9. Rosenthal's Inequality ..................................... 151
10.Problems ................................................... 153
4. Characteristic Functions ................................... 157
1. Definition and Basics ...................................... 157
1.1. Uniqueness; Inversion ................................. 159
1.2. Multiplication ........................................ 164
1.3. Some Further Results .................................. 165
2. Some Special Examples ...................................... 166
2.1. The Cantor Distribution ............................... 166
2.2. The Convolution Table Revisited ....................... 168
2.3. The Cauchy Distribution ............................... 170
2.4. Symmetric Stable Distributions ........................ 171
2.5. Parseval's Relation ................................... 172
3. Two Surprises .............................................. 173
4. Refinements ................................................ 175
5. Characteristic Functions of Random Vectors ................. 180
5.1. The Multivariate Normal Distribution .................. 180
5.2. The Mean and the Sample Variance Are Independent ...... 183
6. The Cumulant Generating Function ........................... 184
7. The Probability Generating Function ........................ 186
7.1. Random Vectors ........................................ 188
8. The Moment Generating Function ............................. 189
8.1. Random Vectors ........................................ 191
8.2. Two Boundary Cases .................................... 191
9. Sums of a Random Number of Random Variables ................ 192
10.The Moment Problem ......................................... 194
10.1 The Moment Problem for Random Sums .................... 196
11.Problems ................................................... 197
5. Convergence ................................................ 201
1. Definitions ................................................ 202
1.1. Continuity Points and Continuity Sets ................. 203
1.2. Measurability ......................................... 205
1.3. Some Examples ......................................... 206
3. Relations Between Convergence Concepts ..................... 209
3.1. Converses ............................................. 212
4. Uniform Integrability ...................................... 214
5. Convergence of Moments ..................................... 218
5.1. Almost Sure Convergence ............................... 218
5.2. Convergence in Probability ............................ 220
5.3. Convergence in Distribution ........................... 222
6. Distributional Convergence Revisited ....................... 225
6.1. Scheffe's Lemma ....................................... 226
7. A Subsequence Principle .................................... 229
8. Vague Convergence; Helly's Theorem ......................... 230
8.1. Vague Convergence ..................................... 231
8.2. Helly's Selection Principle ........................... 232
8.3. Vague Convergence and Tightness ....................... 234
8.4. The Method of Moments ................................. 237
9. Continuity Theorems ........................................ 238
9.1. The Characteristic Function ........................... 238
9.2. The Cumulant Generating Function ...................... 240
9.3. The (Probability) Generating Function ................. 241
9.4. The Moment Generating Function ........................ 242
10.Convergence of Functions of Random Variables ............... 243
10.1.The Continuous Mapping Theorem ........................ 245
11.Convergence of Sums of Sequences ........................... 247
11.1.Applications .......................................... 249
11.2.Converses ............................................. 252
11.3.Symmetrization and Desymmetrization ................... 255
12.Cauchy Convergence ......................................... 256
13.Skorohod's Representation Theorem .......................... 258
14.Problems ................................................... 260
6. The Law of Large Numbers ................................... 265
1. Preliminaries .............................................. 266
1.1. Convergence Equivalence ............................... 266
1.2. Distributional Equivalence ............................ 267
1.3. Sums and Maxima ....................................... 268
1.4. Moments and Tails ..................................... 268
2. A Weak Law for Partial Maxima .............................. 269
3. The Weak Law of Large Numbers .............................. 270
3.1. Two Applications ...................................... 276
4. A Weak Law Without Finite Mean ............................. 278
4.1. The St.Petersburg Game ................................ 283
5. Convergence of Series ...................................... 284
5.1. The Kolmogorov Convergence Criterion .................. 286
5.2. A Preliminary Strong Law .............................. 288
5.3. The Kolmogorov Three-series Theorem ................... 289
5.4. Levy's Theorem on the Convergence of Series ........... 292
6. The Strong Law of Large Numbers ............................ 294
7. The Marcinkiewicz-Zygmund Strong Law ....................... 298
8. Randomly Indexed Sequences ................................. 301
9. Applications ............................................... 305
9.1. Normal Numbers ........................................ 305
9.2. The Glivenko-Cantelli Theorem ......................... 306
9.3. Renewal Theory for Random Walks ....................... 306
9.4. Records ............................................... 307
10.Uniform Integrabihty; Moment Convergence ................... 309
11.Complete Convergence ....................................... 311
11.1.The Hsu-Robbins-Erdos Strong Law ...................... 312
11.2.Complete Convergence and the Strong Law ............... 314
12.Some Additional Results and Remarks ........................ 315
12.1.Convergence Rates ..................................... 315
12.2.Counting Variables .................................... 320
12.3.The Case r = p Revisited .............................. 321
12.4.Random Indices ........................................ 322
13.Problems ................................................... 323
7. The Central Limit Theorem .................................. 329
1. The i.i.d. Case ............................................ 330
2. The Lindeberg-Levy-Feller Theorem .......................... 330
2.1. Lyapounov's Condition ................................. 339
2.2. Remarks and Complements ............................... 340
2.3. Pair-wise Independence ................................ 343
2.4. The Central Limit Theorem for Arrays .................. 344
3. Anscombe's Theorem ......................................... 345
4. Applications ............................................... 348
4.1. The Delta Method ...................................... 349
4.2. Stirling's Formula .................................... 350
4.3. Renewal Theory for Random Walks ....................... 350
4.4. Records ............................................... 351
5. Uniform Integrabihty; Moment Convergence ................... 352
6. Remainder Term Estimates ................................... 354
6.1. The Berry-Esseen Theorem .............................. 355
6.2. Proof of the Berry-Esseen Theorem 6.2 ................. 357
7. Some Additional Results and Remarks ........................ 362
7.1. Rates of Rates ........................................ 362
7.2. Non-uniform Estimates ................................. 363
7.3. Renewal Theory ........................................ 364
7.4. Records ............................................... 364
7.5. Local Limit Theorems .................................. 365
7.6. Large Deviations ...................................... 365
7.7. Convergence Rates ..................................... 366
7.8. Precise Asymptotics ................................... 371
7.9. A Short Outlook on Extensions ......................... 374
8. Problems ................................................... 376
8. The Law of the Iterated Logarithm .......................... 383
1. The Kolmogorov and Hartman-Wintner LILs .................... 384
1.1. Outline of Proof ...................................... 385
2. Exponential Bounds ......................................... 385
3. Proof of the Hartman-Wintner Theorem ....................... 387
4. Proof of the Converse ...................................... 396
5. The LIL for Subsequences ................................... 398
5.1. A Borel-Cantelli Sum for Subsequences ................. 401
5.2. Proof of Theorem 5.2 .................................. 402
5.3. Examples .............................................. 404
6. Cluster Sets ............................................... 404
6.1. Proofs ................................................ 406
7. Some Additional Results and Remarks ........................ 412
7.1. Hartman-Wintner via Berry-Esseen ...................... 412
7.2. Examples Not Covered by Theorems 5.2 and 5.1 .......... 413
7.3. Further Remarks on Sparse Subsequences ................ 414
7.4. An Anscombe LIL ....................................... 416
7.5. Renewal Theory for Random Walks ....................... 417
7.6. Record Times .......................................... 417
7.7. Convergence Rates ..................................... 418
7.8. Precise Asymptotics ................................... 419
7.9. The Other LIL ......................................... 419
8. Problems ................................................... 420
9. Limit Theorems; Extensions and Generalizations ............. 423
1. Stable Distributions ....................................... 424
2. The Convergence to Types Theorem ........................... 427
3. Domains of Attraction ...................................... 430
3.1. Sketch of Preliminary Steps ........................... 433
3.2. Proof of Theorems 3.2 and 3.3 ......................... 435
3.3. Two Examples .......................................... 438
3.4. Two Variations ........................................ 439
3.5. Additional Results .................................... 440
4. Infinitely Divisible Distributions ......................... 442
5. Sums of Dependent Random Variables ......................... 448
6. Convergence of Extremes .................................... 451
6.1. Max-stable and Extremal Distributions ................. 451
6.2. Domains of Attraction ................................. 456
6.3. Record Values ......................................... 457
7. The Stein-Chen Method ...................................... 459
8. Problems ................................................... 464
10.Martingales ................................................ 467
1. Conditional Expectation .................................... 468
1.1. Properties of Conditional Expectation ................. 471
1.2. Smoothing ............................................. 474
1.3. The Rao-Blackwell Theorem ............................. 475
1.4. Conditional Moment Inequalities ....................... 476
2. Martingale Definitions ..................................... 477
2.1. The Defining Relation ................................. 479
2.2. Two Equivalent Definitions ............................ 479
3. Examples ................................................... 481
4. Orthogonality .............................................. 487
5. Decompositions ............................................. 489
6. Stopping Times ............................................. 491
7. Doob's Optional Sampling Theorem ........................... 495
8. Joining and Stopping Martingales ........................... 497
9. Martingale Inequalities .................................... 501
10.Convergence ................................................ 508
10.1.Garsia's Proof ........................................ 508
10.2.The Upcrossings Proof ................................. 511
10.3.Some Remarks on Additional Proofs ..................... 513
10.4.Some Questions ........................................ 514
10.5.A Non-convergent Martingale ........................... 515
10.6.A Central Limit Theorem? .............................. 515
11.The Martingale {E(Z | Fn)} ................................. 515
12.Regular Martingales and Submartingales ..................... 516
12.1.A Main Martingale Theorem ............................. 517
12.2.A Main Submartingale Theorem .......................... 518
12.3.Two Non-regular Martingales ........................... 519
12.4.Regular Martingales Revisited ......................... 519
13.The Kolmogorov Zero-one Law ................................ 520
14.Stopped Random Walks ....................................... 521
14.1.Finiteness of Moments ................................. 521
14.2.The Wald Equations .................................... 522
14.3.Tossing a Coin Until Success .......................... 525
14.4.The Gambler's Ruin Problem ............................ 526
14.5 A Converse ............................................ 529
15.Regularity ................................................. 531
15.1.First Passage Times for Random Walks .................. 535
15.2.Complements ........................................... 537
15.3.The Wald Fundamental Identity ......................... 538
16.Reversed Martingales and Submartingales .................... 541
16.1.The Law of Large Numbers .............................. 544
16.2.U-statistics .......................................... 547
17.Problems ................................................... 548
11.Some Useful Mathematics .................................... 555
1. Taylor Expansion ........................................... 555
2. Mill's Ratio ............................................... 558
3. Sums and Integrals ......................................... 559
4. Sums and Products .......................................... 560
5. Convexity; Clarkson's Inequality ........................... 561
6. Convergence of (Weighted) Averages ......................... 564
7. Regularly and Slowly Varying Functions ..................... 566
8. Cauchy's Functional Equation ............................... 568
9. Functions and Dense Sets ................................... 570
References .................................................... 577
Index ......................................................... 589
| 
