<HEAD><!-- I am an -*- HTML -*- file-->
<TITLE> PROBABILITY THEORY -- THE LOGIC OF SCIENCE </TITLE>
</HEAD>
<body bgcolor="#ffffff">
<H1>PROBABILITY THEORY: <br> THE LOGIC OF SCIENCE</H1>
by <br>
<a href="http://bayes.wustl.edu/etj/etj.html">E. T. Jaynes</a> <br>
Wayman Crow Professor of Physics <br>
Washington University <br>
St. Louis, MO 63130, U.S.A. <br>
</PRE>
<P>
<A HREF="jaynespicts.html"><IMG align=middle SRC="jaynes.gif"></A> <P>
<I> Dedicated to the Memory of Sir Harold Jeffreys,
who saw the truth and preserved it. </I>
<P>
Fragmentary Edition of June 1994
<P>
<H2> Short Contents </H2>
<P>
<A HREF="/ETJ-PS/cpreambl.ps"> PREFACE </A>
<P>
<A HREF="ETJbook-comments.html">COMMENTS</A>
<P> General comments (BY OTHERS, NOT E.T. Jaynes') about the book and maxent in general.
<H2>PART A - PRINCIPLES AND ELEMENTARY APPLICATIONS</H2>
<PRE>
<A HREF="/ETJ-PS/cc1p.ps">Chapter 1 Plausible Reasoning</A>
<A HREF="/ETJ-PS/cc2m.ps">Chapter 2 Quantitative Rules: The Cox Theorems</A>
<A HREF="/ETJ-PS/cfig2-1.ps">Fig. 2-1</A>
<A HREF="/ETJ-PS/cc3o.ps">Chapter 3 Elementary Sampling Theory</A>
<A HREF="/ETJ-PS/cc4o.ps">Chapter 4 Elementary Hypothesis Testing</A>
<A HREF="/ETJ-PS/cfig4-1.ps">Fig. 4-1</A>
<A HREF="/ETJ-PS/cc5d.ps">Chapter 5 Queer Uses for Probability Theory</A>
<A HREF="/ETJ-PS/cc6k.ps">Chapter 6 Elementary Parameter Estimation</A>
<A HREF="/ETJ-PS/cfig6-1.ps">Fig. 6-1</A>
<A HREF="/ETJ-PS/cfig6-2.ps">Fig. 6-2</A>
<A HREF="/ETJ-PS/cc7m.ps">Chapter 7 The Central Gaussian, or Normal, Distribution</A>
<A HREF="/ETJ-PS/cc8k.ps">Chapter 8 Sufficiency, Ancillarity, and All That</A>
<A HREF="/ETJ-PS/cc9g.ps">Chapter 9 Repetitive Experiments: Probability and Frequency</A>
<A HREF="/ETJ-PS/cc10i.ps">Chapter 10 Physics of ``Random Experiments''</A>
<A HREF="/ETJ-PS/cc11g.ps">Chapter 11 The Entropy Principle</A>
<A HREF="/ETJ-PS/cc12X.html">Chapter 12 Ignorance Priors -- Transformation Groups</A>
<A HREF="/ETJ-PS/cc13s.ps">Chapter 13 Decision Theory: Historical Survey</A>
<A HREF="/ETJ-PS/cc14g.ps">Chapter 14 Simple Applications of Decision Theory</A>
<A HREF="/ETJ-PS/cc15w.ps">Chapter 15 Paradoxes of Probability Theory</A>
<A HREF="/ETJ-PS/cfig15-1.ps">Fig. 15-1</A>
<A HREF="/ETJ-PS/cc16u.ps">Chapter 16 Orthodox Statistics: Historical Background</A>
<A HREF="/ETJ-PS/cc17b.ps">Chapter 17 Principles and Pathology of Orthodox Statistics</A>
<A HREF="/ETJ-PS/cc18f.ps">Chapter 18 The A --Distribution and Rule of Succession</A>
</PRE>
<H2>PART B -- ADVANCED APPLICATIONS</H2>
<PRE>
<A HREF="/ETJ-PS/cc19g.ps">Chapter 19 Physical Measurements</A>
<A HREF="/ETJ-PS/cc20b.ps">Chapter 20 Regression and Linear Models</A>
<A HREF="/ETJ-PS/cc21a.ps">Chapter 21 Estimation with Cauchy and t--Distributions</A>
<A HREF="/ETJ-PS/cc22X.html">Chapter 22 Time Series Analysis and Autoregressive Models</A>
<A HREF="/ETJ-PS/cc23X.html">Chapter 23 Spectrum / Shape Analysis</A>
<A HREF="/ETJ-PS/cc24f.ps">Chapter 24 Model Comparison and Robustness</A>
<A HREF="/ETJ-PS/cc25X.html">Chapter 25 Image Reconstruction</A>
<A HREF="/ETJ-PS/cc26X.html">Chapter 26 Marginalization Theory</A>
<A HREF="/ETJ-PS/cc27d.ps">Chapter 27 Communication Theory</A>
<A HREF="/ETJ-PS/cc28X.html">Chapter 28 Optimal Antenna and Filter Design</A>
<A HREF="/ETJ-PS/cc29X.html">Chapter 29 Statistical Mechanics</A>
<A HREF="/ETJ-PS/cc30X.html">Chapter 30 Conclusions</A>
</PRE>
<H2>APPENDICES</H2>
<PRE>
<A HREF="/ETJ-PS/cappal.ps">Appendix A Other Approaches to Probability Theory</A>
<A HREF="/ETJ-PS/cappb6.ps">Appendix B Formalities and Mathematical Style</A>
<A HREF="/ETJ-PS/cappc1.ps">Appendix C Convolutions and Cumulants</A>
Appendix D Dirichlet Integrals and Generating Functions
<A HREF="/ETJ-PS/cappe1.ps">Appendix E The Binomial -- Gaussian Hierarchy of Distributions</A>
Appendix F Fourier Analysis
Appendix G Infinite Series
Appendix H Matrix Analysis and Computation
Appendix I Computer Programs
</PRE>
<H2>REFERENCES</H2>
<A HREF="/ETJ-PS/crefsq.ps">List of references</A>
<H4><A HREF="http://bayes.wustl.edu/etj/prob/book.pdf.tar.gz">To transfer all the chapters (Adobe's pdf format) at once from bayes.wustl.edu</A></H4>
<PRE>
Long Contents
PART A -- PRINCIPLES and ELEMENTARY APPLICATIONS
Chapter 1 PLAUSIBLE REASONING
Deductive and Plausible Reasoning 101
Analogies with Physical Theories 103
The Thinking Computer 104
Introducing the Robot 105
Boolean Algebra 106
Adequate Sets of Operations 108
The Basic Desiderata 111
COMMENTS 113
Common Language vs. Formal Logic 114
Nitpicking 116
Chapter 2 THE QUANTITATIVE RULES
The Product Rule 201
The Sum Rule 206
Qualitative Properties 210
Numerical Values 212
Notation and Finite Sets Policy 217
COMMENTS 218
``Subjective'' vs. ``Objective'' 218
G Theorem 218
Venn Diagrams 220
The ``Kolmogorov Axioms'' 222
Chapter 3 ELEMENTARY SAMPLING THEORY
Sampling Without Replacement 301
Logic Versus Propensity 308
Reasoning from Less Precise Information 311
Expectations 313
Other Forms and Extensions 314
Probability as a Mathematical Tool 315
The Binomial Distribution 315
Sampling With Replacement 318
Digression: A Sermon on Reality vs. Models 318
Correction for Correlations 320
Simplification 325
COMMENTS 326
A Look Ahead 328
Chapter 4 ELEMENTARY HYPOTHESIS TESTING
Prior Probabilities 401
Testing Binary Hypotheses with Binary Data 404
Non-Extensibility Beyond the Binary Case 409
Multiple Hypothesis Testing 411
Continuous Probability Distributions (pdf's) 418
Testing an Infinite Number of Hypotheses 419
Simple and Compound (or Composite) Hypotheses 424
COMMENTS 425
Etymology 425
What Have We Accomplished? 426
Chapter 5 QUEER USES FOR PROBABILITY THEORY
Extrasensory Perception 501
Mrs. Stewart's Telepathic Powers 502
Converging and Diverging Views 507
Visual Perception 511
The Discovery of Neptune 512
Digression on Alternative Hypotheses 514
Horseracing and Weather Forecasting 517
Paradoxes of Intuition 520
Bayesian Jurisprudence 521
COMMENTS 522
CONTENTS CONTENTS
Chapter 6 ELEMENTARY PARAMETER ESTIMATION
Inversion of the Urn Distributions 601
Both N and R Unknown 601
Uniform Prior 604
Truncated Uniform Priors 608
A Concave Prior 609
The Binomial Monkey Prior 611
Metamorphosis into Continuous Parameter Estimation 613
Estimation with a Binomial Sampling Distribution 614
Digression on Optional Stopping 616
The Likelihood Principle 617
Compound Estimation Problems 617
A Simple Bayesian Estimate: Quantitative Prior Information 618
>From Posterior Distribution to Estimate 621
Back to the Problem 624
Effects of Qualitative Prior Information 626
The Jeffreys Prior 629
The Point of it All 630
Interval Estimation 632
Calculation of Variance 632
Generalization and Asymptotic Forms 634
A More Careful Asymptotic Derivation 635
COMMENTS 636
Chapter 7 THE CENTRAL GAUSSIAN, OR NORMAL DISTRIBUTION
The Gravitating Phenomenon 701
The Herschel--Maxwell Derivation 702
The Gauss Derivation 703
Historical Importance of Gauss' Result 704
The Landon Derivation 705
Why the Ubiquitous Use of Gaussian Distributions? 707
Why the Ubiquitous Success? 709
The Near--Irrelevance of Sampling Distributions 711
The Remarkable Efficiency of Information Transfer 712
Nuisance Parameters as Safety Devices 713
More General Properties 714
Convolution of Gaussians 715
Galton's Discovery 715
Population Dynamics and Darwinian Evolution 717
Resolution of Distributions into Gaussians 719
The Central Limit Theorem 722
Accuracy of Computations 723
COMMENTS 724
Terminology Again 724
The Great Inequality of Jupiter and Saturn 726
Chapter 8 SUFFICIENCY, ANCILLARITY, AND ALL THAT
Sufficiency 801
Fisher Sufficiency 803
Generalized Sufficiency 804
Examples
Sufficiency Plus Nuisance Parameters
The Pitman--Koopman Theorem
The Likelihood Principle
Effect of Nuisance Parameters
Use of Ancillary Information
Relation to the Likelihood Principle
Asymptotic Likelihood: Fisher Information
Combining Evidence from Different Sources: Meta--Analysis
Pooling the Data
Fine--Grained Propositions: Sam's Broken Thermometer
COMMENTS
The Fallacy of Sample Re--use
A Folk--Theorem
Effect of Prior Information
Clever Tricks and Gamesmanship
Chapter 9 REPETITIVE EXPERIMENTS -- PROBABILITY AND FREQUENCY
Physical Experiments 901
The Poorly Informed Robot 902
Induction 905
Partition Function Algorithms 907
Relation to Generating Functions 911
Another Way of Looking At It 912
Probability and Frequency 913
Halley's Mortality Table 915
COMMENTS: The Irrationalists 918
Chapter 10 PHYSICS OF ``RANDOM EXPERIMENTS''
An Interesting Correlation 1001
Historical Background 1002
How to Cheat at Coin and Die Tossing 1003
Experimental Evidence 1006
Bridge Hands 1007
General Random Experiments 1008
Induction Revisited 1010
But What About Quantum Theory? 1011
Mechanics Under the Clouds 1012
More on Coins and Symmetry 1013
Independence of Tosses 1017
The Arrogance of the Uninformed 1019
Chapter 11 DISCRETE PRIOR PROBABILITIES~--~THE
ENTROPY PRINCIPLE
A New Kind of Prior Information 1101
Minimum 1103
Entropy: Shannon's Theorem 1104
The Wallis Derivation 1108
An Example 1110
Generalization: A More Rigorous Proof 1111
Formal Properties of Maximum Entropy Distributions 1113
Conceptual Problems: Frequency Correspondence 1120
COMMENTS 1124
Chapter 12 UNINFORMATIVE PRIORS~--~TRANSFORMATION GROUPS
Chapter 13 DECISION THEORY~--~HISTORICAL BACKGROUND
Inference vs. Decision 1301
Daniel Bernoulli's Suggestion 1302
The Rationale of Insurance 1303
Entropy and Utility 1305
The Honest Weatherman 1305
Reactions to Daniel Bernoulli and Laplace 1306
Wald's Decision Theory 1307
Parameter Estimation for Minimum Loss 1310
Reformulation of the Problem 1312
Effect of Varying Loss Functions 1315
General Decision Theory 1316
COMMENTS 1317
``Objectivity'' of Decision Theory 1317
Loss Functions in Human Society 1319
A New Look at the Jeffreys Prior 1320
Decision Theory is not Fundamental 1320
Another Dimension? 1321
Chapter 14 SIMPLE APPLICATIONS OF DECISION THEORY
Definitions and Preliminaries 1401
Sufficiency and Information 1403
Loss Functions and Criteria of Optimal Performance 1404
A Discrete Example 1406
How Would Our Robot Do It? 1410
Historical Remarks 1411
The Widget Problem 1412
Solution for Stage 2 1414
Solution for Stage 3 1416
Solution for Stage 4
Chapter 15 PARADOXES OF PROBABILITY THEORY
How Do Paradoxes Survive and Grow? 1501
Summing a Series the Easy Way 1502
Nonconglomerability 1503
Strong Inconsistency 1505
Finite vs. Countable Additivity 1511
The Borel--Kolmogorov Paradox 1513
The Marginalization Paradox 1516
How to Mass--produce Paradoxes 1517
COMMENTS 1518
Counting Infinite Sets? 1520
The Hausdorff Sphere Paradox 1521
Chapter 16 ORTHODOX STATISTICS -- HISTORICAL BACKGROUND
The Early Problems 1601
Sociology of Orthodox Statistics 1602
Ronald Fisher, Harold Jeffreys, and Jerzy Neyman 1603
Pre--data and Post--data Considerations 1608
The Sampling Distribution for an Estimator 1609
Pro--causal and Anti--Causal Bias 1611
What is Real; the Probability or the Phenomenon? 1613
COMMENTS 1613
Chapter 17 PRINCIPLES AND PATHOLOGY OF ORTHODOX STATISTICS
Unbiased Estimators
Confidence Intervals
Nuisance Parameters
Ancillary Statistics
Significance Tests
The Weather in Central Park
More Communication Difficulties
How Can This Be?
Probability Theory is Different
COMMENTS
Gamesmanship
What Does `Bayesian' Mean?
Chapter 18 THE A --DISTRIBUTION AND RULE OF SUCCESSION
Memory Storage for Old Robots 1801
Relevance 1803
A Surprising Consequence 1804
An Application 1806
Laplace's Rule of Succession 1808
Jeffreys' Objection 1810
Bass or Carp? 1811
So Where Does This Leave The Rule? 1811
Generalization 1812
Confirmation and Weight of Evidence 1815
Carnap's Inductive Methods 1817
PART B - ADVANCED APPLICATIONS
Chapter 19 PHYSICAL MEASUREMENTS
Reduction of Equations of Condition 1901
Reformulation as a Decision Problem 1903
Sermon on Gaussian Error Distributions 1904
The Underdetermined Case: K is Singular 1906
The Overdetermined Case: K Can be Made Nonsingular 1906
Numerical Evaluation of the Result 1907
Accuracy of the Estimates 1909
COMMENTS: a Paradox 1910
Chapter 20 REGRESSION AND LINEAR MODELS
Chapter 21 ESTIMATION WITH CAUCHY AND t--DISTRIBUTIONS
Chapter 22 TIME SERIES ANALYSIS AND AUTOREGRESSIVE MODELS
Chapter 23 SPECTRUM / SHAPE ANALYSIS
Chapter 24 MODEL COMPARISON AND ROBUSTNESS
The Bayesian Basis of it All 2401
The Occam Factors 2402
Chapter 25 MARGINALIZATION THEORY
Chapter 26 IMAGE RECONSTRUCTION
Chapter 27 COMMUNICATION THEORY
Origins of the Theory 2701
The Noiseless Channel 2702
The Information Source 2706
Does the English Language Have Statistical Properties? 2708
Optimum Encoding: Letter Frequencies Known 2709
Better Encoding from Knowledge of Digram Frequencies 2712
Relation to a Stochastic Model 2715
The Noisy Channel 2718
Fixing a Noisy Channel: the Checksum Algorithm 2718
Chapter 28 OPTIMAL ANTENNA AND FILTER DESIGN
Chapter 29 STATISTICAL MECHANICS
Chapter 30 CONCLUSIONS
APPENDICES
Appendix A Other Approaches to Probability Theory
The Kolmogorov System of Probability A 1
The de Finetti System of Probability A 5
Comparative Probability A 6
Holdouts Against Comparability A 7
Speculations About Lattice Theories A 8
Appendix B Formalities and Mathematical Style
Notation and Logical Hierarchy B 1
Our ``Cautious Approach" Policy B 3
Willy Feller on Measure Theory B 3
Kronecker vs. Weierstrasz B 5
What is a Legitimate Mathematical Function? B 6
Nondifferentiable Functions B 8
What am I Supposed to Publish? B 10
Mathematical Courtesy B 11
Appendix C Convolutions and Cumulants
Relation of Cumulants and Moments C 4
Examples C 5
Appendix D Dirichlet Integrals and Generating Functions
Appendix E The Binomial~--~Gaussian Hierarchy of Distributions
Appendix F Fourier Theory
Appendix G Infinite Series
Appendix H Matrix Analysis and Computation
Appendix 3pt I Computer Programs
REFERENCES
NAME INDEX
SUBJECT INDEX
</PRE>