跳到正文
DEV Community· Isaac Land·· 11 小時前AI 評分37

為何檢視悖論偏向較長間隔與較大型班級

Why the Inspection Paradox Favors Long Intervals and Large Classes

AI 導讀

檢視悖論説明,按隨機時點或個人抽樣時,較長的巴士班次間隔和較大型班級更容易被抽中,因此觀察到的結果可能高於整體平均。以文中數據計算,學生平均經歷的班級人數為147,約為每門課平均40.5人的3.7倍;在一個長短班次間隔各佔一半的例子中,乘客有75%機會遇上15分鐘間隔。

正文

Why the Inspection Paradox Favors Long Intervals and Large Classes

A bus stop sign in Geebung, Brisbane, Australia, displaying departure timetables for the routes serving the stop, photographed early one morning in 2009

The timetable board at Sandra Street stop 47 on Piccadilly Street in Geebung, Brisbane, Australia (photographed in 2009). The average departure interval calculated from the timetable is an average across bus departures. Photograph: John Robert McPherson; source: Wikimedia Commons; license: CC BY-SA 4.0.

Arrive at a bus stop at a random moment, and you will often land in a relatively long interval between departures; ask a randomly chosen university student how many people are in their class, and the answer will often exceed the university's published average class size. Probability theory calls phenomena like these the inspection paradox: when the sampling unit differs from the object of interest, longer and larger individuals are more likely to be selected. This article begins with the bus example in Feller's textbook, moves through the class size paradox and the friendship paradox, and then offers an illustration that can be checked by hand.

The Waiting Time Paradox in Feller's Textbook

In 1966, William Feller presented a “waiting time paradox” in Chapter One, Section 4 of Volume Two of An Introduction to Probability Theory and Its Applications. Buses arrive according to a Poisson process, with the average interval between consecutive buses denoted by 1/α; “I” arrive at a time independent of the buses (at precisely noon in the book's example). How long should I expect to wait? The book presents two seemingly reasonable answers: a Poisson process is memoryless, so whenever I arrive, the expected wait should be the full 1/α; yet my arrival also seems to fall randomly between two buses, so symmetry suggests that I should wait only half an interval.

Feller's explanation is that the interval containing “my” arrival time is not an ordinary interval. In the exponential case, its expected length is 2/α, exactly twice the average interval. Taking half still gives 1/α, so the two answers no longer contradict each other. He wrote that this explanation had astonished experienced researchers, but the reasoning can be roughly summed up in one sentence: long intervals have a greater chance than short ones of containing a given fixed moment.

In Chapter Six, Section 7 of the same volume, Feller formally called this the “inspection paradox.” Imagine replacing batteries as soon as they fail, then inspecting the battery in use at a particular moment and recording its lifetime. Intuitively, these batteries should represent the lifetime distribution of all batteries, but they do not: with exponentially distributed lifetimes, the expected lifetime of a selected battery doubles. Feller cautioned that a seemingly unbiased inspection scheme can lead to incorrect conclusions because the observed sample may not represent the population; he also offered a simple remedy—inspect the first battery installed after that moment, and the problem disappears.

The Class Size Paradox and the Friendship Paradox

In 1977, sociologists Scott L. Feld and Bernard Grofman brought the same mechanism into university classrooms in Research in Higher Education. According to the spring 1974 teaching evaluation survey they cited, the State University of New York at Stony Brook offered 737 courses (with data missing for another 9), averaging 40.5 students per course, with a standard deviation of 65.8; averaged across students, however, the mean size of the classes students attended was 147, approximately 3.7 times the former figure. The reason is straightforward: a large class of 300 students is reported once by each of its 300 students, while a small class of 10 is reported only 10 times.

The authors expressed this difference as a “perception ratio”: the average class size experienced by students divided by the university's average class size equals 1 plus the square of the coefficient of variation of class size, or 1 + (σ/μ)². When all classes are the same size, the ratio is 1; whenever sizes differ, the average experienced by students must be larger, and the greater the variation in class size, the larger the gap. The paper reported a ratio of approximately 3.7; substituting its reported mean and standard deviation gives 1 + (65.8/40.5)² ≈ 3.64, consistent with the result of 147 ÷ 40.5.

A black-and-white photograph of a teacher at the lectern teaching all the chemistry students in the chemistry department's large lecture hall at Iowa State College in 1915

A chemistry class in the chemistry department's lecture hall at Iowa State College in 1915. When class sizes are counted by student, a large class like this is counted once for every student present. Source: Iowa State University Library Special Collections and University Archives (Flickr); license: Public Domain Mark 1.0.

In 1991, Feld published “Why Your Friends Have More Friends Than You Do” in the American Journal of Sociology, referring to this family of phenomena collectively as the “class size paradox.” Using friendship data from James Coleman's 1961 The Adolescent Society, he showed that most people have fewer friends than their friends have on average. In network terms, a person reached by following a randomly selected friendship connection (an edge in the network) has a probability of selection proportional to their number of friends—someone with 50 friends appears on 50 edges, while someone with only 1 friend appears on just 1 edge. Allen Downey calculated this using approximately 4000 Facebook users in Stanford's SNAP dataset: users had an average of 42 friends, while randomly selected “friends” had an average of 91.

A Departure Interval Illustration You Can Check

The following figures can be checked using only basic arithmetic. Suppose half the departure intervals on a route are 5 minutes and half are 15 minutes, alternating, so the average interval across departures is 10 minutes. When passengers arrive at random moments, 15 minutes out of every 20 fall within a long interval, so 75% of passengers encounter a 15-minute interval and only 25% encounter a 5-minute interval—the long intervals account for only half the departures but three-quarters of the time.

Thus, the average length of the interval containing a passenger's arrival is 0.75 × 15 + 0.25 × 5 = 12.5 minutes, rather than 10 minutes. Arrival times are uniformly distributed within the interval they fall in, so the average wait is half that interval: 0.75 × 7.5 + 0.25 × 2.5 = 6.25 minutes, which is 25% more than the 5 minutes obtained by taking “half the average interval.” In general, writing the departure interval as X, the expected length of the interval containing a passenger's arrival is E[X²]/E[X] = E[X] + Var(X)/E[X], and the average wait is half of that. In this example, E[X²] = (25 + 225)/2 = 125,125/10 = 12.5; the ratio 12.5/10 = 1 + (5/10)² = 1.25 is precisely Feld and Grofman's perception ratio formula. Only when departures are perfectly regular, with a variance of 0, do the two averages coincide; when intervals follow an exponential distribution, we return to Feller's example, in which the expected length of the containing interval doubles.

To check this with a simulation, you can run the following Python code (the comments give the theoretical values, around which simulation results will fluctuate slightly):

import bisect
import random

random.seed(1)
gaps = [5, 15] * 50_000  # alternating 5- and 15-minute gaps
starts, t = [], 0
for g in gaps:
    starts.append(t)
    t += g

hit, wait = [], []
for _ in range(100_000):
    u = random.uniform(0, t)  # a rider arrives at a random moment
    i = bisect.bisect_right(starts, u) - 1
    hit.append(gaps[i])
    wait.append(starts[i] + gaps[i] - u)

print(sum(gaps) / len(gaps))  # 10.0: mean gap per bus
print(sum(hit) / len(hit))    # theory 12.5: mean gap a rider lands in
print(sum(wait) / len(wait))  # theory 6.25: mean wait

Schematic diagram (drawn with code): alternating colored blocks representing 5-minute (green) and 15-minute (orange) departure intervals along a horizontal timeline.

When half the departure intervals are 5 minutes and half are 15 minutes, the long intervals account for only half the departures but three-quarters of the time, so passengers arriving at random moments more often land in long intervals. The block proportions are illustrative. AI-generated schematic.

Length-Biased Sampling and Ways to Correct It

The common structure in these examples is called length-biased sampling in statistics, also known as size-biased sampling: an individual's probability of selection is proportional to its “length” or “size.” In a 2010 paper, Richard Arratia and Larry Goldstein used a cafeteria example: ask students eating there, “How many people are at your table, including you?” and 20% say 1, 30% say 2, 30% say 3, and 20% say 4; converting this to tables, assuming a total of 100 students, gives 20 + 15 + 10 + 5 = 50 tables, of which 40% seat only 1 person and just 10% have all 4 seats occupied. Asking people and counting tables yield two different distributions.

Intuition often goes astray because we assume that “what I see” represents “the whole population.” In 2015, Downey compiled several everyday examples: he recorded the arrival times of approximately 70 trains on Boston's subway Red Line between 4 and 5 in the afternoon; the average interval between trains was 7.8 minutes, while the average interval experienced by passengers was 8.8 minutes, with an average wait of approximately 4.4 minutes, about 15% longer than the estimate based on the average interval. Airlines complain about too many empty seats, while passengers feel that flights are always full. In his rough estimate of U.S. federal prison sentences, the mean sentence was approximately 121 months, while prisoners encountered randomly at a random moment had a mean sentence of approximately 183 months. The size of the bias depends on how dispersed the distribution is: the Red Line's intervals do not vary greatly, so the bias is modest; for long-tailed distributions such as numbers of friends, the bias can exceed a doubling.

The inspection paradox itself is not a calculation error; the problem lies in treating one kind of average as another. Neither the student-weighted average of 147 nor the course-weighted average of 40.5 is incorrect; they simply answer different questions. Downey's advice is that a student-weighted average may be more meaningful if the aim is to describe students' experiences, but you must be clear about what you are measuring. If only a biased sample is available, it is also possible to work backward: he demonstrated how to recover the actual class size distribution from the student-perspective distribution, using a method equivalent to downweighting each observation by 1/x. Another approach is to change the sampling method, as Feller suggested: inspect the first newly installed product after a particular moment, rather than the one that happens to be in use.

References

  1. Feller, W. (1966). An Introduction to Probability Theory and Its Applications, Vol. II. New York: Wiley. Chapter I, Section 4, Waiting Time Paradoxes; Chapter VI, Section 7, An inspection paradox. Bibliographic page

  2. Feld, S. L., & Grofman, B. (1977). Variation in class size, the class size paradox, and some consequences for students. Research in Higher Education, 6(3), 215–222. doi:10.1007/BF00991287

  3. Feld, S. L. (1991). Why your friends have more friends than you do. American Journal of Sociology, 96(6), 1464–1477. doi:10.1086/229693

  4. Arratia, R., & Goldstein, L. (2010). Size bias, sampling, the waiting time paradox, and infinite divisibility: when is the increment independent? arXiv:1007.3910

  5. Downey, A. (2015, August 18). The inspection paradox is everywhere. Probably Overthinking It. Original blog post

  6. Research Methods · Rituals can be imitated; understanding cannot be faked

  7. This information was compiled and published by Scholay One-Stop Academic Platform. Please report any infringement so the material can be removed.

  8. This information was also published on EnterScholar, a community for scholars in the age of intelligence.


Originally published at Metho.

來源:DEV Community · dev.to