Every year, there are upsets and wild outcomes during the NCAA men’s basketball tournament. But this year felt, well, *wilder*. For example, for the first time in 136 games over the 34 years of the tournament’s current 64-team format, a #16 seed (UMBC) beat a #1 seed (Virginia) in the first round. (I refuse to acknowledge the abomination of the 4 “play in” games in the zero-th round.) And I am a Kansas State fan, who watched my #9 seed Wildcats beat *Kentucky*, a team that went to the Final Four in 4 of the last 8 years.

So I wondered whether this was indeed the “wildest” tournament ever… and it turns out that it was, by several reasonable metrics.

**Modeling game probabilities**

To compare the tournaments in different years, we assume that the probability of outcome of any particular game depends only on the *seeds* of the opposing teams. Schwertman et. al. (see reference below) suggest a reasonable model of the form

where is some measure of the “strength” of seed (ranging from 1 to 16), and the scale factor calibrates the range of resulting probabilities, selected here so that the most extreme value matches the current maximum likelihood estimate based on the 136 observations over the past 34 years.

One simple strength function is the linear , although this would suggest, for example, that #1 vs. #5 and #11 vs. #15 are essentially identical match-ups. A better fit is

where is the quantile of the normal distribution, and is the number of teams in all of Division I. The idea is that team strength is normally distributed, and the tournament invites the 64 teams in the upper tail of the distribution, as shown in the figure below.

**Probability of a perfect bracket**

Armed with these candidate models, I looked at all of the tournaments since 1985, the first year of the current 64-team format. I have provided summary data sets before (a search of this blog for “NCAA” will yield several posts on this subject), but this analysis required more raw data, all of which is now available at the usual location here.

For each year of the tournament, we can ask what is the probability of picking a perfect bracket in that year, correctly identifying the winners of all 63 games? Actually, there are three reasonable variants of this question:

- If we flip a coin to pick each game, what is the probability of picking every game correctly?
- If we pick a “chalk” bracket, always picking the favored higher-seeded (i.e., lower-numbered) team to win each game, what is the probability of picking every game correctly?
- If we managed to pick the perfect bracket for a given year, what is the prior probability of that particular outcome?

The answer to the first question is the 1 in , or the “1 in 9.2 quintillion” that appears in popular press. And this is always exactly correct, no matter how individual teams actually match up in any given year, as long as we are flipping a coin to guess the outcome of each game. But this isn’t very realistic, since seed match-ups *do* matter; a #1 seed will beat a #16 seed… well, *almost* all of the time.

So the second question is more interesting, but also more complicated, since it *does* depend on our model of how different seeds match up… but it doesn’t depend on which year of the tournament we’re talking about, at least as long as we always use the same model. Using the strength models described above, a chalk bracket has a probability of around 1 in 100 billion of being correct (1 in 186 billion for the linear strength model, or 1 in 90 billion for the normal strength model).

The third question is the motivation for this post: the probability of a given year’s actual outcome will generally lie somewhere between the other two “extremes.” How has this probability varied over the years, and was 2018 really an outlier? The results are shown in the figure below.

The constant black line at the bottom is the 1 in 9.2 quintillion coin flip. The constant red and blue lines at the top are the probabilities of a chalk bracket, assuming the linear or normal strength models, respectively.

And in between are the actual outcomes of each tournament. (*Aside*: I tried a bar chart for this, but I think the line plot more clearly shows the comparison of the two models, as well as both the maximum *and* minimum behavior that we’re interested in here.) This year’s 2018 tournament was indeed the most unlikely, so to speak, although it has close competition, all in this decade. At the other extreme, 2007 was the *most* likely bracket.

**Reference:**

- Schwertman, N., McCready, T., and Howard, L., Probability Models for the NCAA Regional Basketball Tournaments,
*The American Statistician*,**45**(1) February 1991, p. 35-38 [JSTOR]

- Paul says, “Neither Steve nor Ted was in on it.”
- Quinn says, “Ray wasn’t in on it, but Paul was.”
- Ray says, “If Ted was in on it, then so was Steve.”
- Steve says, “Paul wasn’t in on it, but Quinn was.”
- Ted says, “Quinn wasn’t in on it, but Paul was.”

You do not know which, nor even how many, of the five suspects were involved in the crime. However, you do know that every guilty suspect is lying, and every innocent suspect is telling the truth. Which suspect or suspects committed the crime?

I think puzzles similar to this one make good, fun homework problems in a discrete mathematics course introducing propositional logic. However, this particular puzzle is a bit more complex than the usual “Which one of three suspects is guilty?” type, not just because there are more suspects, but also because we don’t know *how many* suspects are guilty.

That added complexity is motivated by trying to transform this typically pencil-and-paper mathematical logic problem into a potentially nice computer science programming exercise: consider writing a program to *automate* solving this problem… or even better, writing a program to *generate new random instances* of problems like this one, while ensuring that the resulting puzzle has some reasonably “nice” properties. For example, the solution should be unique; but the puzzle should also be “interesting,” in that we should need *all* of the suspects’ statements to deduce who is guilty (that is, any proper subset of the statements should imply at least two distinct possible solutions).

This is a follow-up to one of last month’s posts that contained some images of randomly-generated “lozenge tilings.” The focus here is not on the tilings themselves, but on the *perfectly random* sampling technique due to Propp and Wilson known as “coupling from the past” (see references below) that was used to generate them. As is often the case here, I don’t have any new mathematics to contribute; the objective of this post is just to go beyond the pseudo-code descriptions of the technique in most of the literature, and provide working Python code with a couple of different example applications.

The basic problem is this: suppose that we want to sample from some probability distribution over a finite discrete space , and that we have an ergodic Markov chain whose steady state distribution is . An *approximate* sampling procedure is to pick an arbitrary initial state, then just run the chain for some large number of iterations, with the final state as your sample. This idea has been discussed here before, in the context of card shuffling, the board game Monopoly, as well as the lozenge tilings from last month.

For example, consider shuffling a deck of cards using the following iterative procedure: select a random *adjacent* pair of cards, and flip a coin to decide whether to put them in ascending or descending order (assuming any convenient total ordering, e.g. first by rank, then by suit alphabetically). Repeat for some “large” number of iterations.

There are two problems with this approach:

- It’s approximate; the longer we iterate the chain, the closer we get to the steady state distribution… but (usually) no matter when we stop, the distribution of the resulting state is never exactly .
- How many iterations are enough? For many Markov chains, the mixing time may be difficult to analyze or even estimate.

**Coupling from the past**

Coupling from the past can be used to address both of these problems. For a surprisingly large class of applications, it is possible to sample *exactly* from the stationary distribution of a chain, by iterating multiple realizations of the chain until they are “coupled,” without needing to know ahead of time when to stop.

First, suppose that we can express the random state transition behavior of the chain using a fixed, deterministic function , so that for a random variable uniformly distributed on the unit interval,

In other words, given a current state and random draw , the next state is .

Now consider a single infinite sequence of random draws . We will use this same *single* source of randomness to iterate *multiple* realizations of the chain, one for each of the possible initial states… but starting *in the past*, and running forward to time zero. More precisely, let’s focus on two particular chains with different initial states and at time , so that

(Notice how the more random draws we generate, the farther *back* in time they are used. More on this shortly.)

A key observation is that if at any point, then the chains are “coupled:” since they experience the same sequence of randomness influencing their behavior, they will continue to move in lockstep for all subsequent iterations as well, up to . Furthermore, if the states at time zero are *all* the same for *all* possible initial states run forward from time , then the distribution of that final state is the stationary distribution of the chain.

But what if all initial states *don’t* end at the same final state? This is where the single source of randomness is key: we can simply look farther back in the past, say time steps instead of , by extending our sequence of random draws… as long as we *re-use the existing random draws* to make the same “updates” to the states at later times (i.e., closer to the end time zero).

**Monotone coupling**

So all we need to do to perfectly sample from the stationary distribution is

- Choose a sufficiently large to look far enough into the past.
- Run realizations of the chain, one for each initial state, all starting at time and running up to time zero.
- As long as the final state is the same for all initial states, output as the sample. Otherwise, look farther back in the past (), generate additional random draws accordingly, and go back to step 2.

This doesn’t seem very helpful, since step 2 requires enumerating elements of the typically enormous state space. Fortunately, in many cases, including both lozenge tiling and card shuffling, it is possible to simplify the procedure by imposing a partial order on the state space– with minimum and maximum elements– that is preserved by the update function . That is, suppose that for all states and random draws , if , then .

Then instead of running the chain for *all* possible initial states, we can just run *two* chains, starting with the minimum and maximum elements in the partial order. If those two chains are coupled by time zero, then they will also “squeeze” any other initial state into the same coupling.

Following is the resulting Python implementation:

import random def monotone_cftp(mc): """Return stationary distribution sample using monotone CFTP.""" updates = [mc.random_update()] while True: s0, s1 = mc.min_max() for u in updates: s0.update(u) s1.update(u) if s0 == s1: break updates = [mc.random_update() for k in range(len(updates) + 1)] + updates return s0

Sort of like the Fisher-Yates shuffle, this is one of those algorithms that is potentially easy to implement incorrectly. In particular, note that the list of “updates,” which is traversed *in order* during iteration of the two chains, is *prepended* with additional blocks of random draws as needed to start farther back in the past. The figure below shows the resulting behavior, mapping each output of the random number generator to the time at which it is used to update the chains.

Coming back once again to the card shuffling example, the code below uses coupling from the past with random adjacent transpositions to generate an exactly uniform random permutation . (A similar example generating random lozenge tilings may be found at the usual location here.)

class Shuffle: def __init__(self, n, descending=False): self.p = list(range(n)) if descending: self.p.reverse() def min_max(self): n = len(self.p) return Shuffle(n, False), Shuffle(n, True) def __eq__(self, other): return self.p == other.p def random_update(self): return (random.randint(0, len(self.p) - 2), random.randint(0, 1)) def update(self, u): k, c = u if (c == 1) != (self.p[k] < self.p[k + 1]): self.p[k], self.p[k + 1] = self.p[k + 1], self.p[k] print(monotone_cftp(Shuffle(52)).p)

The minimum and maximum elements of the partial order are the identity and reversal permutations, as might be expected; but it’s an interesting puzzle to determine the actual partial order that these random adjacent transpositions preserve. Consider, for example, a similar process where we select a random adjacent pair of cards, then flip a coin to decide *whether to swap them or not* (vs. whether to put them in ascending or descending order). This process has the same uniform stationary distribution, but won’t work when applied to monotone coupling from the past.

**Re-generating vs. storing random draws**

One final note: although the above implementation is relatively easy to read, it may be prohibitively expensive to store all of the random draws as they are generated. The slightly more complex implementation below is identical in output behavior, but is more space-efficient by only storing “markers” in the random stream, re-generating the draws themselves as needed when looking farther back into the past.

import random def monotone_cftp(mc): """Return stationary distribution sample using monotone CFTP.""" updates = [(random.getstate(), 1)] while True: s0, s1 = mc.min_max() rng_next = None for rng, steps in updates: random.setstate(rng) for t in range(steps): u = mc.random_update() s0.update(u) s1.update(u) if rng_next is None: rng_next = random.getstate() if s0 == s1: break updates.insert(0, (rng_next, 2 ** len(updates))) random.setstate(rng_next) return s0

**References:**

- Propp, J. and Wilson, D., Exact Sampling with Coupled Markov Chains and Applications to Statistical Mechanics,
*Random Structures and Algorithms*,**9**1996, p. 223-252 [PDF] - Wilson, D., Mixing times of lozenge tiling and card shuffling Markov chains,
*Annals of Applied Probability*,**14**(1) 2004, p. 274-325 [PDF]

I recently saw several different checks, all from the same bank, each with a different dollar amount printed in a fixed-width font, similar to the shortened example below:

`**PAY EXACTLY**twelve thousand thirty-four and 56/100****************`

Counting asterisks confirmed that the amount fields on each check were all padded to exactly the same length… which made me wonder, what is the largest check that the bank could print, using English words for the dollar amount?

Actually, that’s not quite the question I had in mind. There are very short names for very large numbers, but it’s not very helpful to say that a bank can cut a check for, say, “one googol” dollars, when most *smaller* amounts would not fit in the same length of field.

So to make the problem less linguistic and more mathematical, let’s instead ask the following more precise question: what is the largest integer such that *every* dollar amount from zero to can be printed in words using at most characters?

**Converting integers into words**

For short fields, such as in the case of the bank checks, brute force is sufficient; we just need a function to convert integers into words. This is a pretty common programming problem, with one Python implementation shown below, where `integer_name(n)`

works for any :

units = ['zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten', 'eleven', 'twelve', 'thirteen', 'fourteen', 'fifteen', 'sixteen', 'seventeen', 'eighteen', 'nineteen'] tens = ['zero', 'ten', 'twenty', 'thirty', 'forty', 'fifty', 'sixty', 'seventy', 'eighty', 'ninety'] powers = ['zero', 'thousand', 'million', 'billion', 'trillion', 'quadrillion', 'quintillion', 'sextillion', 'septillion', 'octillion', 'nonillion', 'decillion'] hundred = 'hundred' minus = 'minus' comma = ',' and_ = ' and' space = ' ' hyphen = '-' empty = '' def small_integer_name(n, use_and=False): s = empty if n >= 100: q, n = divmod(n, 100) s = units[q] + space + hundred + ( (and_ if use_and else empty) + space if n > 0 else empty) if n >= 20: q, n = divmod(n, 10) s += tens[q] + (hyphen if n > 0 else empty) return (s + units[n] if n > 0 else s) def integer_name(n, use_comma=False, use_and=False, power=0): if n < 0: return minus + space + integer_name(-n, use_comma, use_and) elif n == 0: return units[0] s = empty if n >= 1000: q, n = divmod(n, 1000) s = integer_name(q, use_comma, use_and, power + 1) + ( (comma if use_comma else empty) + space if n > 0 else empty) return (s + small_integer_name(n, use_and) + (space + powers[power] if power > 0 else empty) if n > 0 else s)

There are a couple of things to note. First, although typical American style does not include commas separating thousands, or the word *and* between hundreds and tens (e.g., “one hundred and twenty-three”), it’s nice to have the option here, since it affects the calculation of .

Also, “everything” is a variable, including the `hyphen`

, `comma`

, `space`

, even the `empty`

string. The addition operator is overloaded for string concatenation, but it’s easy to replace the string constants with, say, integer word counts, or syllable counts, or whatever, without changing the code. For example, we can automatically generate lousy haiku:

One hundred twenty-

one thousand seven hundred

seventy-seven

Coming back to the bank checks, it turns out that , so that the bank can print a check for any amount up to $1,113,322.99. Which doesn’t seem terribly large– I wonder if they fall back to using numerals if the amount doesn’t fit using words?

**Twitter scale**

This isn’t a new problem. A couple of years ago, FiveThirtyEight.com’s Riddler posed a problem involving the Twitter account @CountVonCount, of Sesame Street fame, essentially asking for … and then again asking for just last month, in response to the recent increase in maximum length of a tweet from 140 to 280 characters (in each case, less one character to account for an exclamation point).

These field widths are large enough that a brute force approach is no longer feasible. It’s a nice problem to implement a function to compute efficiently and exactly, with results as shown in the figure below.

All of the source code is available at the usual location here.

]]>In the figure below, a regular hexagon with side length 12 is tiled with “lozenges,” sort of like triangular dominoes, each consisting of a pair of unit-length equilateral triangles joined at a common side.

Each lozenge is in one of three possible orientations. Although the orientations appear random, if we count carefully, we find that there are exactly the same number of lozenges in each of the three orientations. In fact, no matter how we tile the hexagon, it is always the case that the number of lozenges (144 in this case) of each “type” is fixed. Can you prove this?

**Proof without words**

Although the (proof of the) Pythagorean theorem seems to be the most common example of a proof without words— usually a picture or diagram that doesn’t require any words to explain– this problem, with its “proof” below, is probably my favorite.

One reason I like this problem, particularly as motivation for student discussion, is that it is somewhat controversial. Is the above “proof without words” really a satisfactory proof?

**Theorem without words**

Moving away from random tilings for the moment, the original motivation for this post wasn’t actually a *proof* without words, but a *theorem* without words. I recently learned an interesting result I had not seen before, involving no more than high school-level physics, that I thought could be presented as an animation without any accompanying explanation:

Unfortunately, I’m not sure this quite succeeds as a “theorem without words.” As with the tiling proof, there is more going on here than the above animation arguably conveys. In particular, one of the most interesting things about this problem– that the animation *doesn’t* really show– is that the *shape* (that is, the eccentricity) of the ellipse of apogees is invariant: it does not depend on the speed of the projectile, or gravity, or any relationship between the two.

**More on random lozenge tilings**

Finally, some source code: although there are plenty of papers and web sites with images of random tilings like those above, I wanted to make my own images, and working code is harder to find. See here for my Python implementation for generating a random lozenge tiling of a hexagon of a given size. For example, the following figure shows a random tiling of a hexagon with side length 64:

My implementation is quick and dirty in the sense that it simply iterates the Markov chain of single-step up-or-down moves in the corresponding family of non-intersecting lattice paths, essentially shuffling long enough for the resulting random tiling to be *approximately* uniform. (Note that the tilings in the above images are more random than they might appear, despite the “frozen” regions near the corners.) It would be interesting to extend this implementation to use coupling from the past, to generate an *exactly* uniform random tiling.

**References:**

This post was motivated by a recent attempt to transform a photograph into a large digital print, in what I hoped would be a mathematically interesting way. The idea was pretty simple: convert the (originally color) image into a black-on-white line drawing– a white background, with a single, continuous, convoluted black curve, one pixel wide.

This isn’t a new idea. One example of how to do this is to convert the image into a solution of an instance of the traveling salesman problem, with more “cities” clustered in darker regions of the source image. But I wanted to do something slightly different, with more explicitly visible structure… which doesn’t necessarily translate to more visual appeal: draw a Hilbert space-filling curve, but vary the *order* of the curve (roughly, the depth of recursion) locally according to the gray level of the corresponding pixels of the source image.

After some experimenting, I settled on the transformation described in the figure below. Each pixel of the source image is “inflated” to an 8-by-8 block of pixels in the output, with a black pixel (lower left) represented by a second-order Hilbert curve, and a white pixel (upper left) by just a line segment directly connecting the endpoints of the block, with two additional gray levels in between, each connecting progressively more/fewer points along the curve.

**Example**

The figure below shows an example of creating an image for input to the algorithm. There are two challenges to consider:

The 8-fold inflation of each pixel means that if we want the final output to be 1024-by-1024, then the input image must be 128-by-128, as shown here. (For my project, I could afford a 512-by-512 input, with larger than single-pixel steps between points on the curve, yielding output suitable for a 36-inch print.)*Size:*The color in the input image must be quantized to just four gray levels, using your favorite photo editor. (I did this in Mathematica.)**Grayscale:**

The figure below shows the resulting output. You may have to zoom in to see the details of the curve, especially in the black regions.

**Source code**

Following is the Python source that does the transformation. It uses the Hilbert curve encoding module from my allRGB experiment, and I used Pygame for the image formatting so that I could watch the output as it was created.

import hilbert import pygame BLACK = (0, 0, 0, 255) DARK_GRAY = (85, 85, 85, 255) LIGHT_GRAY = (160, 160, 160, 255) WHITE = (255, 255, 255, 255) STEPS = {BLACK: [1] * 15, DARK_GRAY: [1, 1, 1, 1, 4, 1, 1, 5], LIGHT_GRAY: [4, 7, 4], WHITE: [15]} class Halftone: def __init__(self, image, step): self.h = hilbert.Hilbert(2) self.index = -1 self.pos = (0, 0) width, height = image.get_size() self.target = pygame.Surface((step * 4 * width, step * 4 * height)) self.target.fill(WHITE) for pixel in range(width * height): self.move(1, step) for n in STEPS[tuple(image.get_at([w // 4 for w in self.pos]))]: self.move(n, step) def move(self, n, step): self.index = self.index + n next_pos = self.h.encode(self.index) pygame.draw.line(self.target, BLACK, [step * w for w in self.pos], [step * w for w in next_pos]) self.pos = next_pos if __name__ == '__main__': import sys step = int(sys.argv[1]) for filename in sys.argv[2:]: pygame.image.save(Halftone(pygame.image.load(filename), step).target, filename + '.ht.png')]]>

Earlier this year, I spent some time calculating the probability of a Scrabble “bingo:” drawing a rack of 7 tiles and playing all of them in a single turn to spell a 7-letter word. The interesting part of the analysis was the use of the Google Books Ngrams data set to limit the dictionary of playable words by their frequency of occurrence, to account for novice players like me that might only know “easier” words in more common use. The result was that the probability of drawing a rack of 7 letters that can play a 7-letter word is about 0.132601… if we allow the *entire* official Scrabble dictionary, including words like *zygoses* (?). The probability is only about 0.07 if we assume that I only know about a third– the most frequently occurring third– of the 7-letter words in the dictionary.

But given just how poorly I play Scrabble, it seems optimistic to focus on a bingo. Instead of asking whether we can play a 7-letter word using the entire rack, let’s consider what turns out to be a much harder problem, namely, whether the rack is *playable at all*: that is,

**Problem:** What is the probability that a randomly drawn rack is *playable*, i.e., contains some subset of tiles, not necessarily all 7, that spell a word in the dictionary?

**Minimal subracks**

The first interesting aspect of this problem is the size of the dictionary. There are 187,632 words in the 2014 *Official Tournament and Club Word List* (the latest for which an electronic version is accessible)… but we can immediately discard nearly 70% of them, those with more than 7 letters, leaving 56,624 words with at most 7 letters. (There is exactly one such word, *pizzazz*, that I’ll leave in our list, despite the fact that it is one of 13 words in the dictionary with no hope of ever being played: it has too many *z*‘s!)

But we can trim the list further, first by noting that the order of letters in a word doesn’t matter– this helps a little but not much– and second by noting that some words may contain other shorter words as subsets. For example, the single most frequently occurring word, *the*, contains the shorter word *he*. So when evaluating a rack to see whether it is playable or not, we only need to check if we can play *he*; if we can’t, then we can’t play *the*, either.

In other words, we only need to consider *minimal subracks*: the minimal elements of the inclusion partial order on sets of tiles spelling words in the dictionary. This is a huge savings: instead of 56,624 words with at most 7 letters, we only need to consider the following 223 minimal “words,” or unordered subsets of tiles, with blank tiles indicated by an underscore:

__, _a, _b, _cv, _d, _e, _f, _g, _h, _i, _j, _k, _l, _m, _n, _o, _p, _q, _r, _s, _t, _u, _vv, _w, _x, _y, _z, aa, ab, aco, acv, ad, ae, af, ag, ah, ai, ak, al, am, an, aov, ap, aqu, ar, as, at, auv, avv, aw, ax, ay, az, bbu, bcu, bdu, be, bfu, bgu, bi, bklu, bllu, bo, brr, bru, by, cciiv, ccko, cdu, cee, cei, cekk, ceu, cffu, cfku, cgku, cgllyy, chly, chrtw, ciirr, ciy, cklu, cllu, cmw, coo, coz, cru, cry, cuz, ddu, de, dfu, dgu, di, djuy, dkuu, dlu, do, dru, dry, duw, eeg, eej, eek, eequu, eev, eez, ef, egg, egk, egv, eh, eiv, eju, eku, ekz, el, em, en, eo, ep, er, es, et, ew, ex, ey, ffpt, fgu, fi, flu, fly, fo, fru, fry, ghlly, gi, gju, glu, go, gpy, grr, gru, gsyyyz, guv, guy, hi, hm, hnt, ho, hpt, hpy, hs, hty, hu, hwy, iirvz, ijzz, ik, il, im, in, io, ip, iq, irry, is, it, ivy, iwz, ix, jjuu, jkuu, jo, kkoo, klru, kouz, kruu, kst, ksy, kuy, lllu, llxyy, lnxy, lo, lpy, lsy, luu, luv, mm, mo, mu, my, no, nsy, nu, nwy, ooz, op, or, os, ot, ow, ox, oy, ppty, pry, pst, psy, ptyy, pu, pxy, rty, ruy, rwy, sty, su, tu, uuyz, uwz, ux, uzz, zzz

The figure below expands on this calculation, showing the number of minimal subracks on the *y*-axis if we limit the dictionary of words we want to be able to play in various ways: either by frequency on the *x*-axis, where dumber is to the left; or by minimum word length, where the bottom curve labeled “>=2” is the entire dictionary, and the top curve labeled “>=7” requires a bingo.

As this figure shows, we benefit greatly– computationally, that is– from being able to play short words. At the other extreme, if we require a bingo, then effectively *every word is a minimal subrack*, and we actually suffer further inflation by including the various ways to spell each word with blanks.

**Finding a subrack in a trie**

At this point, the goal is to compute the probability that a randomly drawn rack of 7 tiles contains at least one of the minimal subracks described above. It turns out that there are only 3,199,724 distinct (but not equally likely) possible racks, so it *should* be feasible to simply enumerate them, testing each rack one by one, accumulating the frequency of those that are playable… as long as each test for containing a minimal subrack is reasonably fast. (Even if we just wanted to *estimate* the probability via Monte Carlo simulation by sampling randomly drawn racks, we still need the same fast test for playability.)

This problem is ready-made for a trie data structure, an ordered tree where each edge is labeled with a letter (or blank), and each leaf vertex represents a minimal playable subrack containing the *sorted* multiset of letters along the path to the leaf. (We don’t have to “mark” any non-leaf vertices; only leaves represent playable words, since we reduced the dictionary to the antichain of *minimal* subracks.)

The following Python code represents a trie of multisets as a nested dictionary with single-element keys (letters in this case), implementing the two operations needed for our purpose:

- Inserting a “word” multiset given as a sorted list of elements (or a sorted string of characters in this case),
- Testing whether a given word (i.e., a drawn rack of 7 tiles, also sorted) contains a word in the trie as a subset.

Insertion is straightforward; the recursive subset test is more interesting:

def trie_insert(t, word): for letter in word: if not letter in t: t[letter] = {} t = t[letter] def trie_subset(t, word): if not t: return True if not word: return False return (word[0] in t and trie_subset(t[word[0]], word[1:]) or trie_subset(t, word[1:]))

**Enumerating racks**

Armed with a test for whether a given rack is playable, it remains to enumerate all possible racks, testing each one in turn. In more general terms, we have a multiset of 100 tiles of 27 types, and we want to enumerate all possible 7-subsets. The following Python code does this, representing a multiset (or subset) as a list of *counts* of each element type. This is of interest mainly because it’s my first occasion to use the yield from syntax:

def multisubsets(multiset, r, prefix=[]): if r == 0: yield prefix + [0] * len(multiset) elif len(multiset) > 0: for k in range(min(multiset[0], r) + 1): yield from multisubsets(multiset[1:], r - k, prefix + [k])

(This and the rest of the Scrabble-specific code are available at the usual location here.)

The final results are shown in the figure below, with convention for each curve similar to the previous figure.

For example, the bottom curve essentially reproduces the earlier bingo analysis of the probability of playing a word of at least (i.e., exactly) 7 letters from a randomly drawn rack. At the other extreme, if we have the entire dictionary at our disposal (with all those short words of >=2 letters), it’s really hard to draw an unplayable rack (probability 0.00788349). Even if we limit ourselves to just the 100 or so most frequently occurring words– many of which are still very short– we can still play 9 out of 10 racks.

In between, suppose that even though we might *know* many short words, ego prevents actually playing anything shorter than, say, 5 letters in the first turn. Now the probabilities start to depend more critically on whether we are an expert who knows most of the words in the official dictionary, or a novice who might only know a third or fewer of them.

Suppose that Alice and Bob play a game for a dollar: they roll a single six-sided die repeatedly, until either:

- Alice wins if they observe each of the six possible faces at least once, or
- Bob wins if they observe any one face six times.

Would you rather be Alice or Bob in this scenario? Or does it matter? You can play a similar game with a deck of playing cards: shuffle the deck, and deal one card at a time from the top of the deck, with Alice winning when all four suits are dealt, and Bob winning when any particular suit is dealt four times. (This version has the advantage of providing a built-in record of the history of deals, in case of argument over who actually wins.) Again, would you rather be Alice or Bob?

It turns out that Alice has a distinct advantage in both games, winning nearly three times more often than Bob in the dice version, and nearly twice as often in the card version. The objective of this post is to describe some interesting mathematics involved in these games, and relate them to the game of Bingo, where a similar phenomenon is described in a recent *Math Horizons* article (see reference below): the winning Bingo card among multiple players is much more likely to have a *horizontal* bingo (all numbers in some row) than vertical (all numbers in some column).

**Bingo with a single card**

First, let’s describe how Bingo works with just one player. A Bingo card is a 5-by-5 grid of numbers, with each column containing 5 numbers randomly selected without replacement from 15 possibilities: the first “B” column is randomly selected from the numbers 1-15, the second “I” column is selected from 16-30, the third column from 31-45, the fourth column from 46-60, and the fifth “O” column from 61-75. An example is shown below.

A “caller” randomly draws, without replacement, from a pool of balls numbered 1 through 75, calling each number in turn as it is drawn, with the player marking the called number if it is present on his or her card. The player wins by marking all 5 squares in any row, column, or diagonal. (One minor wrinkle in this setup is that, in typical American-style Bingo, the center square is “free,” considered to be already marked before any numbers are called.)

It will be useful to generalize this setup with parameters , where each card is with each column selected from possible values, so that standard Bingo corresponds to .

We can compute the probability distribution of the number of draws required for a single card to win. Bill Butler describes one approach, enumerating the possible partially-marked cards and computing the probability of at least one bingo for each such marking.

Alternatively, we can use inclusion-exclusion to compute the cumulative distribution directly, by enumerating just the possible combinations of horizontal, vertical, and diagonal bingos (of which there are 5, 5, and 2, respectively) on a card with marks. In Mathematica:

bingoSets[n_, free_: True, diag_: True] := Module[{card = Partition[Range[n^2], n]}, If[free, card[[Ceiling[n/2], Ceiling[n/2]]] = 0]; DeleteCases[Join[ card, Transpose[card], If[diag, {Diagonal[card], Diagonal[Reverse[card]]}, {}]], 0, Infinity]] bingoCDF[k_, nm_, bingos_] := Module[{j}, 1 - Total@Map[ (j = Length[Union @@ #]; (-1)^Length[#] Binomial[nm - j, k - j]) &, Subsets[bingos]]/Binomial[nm, k]] bingos = bingoSets[5, False, True]; cdf = Table[bingoCDF[k, 75, bingos], {k, 1, 75}];

(Note the optional arguments specifying whether the center square is free, and whether diagonal bingo is allowed. It will be convenient shortly to consider a simplified version of the game, where these two “special” rules are discarded.)

The following figure shows the resulting distribution, with the expected number of 43.546 draws shown in red.

**Independence with 2 or more cards**

Before getting to the “horizontal is more likely than vertical” phenomenon, it’s worth pointing out another non-intuitive aspect of Bingo. If instead of just a single card, we have a game with multiple players, possibly with *thousands* of different cards, what is the distribution of number of draws until someone wins?

If is the cumulative distribution for a *single* card as computed above, then since each of multiple cards is randomly– and *independently–* “generated,” intuitively it seems like the probability of at least one winning bingo among cards in at most draws should be given by

Butler uses exactly this approach. However, this is incorrect; although the *values* in the squares of multiple cards are independent, the presence or absence of winning bingos are not. Perhaps the best way to see this is to consider a “smaller” simplified version of the game, with , so that there are only four equally likely possible distinct cards:

Let’s further simplify the game so that only horizontal and vertical bingos are allowed, with no diagonals. Then the game must end after either two or three draws: with a single card, it ends in two draws with probability 2/3. However, with two cards, the probability of a bingo in two draws is 5/6, not 1-(1-2/3)^2=8/9.

**Horizontal bingos with many cards**

Finally, let’s come back to the initial problem: suppose that there are a large number of players, with so many cards in play that we are effectively guaranteed a winner as soon as either:

- At least one number from each of the column groups is drawn, resulting in a horizontal bingo on some card; or
- At least of possible numbers is drawn from any one particular column group, resulting in a vertical bingo on some card.

(Let’s ignore the free square and diagonal bingos for now; the former is easily handled but unnecessarily complicates the analysis, while the latter would mean that (1) and (2) are not mutually exclusive.)

Then the interesting observation is that a horizontal bingo (1) is over three times more likely to occur than a vertical bingo (2). Furthermore, this setup– Bingo with a large number of cards– is effectively the same as the card and dice games described in the introduction: Bingo is , the card game is , and the dice version is effectively .

The *Math Horizons* article referenced below describes an approach to calculating these probabilities, which involves enumerating integer partitions. However, this problem is ready-made for generating functions, which takes care of the partition house-keeping for us: let’s define

so that, for example, for Bingo with no free square,

Intuitively, each factor corresponds to a column, where each coefficient of indicates the number of ways to draw exactly numbers from that column (with some minimum number from each column specified by ). The overall coefficient of indicates the number of ways to draw numbers in total, *with neither a horizontal nor vertical bingo*.

Then using the notation from the article, the probability of a horizontal bingo on *exactly* the -th draw is

and the probability of a vertical bingo on exactly the -th draw is

The summation

over all possible numbers of draws yields the overall probability of about 0.752 that a horizontal bingo is observed before a vertical one. Similarly, for the card game with , the probability that Alice wins is 22543417/34165005, or about 0.66. For the dice game– which requires a slight modification to the above formulation, left as an exercise for the reader– Alice wins with probability about 0.747.

**Reference:**

- Benjamin, A., Kisenwether, J., and Weiss, B., The Bingo Paradox,
*Math Horizons*,**25**(1) September 2017, p. 18-21 [PDF]

This post was initially motivated by an interesting recent article by Chris Wellons discussing the Blowfish cipher. The Blowfish cipher’s subkeys are initialized with values containing the first 8336 hexadecimal digits of , the idea being that implementers may compute these digits for themselves, rather than trusting the integrity of explicitly provided “random” values.

So, how do we compute hexadecimal– or decimal, for that matter– digits of ? This post describes several methods for computing digits of and other well-known constants, as well as some implementation issues and open questions that I encountered along the way.

**Pi is easy with POSIX**

First, Chris’s implementation of the Blowfish cipher includes a script to automatically generate the code defining the subkeys. The following two lines do most of the work:

cmd='obase=16; scale=10040; a(1) * 4' pi="$(echo "$cmd" | bc -l | tr -d '\n\\' | tail -c+3)"

This computes base 16 digits of as `a(1) * 4`

, or 4 times the arctangent of 1 (i.e., ), using the POSIX arbitrary-precision calculator bc. Simple, neat, end of story.

How might we do the same thing on Windows? There are plenty of approximation formulas and algorithms for computing digits of , but to more precisely specify the requirements I was interested in, is there an algorithm that generates digits of :

- in any given base,
- one digit at a time “indefinitely,” i.e., without committing to a fixed precision ahead of time,
- with a relatively simple implementation,
- using only arbitrary-precision
*integer*arithmetic (such as is built into Python, or maybe C++ with a library)?

**Bailey-Borwein-Plouffe**

The Bailey-Borwein-Plouffe (BBP) formula seems ready-made for our purpose:

This formula has the nice property that it may be used to efficiently compute the -th hexadecimal digit of , without having to compute all of the previous digits along the way. Roughly, the approach is to multiple everything by , then use modular exponentiation to collect and discard the integer part of the sum, leaving the fractional part with enough precision to accurately extract the -th digit.

However, getting the implementation details right can be tricky. For example, this site provides source code and example data containing one million hexadecimal digits of generated using the BBP formula… but roughly one out of every 55 digits or so is incorrect.

But suppose that we don’t want to “look ahead,” but instead want to generate *all* hexadecimal digits of , one after the other from the beginning. Can we still make use of this formula in a simpler way? For example, consider the following Python implementation:

def pi_bbp(): """Conjectured BBP generator of hex digits of pi.""" a, b = 0, 1 k = 0 while True: ak, bk = (120 * k**2 + 151 * k + 47, 512 * k**4 + 1024 * k**3 + 712 * k**2 + 194 * k + 15) a, b = (16 * a * bk + ak * b, b * bk) digit, a = divmod(a, b) yield digit k = k + 1 for digit in pi_bbp(): print('{:x}'.format(digit), end='')

The idea is similar to converting a fraction to a string in a given base: multiply by the base (16 in this case), extract the next digit as the integer part, then repeat with the remaining fractional part. Here is the running fractional part, and is the current term in the BBP summation. (Using the `fractions`

module doesn’t significantly improve readability, and is *much* slower than managing the numerators and denominators directly.)

Now for the interesting part: although this implementation appears to behave correctly– at least for the first 500,000 digits where I stopped testing– it isn’t clear to me that it is *always* correct. That is, I don’t see how to prove that this algorithm will continue to generate correct hexadecimal digits of indefinitely. Perhaps a reader can enlighten me.

(Initial thoughts: Since it’s relatively easy to show that each term in the summation is positive, I think it would suffice to prove that the algorithm never generates an “invalid” hexadecimal digit that is greater than 15. But I don’t see how to do this, either.)

Interestingly, Bailey *et. al.* conjecture (see Reference 1 below) a similar algorithm that they have verified out to 10 million hexadecimal digits. The algorithm involves a strangely similar-but-slightly-different approach and formula:

def pi_bbmw(): """Conjectured BBMW generator of hex digits of pi.""" a, b = 0, 1 k = 0 yield 3 while True: k = k + 1 ak, bk = (120 * k**2 - 89 * k + 16, 512 * k**4 - 1024 * k**3 + 712 * k**2 - 206 * k + 21) a, b = (16 * a * bk + ak * b, b * bk) a = a % b yield 16 * a // b

Unfortunately, this algorithm is slower, requiring one more expensive arbitrary-precision division operation per digit than the BBP version.

**Proven algorithms**

Although the above two algorithms are certainly short and sweet, (1) they only work for generating hexadecimal digits (vs. decimal, for example), and (2) we don’t actually know if they are correct. Fortunately, there are other options.

Gibbons (Reference 2) describes an algorithm that is not only proven correct, but works for generating digits of in any base:

def pi_gibbons(base=10): """Gibbons spigot generator of digits of pi in given base.""" q, r, t, k, n, l = 1, 0, 1, 1, 3, 3 while True: if 4 * q + r - t < n * t: yield n q, r, t, k, n, l = (base * q, base * (r - n * t), t, k, (base * (3 * q + r)) // t - base * n, l) else: q, r, t, k, n, l = (q * k, (2 * q + r) * l, t * l, k + 1, (q * (7 * k + 2) + r * l) // (t * l), l + 2)

The bad news is that this is by far the slowest algorithm that I investigated, nearly an order of magnitude slower than BBP on my laptop.

The good news is that there is at least one other algorithm, that is not only competitive with BBP in terms of throughput, but is also general enough to easily compute– in any base– not just the digits of , but also (the base of the natural logarithm), (the golden ratio), , and others.

The idea is to express the desired value as a generalized continued fraction:

where in particular may be represented as

Then digits may be extracted similarly to the BBP algorithm above: iteratively refine the *convergent* (i.e., approximation) of the continued fraction until the integer part doesn’t change; extract this integer part as the next digit, then multiply the remaining fractional part by the base and continue. In Python:

def continued_fraction(a, b, base=10): """Generate digits of continued fraction a(0)+b(1)/(a(1)+b(2)/(...).""" (p0, q0), (p1, q1) = (a(0), 1), (a(1) * a(0) + b(1), a(1)) k = 1 while True: (d0, r0), (d1, r1) = divmod(p0, q0), divmod(p1, q1) if d0 == d1: yield d1 p0, p1 = base * r0, base * r1 else: k = k + 1 x, y = a(k), b(k) (p0, q0), (p1, q1) = (p1, q1), (x * p1 + y * p0, x * q1 + y * q0)

This approach is handy because not only , but other common constants as well, have generalized continued fraction representations in which the sequences are “nice.” To generate decimal digits of :

for digit in continued_fraction(lambda k: 0 if k == 0 else 2 * k - 1, lambda k: 4 if k == 1 else (k - 1)**2, 10): print(digit, end='')

Or to generate digits of the golden ratio :

for digit in continued_fraction(lambda k: 1, lambda k: 1, 10): print(digit, end='')

**Consuming blocks of a generator**

Finally, once I got around to actually using the above algorithm to try to reproduce Chris’s original code generation script, I accidentally injected a bug that took some thought to figure out. Recall that the Blowfish cipher has a couple of (sets of) subkeys, each populated with a segment of the sequence of hexadecimal digits of . So we would like to extract a block of digits, do something with it, then extract a *subsequent* block of digits, do something else, etc.

A simple way to do this in Python is with the built-in zip function, that takes multiple iterables as arguments, and returns a single generator that outputs tuples of elements from each of the inputs… and “truncates” to the length of the shortest input. In this case, to extract a fixed number of digits of , we just `zip`

the “infinite” digit generator together with a range of the desired length.

To more clearly see what happens, let’s simplify the context a bit and just try to print the first 10 *decimal* digits of in two groups of 5:

digits = continued_fraction(lambda k: 0 if k == 0 else 2 * k - 1, lambda k: 4 if k == 1 else (k - 1)**2, 10) for digit, k in zip(digits, range(5)): print(digit, end='') print() for digit, k in zip(digits, range(5)): print(digit, end='')

This doesn’t work: the resulting output blocks are (31415) and (26535)… but . We “lost” the 9 in the middle.

The problem is that `zip`

evaluates each input iterator in turn, stopping only when one of them is exhausted. In this case, during the 6th iteration of the first loop, we “eat” the 9 from the digit generator *before* we realize that the `range`

iterator is exhausted. When we continue to the second block of 5 digits, we can’t “put back” the 9.

This is easy to fix: just reverse the order of the `zip`

arguments, so the `range`

is exhausted first, *before* eating the extra element of the “real” sequence we’re extracting from.

for k, digit in zip(range(5), digits): print(digit, end='') print() for k, digit in zip(range(5), digits): print(digit, end='')

This works as desired, with output blocks (31415) and (92653).

**References:**

- Bailey, D., Borwein, J., Mattingly, A., Wightwick, G., The Computation of Previously Inaccessible Digits of and Catalan’s Constant,
*Notices of the American Mathematical Society*,**60**(7) 2013, p. 844-854 [PDF] - Gibbons, J., Unbounded Spigot Algorithms for the Digits of Pi,
*American Mathematical Monthly*,**113**(4) April 2006, p. 318-328 [PDF]

A common development approach in MATLAB is to:

- Write MATLAB code until it’s unacceptably slow.
- Replace the slowest code with a C++ implementation called via MATLAB’s MEX interface.
- Goto step 1.

Regression testing the faster MEX implementation against the slower original MATLAB can be difficult. Even a seemingly line-for-line, “direct” translation from MATLAB to C++, when provided with exactly the same numeric inputs, executed on the same machine, with the same default floating-point rounding mode, can still yield drastically different output. How does this happen?

There are three primary causes of such differences, none of them easy to fix. The purpose of this post is just to describe these causes, focusing on one in particular that I learned occurs more frequently than I realized.

**1. The butterfly effect**

This is where the *drastically* different results typically come from. Even if the *inputs* to the MATLAB and MEX implementations are identical, suppose that just one *intermediate* calculation yields even the smallest possible difference in its result… and is followed by a long sequence of *further* calculations using that result. That small initial difference can be greatly magnified in the final output, due to cumulative effects of rounding error. For example:

x = 0.1; for k in 1:100 x = 4 * (1 - x); end % x == 0.37244749676375793 double x = 0.10000000000000002; for (int k = 1; k <= 100; ++k) { x = 4 * (1 - x); } // x == 0.5453779481420313

This example is only contrived in its simplicity, exaggerating the “speed” of divergence with just a few hundred floating-point operations. Consider a more realistic, more complex Monte Carlo simulation of, say, an aircraft autopilot maneuvering in response to simulated sensor inputs. In a *particular* Monte Carlo iteration, the original MATLAB implementation might successfully dampen a severe Dutch roll, while the MEX version might result in a crash (of the aircraft, I mean).

Of course, for this divergence of behavior to occur at all, there must be that *first* difference in the result of an intermediate calculation. So this “butterfly effect” really is just an *effect*— it’s not a cause at all, just a magnified symptom of the two real causes, described below.

**2. Compiler non-determinism**

As far as I know, the MATLAB interpreter is pretty well-behaved and predictable, in the sense that MATLAB source code explicitly specifies the order of arithmetic operations, and the precision with which they are executed. A C++ compiler, on the other hand, has a lot of freedom in how it translates source code into the corresponding sequence of arithmetic operations… and possibly even the precision with which they are executed.

Even if we assume that all operations are carried out in double precision, order of operations can matter; the problem is that this order is not explicitly controlled by the C++ source code being compiled (*edit*: at least for some speed-optimizing compiler settings). For example, when a compiler encounters the statement `double x = a+b+c;`

, it could emit code to effectively calculate , or , which do not necessarily produce the same result. That is, double-precision addition is not associative:

(0.1 + 0.2) + 0.3 == 0.1 + (0.2 + 0.3) % this is false

Worse, explicit parentheses in the source code *may* help, but it doesn’t *have* to.

Another possible problem is intermediate precision. For example, in the process of computing , the intermediate result might be computed in, say, 80-bit precision, before clamping the final sum to 64-bit double precision. This has bitten me in other ways discussed here before; Bruce Dawson has several interesting articles with much more detail on this and other issues with floating-point arithmetic.

**3. Transcendental functions**

So suppose that we are comparing the results of MATLAB and C++ implementations of the same algorithm. We have verified that the numeric inputs are identical, and we have also somehow verified that the arithmetic operations specified by the algorithm are executed in the same order, all in double precision, in both implementations. Yet the output *still* differs between the two.

Another possible– in fact likely– cause of such differences is in the implementation of transcendental functions such as `sin`

, `cos`

, `atan2`

, `exp`

, etc., which are not required by IEEE-754-2008 to be correctly rounded due to the table maker’s dilemma. For example, following is the first actual instance of this problem that I encountered years ago, reproduced here in MATLAB R2017a (on my Windows laptop):

x = 193513.887169782; y = 44414.97148164646; atan2(y, x) == 0.2256108075334872

while the corresponding C++ implementation (still called from MATLAB, built as a MEX function using Microsoft Visual Studio 2015) yields

#include <cmath> ... std::atan2(y, x) == 0.22561080753348722;

The two results differ by an ulp, with (in this case) the MATLAB version being correctly rounded.

(*Rant*: Note that *both* of the above values, despite actually being different, display as the same 0.225610807533487 in the MATLAB command window, which for some reason neglects to provide “round-trip” representations of its native floating-point data type. See here for a function that I find handy when troubleshooting issues like this.)

What I found surprising, after recently exploring this issue in more detail, is that the above example is not an edge case: the MATLAB and C++ `cmath`

implementations of the trigonometric and exponential functions disagree quite frequently– and furthermore, the above example notwithstanding, the C++ implementation tends to be more accurate more of the time, significantly so in the case of `atan2`

and the exponential functions, as the following figure shows.

The setup: on my Windows laptop, with MATLAB R2017a and Microsoft Visual Studio 2015 (with code compiled from MATLAB as a MEX function with the provided default compiler configuration file), I compared each function’s output for 1 million randomly generated inputs– or pairs of inputs in the case of `atan2`

— in the unit interval.

Of those 1 million inputs, I calculated how many yielded different output from MATLAB and C++, where the differences fell into 3 categories:

- Red indicates that MATLAB produced the correctly rounded result, with the exact value
*between*the MATLAB and C++ outputs (i.e., both implementations had an error less than an ulp). - Gray indicates that C++ produced the correctly rounded result, with both implementations having an error of less than an ulp.
- Blue indicates that C++ produced the correctly rounded result,
*between*the exact value and the MATLAB output (i.e., MATLAB had an error*greater*than an ulp).

(A few additional notes on test setup: first, I used random inputs in the unit interval, instead of evenly-spaced values with domains specific to each function, to ensure testing all of the mantissa bits in the input, while still allowing some variation in the exponent. Second, I also tested addition, subtraction, multiplication, division, and square root, just for completeness, and verified that they agree exactly 100% of the time, as guaranteed by IEEE-754-2008… at least for *one* such operation in isolation, eliminating any compiler non-determinism mentioned earlier. And finally, I also tested against MEX-compiled C++ using MinGW-w64, with results identical to MSVC++.)

For most of these functions, the inputs where MATLAB and C++ differ are distributed across the entire unit interval. The two-argument form of the arctangent is particularly interesting. The following figure “zooms in” on the 16.9597% of the sample inputs that yielded different outputs, showing a scatterplot of those inputs using the same color coding as above. (The red, where MATLAB provides the correctly rounded result, is so infrequent it is barely visible in the summary bar chart.)

**Conclusion**

This might all seem rather grim. It certainly does seem hopeless to expect that a C++ translation of even modestly complex MATLAB code will preserve exact agreement of output for all inputs. (In fact, it’s even worse than this. Even the same MATLAB code, executed in the same version of MATLAB, but on two different machines with different architectures, will not necessarily produce identical results given identical inputs.)

But exact agreement between different implementations is rarely the “real” requirement. Recall the Monte Carlo simulation example described above. If we run, say, 1000 iterations of a simulation in MATLAB, and the same 1000 iterations with a MEX translation, it may be that iteration #137 yields different output in the two versions. But that *should* be okay… as long as the *distribution* over all 1000 outputs is the same– or sufficiently similar– in both cases.