## Anagrams

Introduction

This was a fun exercise, motivated by several interesting recent posts by Mark Dominus at The Universe of Discourse about finding anagrams of individual English words, such as (relationshipsrhinoplasties), and how to compute a “score” for such anagrams by some reasonable measure of the complexity of the rearrangement, so that (attentivenesstentativeness), with a common 8-letter suffix, may be viewed as less “interesting” than, say, the more thoroughly shuffled (microclimates, commercialist).

The proposed scoring metric is the size of a “minimum common string partition” (MCSP): what is the fewest number of blocks of consecutive letters in a partition of the first word that may be permuted and re-concatenated to yield the second word?  For example, the above word attentiveness may be partitioned into 3 blocks, at+tent+iveness, and transposing the first two blocks yields tent+at+iveness.  Thus, the score for this anagram is only 3.  Compare this with the score of 12 for (intolerances, crenelations), where all 12 letters must be rearranged.

Computing MCSP

I wanted to experiment with this idea in a couple of different ways.  First, as Mark points out, the task of finding the anagrams themselves is pretty straightforward, but computing the resulting MCSP scores is NP-complete.  Fortunately, there is a nice characterization of the solution– essentially the same “brute force” approach described by Mark– that allows concise and reasonably efficient implementation.

Consider an anagram of two words $(w_1, w_2)$ with $n$ letters each, where the necessary rearrangement of letters of $w_1$ to produce $w_2$ is specified by a permutation

$\pi:\{1,2,\ldots,n\} \rightarrow \{1,2,\ldots,n\}$

where the $i$-th letter in $w_2$ is the $\pi(i)$-th letter in $w_1$.  This permutation of individual letters corresponds to a permutation of blocks of consecutive letters, where the number of such blocks– the MCSP score– is

$s(\pi) = n - \left|\{i

Computing an MCSP is hard because this permutation transforming $w_1$ into $w_2$ is not necessarily unique; we need the permutation that minimizes $s(\pi)$.  The key observation is that each candidate permutation may be decomposed into $\pi = \pi_2 \pi_1^{-1}$, where $\pi_j$ transforms any canonical (e.g., sorted) ordering of letters into $w_j$.  So we can fix, say, $\pi_2$, and the enumeration of possible $\pi_1$ is easy to express, since we are using the sorted list of letters as our starting point.

The following Mathematica function implements this approach:

anagramScore[w1_String, w2_String] :=
Module[
{s1 = Characters[w1], s2 = Characters[w2], p1, p2, i},
p1 = Ordering@Ordering[s1];
p2 = Ordering@Ordering[s2];
Length[s1] - Max@Outer[
Count[
Differences[Ordering[ReplacePart[p1, {##} // Flatten]][[p2]]],
1] &,
Sequence @@ Map[
(i = Position[s1, #] // Flatten;
Thread[i -> #] & /@ Permutations[p1[[i]]]) &,
Union[s1]
], 1]]


Using this, we find, as Mark does, that an anagram with maximum MCSP score of 14 is (cinematographer, megachiropteran)… along with the almost-as-interesting (involuntariness, nonuniversalist), but also other fun ones farther down the list, such as (enthusiastic, unchastities) with a score of 9.

Scoring Anagrams Using MCSP and Frequency

From Mark’s post:

Clearly my chunk score is not the end of the story, because “notaries / senorita” should score better than “abets / baste” (which is boring) or “Acephali / Phacelia” (whatever those are), also 5-pointers. The length of the words should be worth something, and the familiarity of the words should be worth even more [my emphasis].

The problem is that an MCSP score alone is a pretty coarse metric, since it’s an integer bounded by the length of the words in the dictionary.  So the second idea was to refine the ordering of the list of anagrams as Mark suggests, with a lexicographic sort first by MCSP score, then by (average) frequency of occurrence in language, as estimated using the Google Books Ngrams data set (methodology described in more detail here).  The expectation was that this would make browsing a long list easier, with more “recognizable” anagrams appearing together near the beginning of each MCSP grouping.

However, because I wanted to try to reproduce Mark’s results, I also needed a larger dictionary that contained, for example, megachiropteran. (which, by the way, is a bat that can have a wing span of over 5 feet).  I used the American English version of the Spell Checker Oriented Word List (SCOWL), combined with the Scrabble and ENABLE2k word lists used in similar previous experiments– which, interestingly, alone contain many anagrams not found in the earlier list.  (The SCOWL was really only needed to “reach” megachiropteran; with the exception of it and nonuniversalist, all of the other examples in this post are valid Scrabble words!).  The resulting word lists and corresponding counts of occurrences in the Ngrams data set are available here.

The resulting list of anagrams are in the text file linked below, sorted by MCSP score, then by the average frequency of the pair of words in each anagram.  Interesting examples high on the list are (personality, antileprosy) with a score of 11, (industries, disuniters) with a score of 10, etc.

The full list of 82,776 anagrams sorted by MCSP and frequency

The individual word frequencies are included in the list, to allow investigation of other sorting methods.  For example, it might be useful to normalize MCSP score by word length.  Or instead of using the average frequency of the two words in an anagram, the ratio of frequencies would more heavily weight anagrams between a really common word and a relatively unknown one, such as (penalties, antisleep)– I have never heard of the latter, but both are Scrabble-playable.

References:

1. Dominus, M., I found the best anagram in English, The Universe of Discourse, 21 February 2017 [HTML]
2. Goldstein, A., Kolman, P., Zheng, J., Minimum Common String Partition Problem: Hardness and Approximations, Electronic Journal of Combinatorics, 12 (2005), #R50 [PDF]

## Array resizing in MATLAB

I encountered the following MATLAB code recently, simplified for this discussion; it builds a 3-by-4-by-2 array by assigning each of its three 4-by-2 “block” sub-arrays in turn to an initially empty array:

sz = [3, 4, 2];
block = reshape(0:(prod(sz(2:end)) - 1), sz(2:end));
a = [];
for k = 1:sz(1)
a(k,:,:) = block; block = block + numel(block);
end


As this example shows, MATLAB allows resizing arrays on the fly, so to speak; assignment to an element– or in this case, a range of elements– whose indices would normally be out of bounds automatically re-allocates memory to accommodate the new, larger array.  However, these incremental re-allocation and re-copying operations can be costly in terms of execution time.  Granted, this toy example is small enough not to matter, but in the “real” example, the eventual array size was approximately sz=[256,16384,4], in which case executing the above code takes about 8 seconds on my laptop.

Pre-allocation is usually recommended as a fix: that is, size the entire array in advance, before assigning any elements.  But what if you don’t know the size of the array in advance?  Although there are several approaches to dealing with this situation, with varying complexity, the subject of this post is to describe just how much execution time may be eliminated by only a modest change to the above code.

A major source of slowness is that the array expansion occurs along the first dimension… which in MATLAB, with its Fortran roots, is the index that changes most frequently in the underlying block of memory containing the array elements.  So not only must we re-allocate memory to accommodate each new sub-array, even copying the existing array elements is a non-trivial task, as the following figure shows.  As we “append” first the cyan block, then the red block, then the green block, each incremental re-allocation also requires “re-interleaving” the old and new elements in the new larger block of memory:

Expanding an eventual 3x4x2 array in MATLAB along the first dimension.

Instead, consider the following code, modified to append each block along the last— and most slowly changing– dimension, followed by the appropriate transposition via permute to shuffle everything back into the originally intended shape:

a = [];
for k = 1:sz(1)
a(:,:,k) = block; block = block + numel(block);
end
a = permute(a, [3, 1, 2]);


The effect of this change is that, although we still incur all of the incremental re-allocation cost, each memory copy operation is a more straightforward and much faster “blit”:

Expanding an eventual 3x4x2 array in MATLAB along the last dimension, followed by transposing to yield the desired order of dimensions.

This version executes in less than a quarter of a second on my laptop, about 35 times faster than the original 8 seconds.

Posted in Uncategorized | 4 Comments

## Guess the number

I haven’t posted a puzzle in a while.  The following problem has the usual nice characteristics; it works on a cocktail napkin or as a programming problem, via exact solution or simulation, etc.

I am thinking of a randomly selected integer between 1 and $m=10$ (inclusive).  You are the first of $n=3$ players who will each, in turn, get a single guess at the selected number.  The player whose guess is closest wins $300, with ties splitting the winnings evenly. Here is the catch: each player may not guess a number that has already been guessed by a previous player. As the first player, what should your strategy be? Which player, if any, has the advantage? And what happens if we vary $m$ and $n$? Posted in Uncategorized | Leave a comment ## Probability of a Scrabble bingo My wife and I have been playing Scrabble recently. She is much better at the game than I am, which seems to be the case with most games we play. But neither of us are experts, so that bingos— playing all 7 tiles from the rack in a single turn, for a 50-point bonus– are rare. I wondered just how rare they should be… accounting for the fact that I am a novice player? Let’s focus the problem a bit, and just consider the first turn of the game, when there are no other tiles on the board: what is the probability that 7 randomly drawn Scrabble tiles may be played to form a valid 7-letter word? There are ${100 \choose 7}$, or over 16 billion equally likely ways to draw a rack of 7 tiles from the 100 tiles in the North American version of the game. But since some tiles are duplicated, there are only 3,199,724 distinct possible racks (not necessarily equally likely). Which of these racks form valid words? It depends on what we mean by valid. According to the 2014 Official Tournament and Club Word List (the latest for which an electronic version is accessible), there are 25,257 playable words with 7 letters… but many of those are words that I don’t even know, let alone expect to be able to recognize from a scrambled rack of tiles. We need a way to reduce this over-long official list of words down to a “novice” list of words– or better yet, rank the entire list from “easiest” to “hardest,” and compute the desired probability as a function of the size of the accepted dictionary. The Google Books Ngrams data set (English version 20120701) provides a means of doing this. As we have done before (described here and here), we can map each 7-letter Scrabble word to its frequency of occurrence in the Google Books corpus, the idea being that “easier” words occur more frequently than “harder” words. The following figure shows the sorted number of occurrences of all 7-letter Scrabble words on a logarithmic scale, with some highlighted examples, ranging from between, the single most frequently occurring 7-letter Scrabble word, to simioid, one of the least frequently occurring words… and this doesn’t even include the 1444 playable words– about 5.7% of the total– that appear nowhere in the entire corpus, such as abaxile and zygoses. Scrabble 7-letter words ranked by frequency of occurrence in Google Books Ngrams data set. The least frequent word shown here that I recognize is “predate.” Armed with this sorted list of 25,257 words, we can now compute, as a function of $n \leq 25257$, the probability that a randomly drawn rack of 7 tiles may be played to form one of the $n$ easiest words in the list. Following is Mathematica code to compute these probabilities. This would be slightly simpler– and much more efficient– if not for the wrinkle of dealing with blank tiles, which allow multiple different words to be played from the same rack of tiles. tiles = {" " -> 2, "a" -> 9, "b" -> 2, "c" -> 2, "d" -> 4, "e" -> 12, "f" -> 2, "g" -> 3, "h" -> 2, "i" -> 9, "j" -> 1, "k" -> 1, "l" -> 4, "m" -> 2, "n" -> 6, "o" -> 8, "p" -> 2, "q" -> 1, "r" -> 6, "s" -> 4, "t" -> 6, "u" -> 4, "v" -> 2, "w" -> 2, "x" -> 1, "y" -> 2, "z" -> 1}; {numBlanks, numTiles} = {" " /. tiles, Total[Last /@ tiles]}; racks[w_String] := Map[ StringJoin@Sort@Characters@StringReplacePart[w, " ", #] &, Map[{#, #} &, Subsets[Range[7], numBlanks], {2}]] draws[r_String] := Times @@ Binomial @@ Transpose[Tally@Characters[r] /. tiles] all = {}; p = Accumulate@Map[( new = Complement[racks[#], all]; all = Join[all, new]; Total[draws /@ new] ) &, words] / Binomial[numTiles, 7];  The results are shown in the following figure, along with another sampling of specific playable words. For example, if we include the entire official word list, the probability of drawing a playable 7-letter word is 21226189/160075608, or about 0.132601. Probability that 7 randomly drawn tiles form a word, vs. dictionary size. A coarse inspection of the list suggests that I confidently recognize only about 8 or 9 thousand– roughly a third– of the available words, meaning that my probability of playing all 7 of my tiles on the first turn is only about 0.07. In other words, instead of a first-turn bingo every 7.5 games or so on average, I should expect to have to wait nearly twice as long. We’ll see if I’m even that good. Posted in Uncategorized | Leave a comment ## Risk of (gambler’s) ruin Suppose that you start with an initial bankroll of $m$ dollars, and repeatedly make a wager that pays$1 with probability $p$, and loses $1 with probability $1-p$. What is the risk of ruin, i.e., the probability that you will eventually go broke? This is the so-called gambler’s ruin problem. It is a relatively common exercise to show that if $p \leq 1/2$, the probability of ruin is 1, and if $p > 1/2$, then the probability is $(\frac{1-p}{p})^m$ But what if the wager is not just win-or-lose a dollar, but is instead specified by an arbitrary probability distribution of outcomes? For example, suppose that at each iteration, we may win any of $(-2,-1,0,+1,+2)$ units, with respective probabilities $(1/15,2/15,3/15,4/15,5/15)$. The purpose of this post is to capture my notes on some seemingly less well-known results in this more general case. (The application to my current study of blackjack betting is clear: we have shown that, at least for a shoe game, even if we play perfectly, we are still going to lose if we don’t vary our bet. We can increase our win rate by betting more in favorable situations… but a natural constraint is to limit our risk of ruin, or probability of going broke.) Schlesinger (see Reference (2) below) gives the following formula for risk of ruin, due to “George C., published on p. 8 of ‘How to Win$1 Million Playing Casino Blackjack'”:

$(\frac{1 - \frac{\mu}{\sigma}}{1 + \frac{\mu}{\sigma}})^\frac{m}{\sigma}$

where $\mu$ and $\sigma$ are the mean and standard deviation, respectively, of the outcome of each round (or hourly winnings).  It is worth emphasizing, since it was unclear to me from the text, that this formula is an approximation, albeit a pretty good one.  The derivation is not given, but the approach is simple to describe: normalize the units of both bankroll and outcome of rounds to have unit variance (i.e., divide everything by $\sigma$), then use the standard two-outcome ruin probability formula above with win probability $p$ chosen to reflect the appropriate expected value of the round, i.e., $p - (1-p) = \mu / \sigma$.

The unstated assumption is that $0 < \mu < \sigma$ (note that ruin is guaranteed if $\mu < 0$, or if $\mu = 0$ and $\sigma > 0$), and that accuracy of the approximation depends on $\mu \ll \sigma \ll m$, which is fortunately generally the case in blackjack.

There is an exact formula for risk of ruin, at least as long as outcomes of each round are bounded and integral.  In Reference (1) below, Katriel describes a formula involving the roots inside the complex unit disk of the equation

$\sum p_k z^k = 1$

where $p_k$ is the probability of winning $k$ units in each round.  Execution time and numeric stability make effective implementation tricky.

Finally, just to have some data to go along with the equations, following is an example of applying these ideas to analysis of optimal betting in blackjack.  Considering the same rules and setup as in the most recent posts (6 decks, S17, DOA, DAS, SPL1, no surrender, 75% penetration), let’s evaluate all possible betting ramps with a 1-16 spread through a maximum (floored) true count of 10, for each of five different betting and playing strategies, ranging from simplest to most complex:

1. Fixed basic “full-shoe” total-dependent zero-memory strategy (TDZ), using Hi-Lo true count for betting only.
2. Hi-Lo with the Illustrious 18 indices.
3. Hi-Lo with full indices.
4. Hi-Opt II with full indices and ace side count.
5. “Optimal” betting and playing strategy, where playing strategy is CDZ- optimized for each pre-deal depleted shoe, and betting strategy is ramped according to the corresponding exact pre-deal expected return.

Then assuming common “standard” values of $10,000 initial bankroll, a$10 minimum bet, and 100 hands per hour, the following figure shows the achievable win rate ($per hour) and corresponding risk of ruin for each possible strategy and betting ramp: Win rate vs. risk of ruin for various betting and playing strategies. There are plenty of interesting things to note and investigate here. The idea is that we can pick a maximum acceptable risk of ruin– such as the red line in the figure, indicating the standard Kelly-derived value of $1/e^2$, or about 13.5%– and find the betting ramp that maximizes win rate without exceeding that risk of ruin. Those best win rates for this particular setup are: 1. Not achievable for TDZ (see below). 2.$20.16/hr for Hi-Lo I18.
3. $21.18/hr for Hi-Lo full. 4.$26.51/hr for Hi-Opt II.
5. \$33.09/hr for optimal play.

Fixed basic TDZ strategy, shown in purple, just isn’t good enough; that is, there is no betting ramp with a risk of ruin smaller than about 15%.  And some betting ramps, even with the 1-16 spread constraint, still yield a negative overall expected return, resulting in the “tail” at P(ruin)=1.  (But that’s using Hi-Lo true count as the “input” to the ramp; it is possible that “perfect” betting using exact pre-deal expected return could do better.)

References:

1. Katriel, Guy, Gambler’s ruin probability – a general formula, arXiv:1209.4203v4 [math.PR], 2 July 2013
2. Schlesinger, Don, Blackjack Attack: Playing the Pros’ Way, 3rd ed. Las Vegas: RGE Publishing, Ltd., 2005

## A harder birthday problem

It is a well-known non-intuitive result that in a group of $n=23$ people– conveniently the size of a classroom of students– the probability is at least 1/2 that $k=2$ or more of them share a birthday.  This is a nice problem for several reasons:

1. Its solution involves looking at the problem in a non-obvious way, in this case by considering the complementary event that all birthdays are distinct.
2. Once the approach in (1) is understood, computing the answer is relatively easy: the probability is $1-(365)_{n}/365^n$, where $(x)_{n}$ is the falling factorial.
3. The answer is surprising.  When I ask students, “How many people are needed for the probability of a shared birthday to exceed 1/2?”, guesses as high as 180 are common.
4. Picking on the not-quite-realistic assumption of all $d=365$ birthdays being equally likely actually helps; that is, with a non-uniform distribution of birthdays, the probability of coincidence is higher.

But what about larger $k$?  For example, suppose that while surveying your students’ birthdays in preparation for this problem, you find that three of them share a birthday?  What is the probability of this happening?

The Wikipedia page on the birthday problem doesn’t address this generalization at all.  There are several blog and forum posts addressing the $k=3$ case specifically, by grouping the prohibited cases according to the number of pairs of people with shared birthdays.  This is generalized to arbitrary $k \geq 2$ on the Wolfram MathWorld page; the resulting recurrence relation is pretty complex, but it’s a nice exercise to prove that it works.

Probability that at least (2,3,4,5) people share a birthday, vs. group size.

The motivation for this post is to describe what I think is a relatively simpler solution, for arbitrary $k$, including Python source code to perform the calculation.  Let’s fix the number of equally likely possible birthdays $d=365$, and the desired number $k$ of people sharing a birthday, and define the function

$G(x)=(1+x+\frac{x^2}{2} \ldots + \frac{x^{k-1}}{(k-1)!})^d$

Then $G(x)$ is the exponential generating function for the number of “prohibited” assignments of birthdays to $n$ people where no more than $k-1$ share a birthday.  That is, the number of such prohibited assignments is $n!$ times the coefficient of $x^n$ in $G(x)$.

(When working through why this works, it’s interesting how often it can be helpful to transform the problem into a different context.  For example, in this case, we are also counting the number of length-$n$ strings over an alphabet of $d$ characters, where no character appears more than $k-1$ times.)

The rest of the calculation follows in the usual manner: divide by the total number of possible assignments $d^n$ to get the complementary probability, then subtract from 1.  The following Python code performs this calculation, either exactly– using the fractions module, which can take a while– or in double precision, which is much faster.

import numpy as np
from numpy.polynomial import polynomial as P
import fractions
import operator

def p(k, n, d=365, exact=False):
f = fractions.Fraction if exact else operator.truediv
q = P.polypow(np.array(
[f(1, np.math.factorial(j)) for j in range(k)], dtype=object),
d)[n]
for j in range(1, n + 1):
q = q * f(j, d)
return 1 - q


## Strong induction

What is the best way to explain induction to a student?  That is, given a true-or-false statement $P(n)$ involving a natural number $n \geq 0$, we would like to prove that the statement is true for all such $n \geq 0$.  How does one prove such a claim, when it seems to require checking an infinite number of cases?  My motivation for this thinking-out-loud post involves a particular form of induction whose presentation in some textbooks may cause students some confusion (or at least, it did cause confusion in at least one student).

Often the first exposure to induction is the so-called “weak” form: to show that $P(n)$ is true for all $n \geq 0$, it suffices to prove the following two claims:

• Base step: $P(0)$ is true.
• Induction step: For all $n \geq 0$, if $P(n)$ is true, then $P(n+1)$ is true.

That is,

$(P(0) \land \forall n \geq 0 P(n) \Rightarrow P(n+1)) \Rightarrow \forall n \geq 0 P(n)$

So far, so good.  Although this is technically all that we need, that this is referred to as “weak” induction suggests that there is a stronger form.  There is… sort of.  In his Applied Combinatorics (see Reference 1 below), Tucker skips weak induction altogether, presenting induction in the following “strong” form:

• Base step: $P(0)$ is true.
• Induction step: If $P(0), P(1), P(2), \ldots, P(n-1)$ are true, then $P(n)$ is true.

The idea is that the induction hypothesis is stronger, which can make the resulting proof simpler; in the process of proving $P(n)$ in the induction step, we have at our disposal not only the assumed truth of $P(n-1)$, but also all of the other previous statements.  (I say “sort of,” and wrap “weak” and “strong” in quotes, because the two forms are actually equivalent.  Strong induction isn’t really any stronger in the sense of more “proving power”; statements provable using strong induction are also provable by weak induction, and vice versa.)

Notice that the induction step above, quoted directly from Tucker, lacks some rigor in that there is no explicit universal quantifier for $n$.  That is, for what values of $n$ do we need to demonstrate the induction step?  This is important, because it turns out that if we are more careful, we can economize a bit and demonstrate that $P(n)$ is true for all $n \geq 0$ by proving the single more compact claim:

• Strong induction “single-step”: For all $n \geq 0$, if $P(k)$ is true for all $k, then $P(n)$ is also true.

That is,

$(\forall n \geq 0 (\forall k

Velleman (Reference 2) takes this presentation approach, which certainly has elegance going for it.  But he then goes on to “note that no base case is necessary in a proof by strong induction [my emphasis].”  This is where I think things can get confusing.  It’s certainly true that, once we have demonstrated the implication in the single-step version above, then we know that $P(0)$ is true.  As Velleman explains, “plugging in 0 for $n$, we can conclude that $(\forall k<0 P(k)) \Rightarrow P(0)$.  But because there are no natural numbers smaller than 0, the statement $\forall k<0 P(k)$ is vacuously true.  Therefore, by modus ponens, $P(0)$ is true.  (This explains why the base case doesn’t have to be checked separately in a proof by strong induction [my emphasis]; the base case $P(0)$ actually follows from the modified form of the induction step used in strong induction.)”

There is nothing strictly incorrect here.  What may be misleading, though, is the repeated assurance that no special attention is required for treatment of the base case.  When writing the actual proof of the implication in the single-step version of strong induction, the argument may “look different” for $n=0$ than it does for $n>0$.  I think Wikipedia actually does the most admirable job of explaining this wrinkle: “Sometimes the same argument applies for $n=0$ and $n>0$, making the proof simpler and more elegant.  In this method it is, however, vital to ensure that the proof of $P(n)$ does not implicitly assume that $n>0$, e.g. by saying “choose an arbitrary $k” or assuming that a set of $n$ elements has an element [my emphasis added].”

From a teaching perspective, what are good examples of induction proofs that highlight these issues?  This is a good question.  When discussing strong induction, the most common example seems to be the existence part of the fundamental theorem of arithmetic: every integer greater than 1 is the product of one or more primes.  Here we really can make just one argument, so to speak, without treating the first prime number 2 as a special base case (interestingly, Wikipedia does address it separately).

In a course on graph theory, I think another nice example is the proof that any round-robin tournament contains a Hamiltonian path.  Again, the single-step version of a strong induction argument works here as well (although we have to be a little more careful about what happens when some constructed subsets of vertices end up being empty).

Finally, another common example where strong induction is useful, but special treatment of base case(s) is required, involves postage stamps: show that it is possible to use 4- and 5-cent stamps to form every amount of postage of 12 cents or more.

References:

1. Tucker, Alan, Applied Combinatorics, 6th ed. New York: Wiley and Sons, 2012
2. Velleman, Daniel J., How to Prove It: A Structured Approach. New York: Cambridge University Press, 2006