Probabilistic Analysis and Randomized Algorithms

B403: Introduction to Algorithm Design and Analysis

Motivating Example

   Hire-Assistant (n)

   1  best = 0  // candidate 0 is a dummy, least-qualified
   2  for i = 1 to n
   3    interview candidate i
   4    if candidate i is better than candidate best
   5       best = i
   6       hire candidate i

What is the running time of this algorithm?

O(cin + chm)
Worst case: O(chn)

An average running time would be more useful in this case

Motivating Example

   Randomized-Hire-Assistant (n)

   1  randomly permute the list of candidates
   2  best = 0  // candidate 0 is a dummy, least-qualified
   3  for i = 1 to n
   4    interview candidate i
   5    if candidate i is better than candidate best
   6       best = i
   7       hire candidate i

Basics of Probability Theory

  • A sample space S is a set of elementary events
    • Each elementary event is an outcome of an experiment
    • For example, for an experiment of flipping two coins, the sample space, S = {HH, HT, TH, TT}
  • An event is a subset of the sample space S.
  • Note that:
    • Event S is certain event (probability 1)
    • Event φ is null event (probability 0)
    • Events A and B are mutually exclusive if A ∩ B = φ

Axioms of Probability

  1. Pr{A} ≥ 0 for any event A
  2. Pr{S} = 1
  3. Pr{A ∪ B} = Pr{A} + Pr{B} for any two mutually exclusive events
  4. Pr{A ∪ B} = Pr{A} + Pr{B} − Pr{A ∩ B}, in general

Conditional Probability

The conditional probability of an event A given that another event B occurs is defined to be:

Pr{A | B} = Pr{A ∩ B} Pr{B}

Two events are independent if

Pr{A ∩ B} = Pr{A} Pr{B}

If Pr{B} ≠ 0, this is equivalent to:

Pr{A | B} = Pr{A}

Bayes's Theorem

Pr{A ∩ B} = Pr{B} Pr{A | B}
= Pr{A} Pr{B | A}

Solving for Pr{A | B}, we get Bayes's Theorem:

Pr{A | B} = Pr{A} Pr{B | A} Pr{B}

Example 1

There are two envelopes in front of you. One contains x dollars and the other contains 2x dollars. You choose one, open it, and get $100. You are then offered a switch if you want it. You can take the other envelope instead. Here are two arguments:

  1. There were 2 envelopes, one contained x dollars and the other 2x. You got either x or 2x. Each was equally likely, so switching doesn’t help.
  2. The other envelope contains either $200 or $50. These are equally likely. So the expected value of the other envelope is ($200 + $50)/2 = $125. You should switch.

Which if any of these arguments do you believe? What is the bug in the other argument?

Example 2

You are a contestant in a game show in which a prize is hidden behind one of three curtains. You will win the prize if you select the correct curtain. After you have picked one curtain out before the curtain is lifted, the emcee lifts one of the other curtains, revealing an empty stage, and asks if you would like to switch from your current selection to the remaining curtain. How will your chances change if you switch?

(The celebrated Monty Hall problem.)

http://www.marilynvossavant.com/articles/gameshow.html

Example 3

  • Mr. Jones has two children. The older child is a girl. What is the probability that both children are girls?
  • Mr. Smith has two children. At least one of them is a boy. What is the probability that both children are boys?
  • From all families with two children, at least one of whom is a boy, a family is chosen at random. This would yield the answer of 1/3.
  • From all families with two children, one child is selected at random, and the sex of that child is specified. This would yield an answer of 1/2.

http://en.wikipedia.org/wiki/Boy_or_Girl_paradox

Discrete Random Variables

A discrete random variable X is a function from finite or countably infinite sample space S to the real numbers. Thus, it associates a real number with each possible outcome of an experiment.

For a random variable X and a real number x, we define the event X = x to be {s ∈ S : X(s) = x}

Thus: Pr{X = x} = Σ   s∈S:X(s)=x Pr{s}

The function f(x) = Pr{X = x} is the probability density function of the random variable X. Thus:

Σ   x Pr{X = x} = 1

Expected Value

The expected value (also called expectation or mean) of a discrete random variable X is

E[X] = Σx x⋅Pr{X = x}

E[X] is well-defined as long as the summation is finite or converges. Expectation is usually denoted by μX.

Suppose that two fair coins are flipped and you earn $3 for each head but lose $2 for each tail. Your average earnings is the expectation of the random variable X representing your earnings:

E[X] = 6⋅Pr{2 H's} + 1⋅Pr{1 H, 1 T} − 4.Pr{2 T's}
= 6(1/4) + 1(1/2) − 4(1/4) = 1

Linearity of Expectations

  • E[X + Y] = E[X] + E[Y]

  • If X is any random variable, then any function g(x) defines a new random variable g(X). The expectation of g(X) is:

    E[g(X)] = Σx g(x)⋅Pr{X = x}

  • If we let g(x) = ax, we have for any constant s

    E[aX] = aE[X]

  • For any two random variables X and Y and any constant a

    E[aX + Y] = aE[X] + E[Y]

Randomly Permuting Arrays

   Permute-By-Sorting (A)

   1  n = A.length
   2  let P[1..n] be a new array
   3  for i = 1 to n
   4    P[i] = Random(1, n3)
   5  sort A, using P as sort keys

Permuting Arrays: Another (Possible) Way

   Permute-By-Cyclic (A)

   1  n = A.length
   2  let B[1..n] be a new array
   3  offset = Random(1, n)
   4  for i = 1 to n
   5    dest = i + offset
   6    if dest > n
   7      dest = dest - n
   8    B[dest] = A[i]
   9  return B

Does this work? Why or why not?

You need to be extra careful in making sure that you get the random probability distribution you want. See the solution to Exercise 5.3-4 online on the textbook companion site for an explanation of why the above algorithm does not work.

Permuting in Place

   Randomize-In-Place (A)

   1  n = A.length
   2  for i = 1 to n
   3    swap A[i] with A[Random(i, n)]

Permuting in Place: Another (Possible) Way

   Permute-Without-Identity (A)

   1  n = A,length
   2  for i = 1 to n−1
   3    swap A[i] with A[Random(i+1, n)]