Alice: My algorithm is faster!

Bob: No, mine is!

Where will this lead? Will Alice resort to silencing Bob? Will Bob just give up? Is there any way they can settle their dispute? Find out today with...

(with special guest star Spot!)

Approach 1: Implement and Test

Alce and Bob could program their algorithms and try them out on some sample inputs.

Bob: But my algorithm is too complicated to implement if we're just going to throw it away!

Alice: Maybe we'll miss some inputs on which your algorithm is bad!

Spot: Arf! [translation: Maybe Alice's algorithm will look better on small test inputs, but when we get to the really big, interesting problems, Bob's is better.]

Eve: Doesn't this depend on which machine you use?

This isn't really a bad idea, but they have a point.

Approach 2: Graph and Extrapolate

Graph performance against problem size, and extrapolate.

     |                  .
     |     current      .
     |    computing     .
     |      power -+   .
     |     limits  |   .
     |             |  .
  ^  |             | .
  |  |             v.
time |            _~
     |           -
     |         _~
     |      _-~
     |___--~
     |
     +---------------------------
          problem size -->

Bob: But how do I know if the extrapolation is correct?

It would be better if we didn't treat the algorithm as a black box.

Approach 2: Create a formula

Write down the algorithm and create a formula for the running time.

  Prime-Test-All

       i <- 2
         |
         v          no
 +-> i <= sqrt(N)? -----> output IS_PRIME
 |       |
 |       | yes
 |       v           yes
 |  does i divide N? ---> output NOT_PRIME
 |       |
 |       | no
 |       v
 +--- i <- i + 1

The time consumed by Prime-Test-All for input N is at most

T[PTA](N) <= T[i <- 2] + (sqrt(N) - 1) (T[i <= sqrt(N)?] + T[i divide N?]
                                          + T[i <- i + 1]) + T[output]

Bob: No implementation!

Alice: But what a mess!

Approach 3: Approximate

For really big inputs, we can ignore everything but the fastest-growing term: T[PTA](N) <= c[PTA] sqrt(N) (where c[PTA] is a constant that will change between machines).

Say Bob's algorithm takes T[B](N) <= c[B] sqrt(N) log_2(N) time.

For what N is Bob's algorithm better?

Ignore the Constants

To compare algorithms with very different bounds, we can just ignore the constants. We write T[PTA] = O(sqrt(N)) or T[B] = O(sqrt(N) log_2(N)).

This is called big-O notation.

Alice: Aren't you simplifying a bit too much?

Yes, this is a gross simplification. But it often works!

Practice with Big-O

  50 x^2 + 25 x + 40 = O(x^2)

  5,096 log_2 n + 0.02 n = O(n)

  4,236,121 = O(1)

  4 * 2^n log_2 n + n^2 = O(2^n log_2 n)

Going from Pseudocode

  Algorithm Prime-Test-Odds(n)
  if n = 2 then return YES fi
  if n is even then return NO fi
  for each odd i such that 3 <= i <= sqrt(n) do
    if i divides n then
      return NO
    fi
  od
  return YES

  O(sqrt(n))

Going from Java

  public static int AddIntegers(int[] result, int[] a, int[] b) {
    int i;
    int carry;

    carry = 0;
    for(i = 0; i < a.length; i = i + 1) {
      result[i] = (a[i] + b[i] + carry) % 2;
      carry = (a[i] + b[i] + carry) / 2;
    }
    if(carry > 0) {
      return -1;
    } else {
      return 0;
    }
  }

  O(a.length)

More examples

  Algorithm Exponentiate(x, n)
  result <- 1
  for i <- 1 to n do
    result <- result * x
  od
  return result

  O(n)

  Algorithm Add-Matrices(a, b)
  for i <- 1 to m do
    for j <- 1 to n do
      c[i,j] <- a[i,j] + b[i,j]
    od
  od
  return c

  O(mn)

  Algorithm Fast-Exponentiate(x, n)
  result <- 1
  y <- x
  i <- n
  while i > 0 do
    if i is even then
      i <- i / 2
    else
      result <- result * y
      i <- (i - 1) / 2
    fi
    y <- y^2
  od
  return result

  O(log_2 n)

More examples

We multiply two binary numbers a and b as on the Web page.

 a =     1101 (= 13 base 10)
 b =     1011 (= 11 base 10)
    ---------
         1101
        1101
       0000
    + 1101
    ---------
     10001111 (= 143 base 10)

 O(a.length b.length + b.length b.length)

More examples

  Algorithm Count-Primes(N)
  count <- 0
  for i <- 1 to N do
    if Prime-Test-Odds(i) == YES then count <- count + 1 fi
  od
  return count

  O(N sqrt(N))

One last example

How much time does it take to determine whether white is guaranteed a win in chess?

  O(1)