Recursion -- Doing It Again

by William Shoaff with lots of help

Contents

• Contents
• Recursion
  • The Tower of Hanoi
  • Solving difference equations via mathematical induction
  • Solving difference equations via back substitution
  • Solving difference equations via summing factors
  • Solving difference equations via the Master theorem
  • Solving difference equations via characteristic polynomials
  • Solving difference equations via generating functions
• A collection of recursive algorithms
  • Sorting
  • Multi-Precision Integer Multiplication
  • Locating the max and min of an Array
• Testing the Algorithms
• Problems
• Bibliography

You can download a postscript version of this file (which is prettier) at

http://www.cs.fit.edu/%7Ewds/classes/cse5081/Recur/recur.ps

Recursion

Recursion is a core idea in computer science. Our goal is to learn introductory techniques useful in analyzing the time and space complexity of recursive algorithms. Such programs fall into the divide and conquer paradigm for problem solving. As every general knows, the key to this strategy is in the division. To illustrate the ideas to be presented, we'll start with a division process that is successful because it leads to a slow process.

The Tower of Hanoi

This story, attributed to Lucas, gives us hope that the world will be around for a number of year yet. To paraphrase it, once God created the world he established an order of monks in Asia and set before them the tasks of moving 64 golden disks arranged on a diamond needle to a second needle. The disks were graduated in size and God established the rule that a large disk could never be placed on a smaller one. To complete the task without violating the rule, a third diamond needle was provided. Once all the disks have been moved, God decreed that the world would end.

We will see that 2⁶⁴ - 1 moves are required, thus one can easily calculate that about three hundred billion years will elapse before the world is finished, if the monks can move one disk per second. 10To see this we make the approximation 2¹⁰ $\approx$ 10³ leading to the estimate of about 10¹⁹ moves. At one move a second, that's 10¹⁹seconds, and since there are about $\pi$ billion seconds in a century it will take about

10¹⁹ seconds$${\frac{1 \,\mbox{century}}{\pi \times 10^9\,\mbox{seconds}}}$$ $$\approx$$ $$\mbox{$\pi$ billion centuries or about 300,000,000,000 years.}$$

The Hanoi monks could use this scheme to move all the disks:

Find a method to move the top 63 disk from the first needle to the third. Then move the 64th disk from the first to the second. And finish by moving the 63 disks from the third needle to the second.

We can easily translate this into a recursive program, where Strings first, second, and third represent the needles and integers 1 through 64 the disks.

$<$Tower of Hanoi$>$=
public void towerOfHanoi(String first, String second, String third, int disks) {
  if (disks $>$ 0) {
    towerOfHanoi(first, third, second, disks-1);
    System.out.println(Moving disk +disks+ from +first+ to +second);
    towerOfHanoi(third, second,  first, disks-1);
  }
}

The first resource we wish to measure is the number of moves made by the monks in completing their task. To do so let M(n) = M_n name the numbers of moves to transfer ndisks from one needle to another (we'll often go back and forth between functional (M(n)) and subscript (M_n) notation, depending on what seems appropriate). The towerOfHanoi() program, called with n disks, makes M_{n - 1} moves in transferring the top n - 1 disks from the first to third needle. It makes 1 move in transferring the nth disk from the first to second needle. And finally, it makes M_{n - 1} moves while taking n - 1 disks from the third to second needle. Summing these move counts yields the recurrence relation

 

$$\geq$$ (1)

Equation 1 is called a relation since there are multiple solutions of it. We can fix on one solution by providing an initial condition. For our problem M₀ = 0 is reasonable initial condition as no moves are necessary when there are no disks to move.

Recurrence relations are called difference equations when the initial condition is specified as part of the recursion formula. The study of difference equations using the sum and difference calculus is a fascinating topic predating differential and integral calculus. Difference equation theory parallels the theory of differential equations; the analogies between the two are beautiful. We will not explore these ideas here. Our goal is more immediate: how does one ``solve'' the Tower of Hanoi recurrence?

A simple, yet useful tool is to explore the sequence defined by the recurrence to see if it is familiar. We know M₀ = 0, so using 1 M₁ = 2*0 + 1 = 1, M₂ = 2*1 + 1 = 3 and M₃ = 2*3 + 1 = 7. If you're a computer scientist, you recognize this as the start of the sequence defined by M_n = 2ⁿ - 1, which gives the largest unsigned integer representable with n bits.

But exploring a few cases should never be substituted for a proof, unless one does not have the tools for a proof. Here, several proof techniques exist and provide an arsenal for solving difference equations.

Solving difference equations via mathematical induction

Suppose we can deduce a solution from sampling the sequence. Then the solution can often be proved correct by using induction. For the Tower of Hanoi problem 1, letÕs pretend that M_n = 2ⁿ - 1 is correct. Induction requires us to demonstrate the proposed solution is valid for a base case. Here, we know M₀ = 2⁰ - 1 = 0 is correct. Then we must establish the inductive step: pretending the formula is correct for n - 1, we show it is correct for the next value n. That is, if M_{n - 1} = 2^{n - 1} - 1 then from equation 1

M_n = 2M_{n - 1} + 1 = 2(2^{n - 1} - 1) + 1 = 2ⁿ - 1.

Solving difference equations via back substitution

One simple method for solving difference equations is back substitution. To see how this works, we'll start with the recurrence

 

M_n = 2M_{n - 1} + 1 (2)

and its previous incarnation:

 

M_{n - 1} = 2M_{n - 2} + 1. (3)

Substituting the expression for M_{n - 1} from 3 where M_{n - 1} occurs in 2 yields

 

M_n = 2(2M_{n - 2} + 1) + 1 = 2²M_{n - 2} + 2 + 1. (4)

Now we also know that

 

M_{n - 2} = 2M_{n - 3} + 1, (5)

and substituting this into 4 gives

 

M_n = 2²(2M_{n - 3} + 1) + 2 + 1 = 2³M_{n - 3} + 2² + 2 + 1. (6)

Making an inductive leap we find

 

M_n = 2ⁿM₀ + 2^{n - 1} + 2^{n - 2} + ^... + 2² + 2 + 1, (7)

and since M₀ = 0 we know

 

M_n = 2^{n - 1} + 2^{n - 2} + ^... + 2² + 2 + 1 = 2ⁿ - 1. (8)

Solving difference equations via summing factors

Summing factors are the forward elimination counterpart to back substitution. A key idea is telescoping sums:

(a_n - a_{n - 1}) + (a_{n - 1} - a_{n - 2}) + (a_{n - 2} - a_{n - 3}) + ^... + (a₂ - a₁) + (a₁ - a₀) = a_n - a₀.

For the Tower of Hanoi equation we can make it telescope writing it in a functional form

M_n - 2M_{n - 1} = 1

and multiplying successive versions of it by powers of 2. To wit,

M_n - 2M_{n - 1} = 1

2M_{n - 1} - 2²M_{n - 2} = 2

2²M_{n - 2} - 2³M_{n - 3} = 2²

2³M_{n - 3} - 2⁴M_{n - 4} = 2³

$$\vdots \vdots \vdots$$

2^{n - 1}M₁ - 2ⁿM₀ = 2^{n - 1}

Adding all equations together, and using M₀ = 0, shows M_n = 2ⁿ - 1.

Solving difference equations via the Master theorem

The solution to the general difference equation

T(n) = aT($$\lfloor$$n/b$$\rfloor$$) + cn^k, T(0) = d

depends on the relationship between a, b and k. Note that n/b may not be an integer; it might be more appropriate to use a floor (or ceiling) function:

T(n) = aT($$\lfloor$$n/b$$\rfloor$$) + cn^k, T(0) = d.

Another alternative is to assume n can only be a power of b, which is what we will pretend. It turns out that the asymptotic behavior of T(n) is not changed by either of these two changes.

Theorem 1   Master Theorem Let

T(n) = aT(n/b) + f (n)

Then

1.

If f (n) = O(n^{log_ba - $\epsilon$}) for some $\epsilon$ > 0, then T(n) = $\Theta$(n^log_ba).

2.

If f (n) = $\Theta$(n^log_ba), then T(n) = $\Theta$(n^log_balg n).

3.

If f (n) = $\Omega$(n^{log_ba + $\epsilon$}) for some $\epsilon$ > 0, and if af (n/b) < cf (n) for some constant c < 1 and all sufficiently large n, then T(n) = $\Theta$(f (n)).

A useful special case occurs when f (n) = cn^k and T(1) = 1, then we can conclude the exact solution of the recurrence for powers of b is

T(b^j) = a^j + cb^jk$$\sum_{i=0}^{j-1}$$$$\left(\vphantom{\frac{a}{b^k}}\right.$$$${\frac{a}{b^k}}$$ $$\left.\vphantom{\frac{a}{b^k}}\right)^{i}_{}$$.

Thus there are three cases:

1.

If a < b^k, then T(n) = c''^k - (c'' - 1)n^log_ba = $\Theta$(n^k).

2.

if a = b^k then T(n) = (1 + clog_bn)n^k = $\Theta$(n^klog_bn).

3.

if a > b^k then T(n) = (c' + 1)n^log_ba - c'n^k = $\Theta$(n^log_ba),

for some constants c' and c''.

Let's apply the Master theorem to binary search:

T(n) = T(n/2) + c, T(1) = 1.

We can use the special case where f (n) = cn^k with k = 0. Since a = 1 and b = 2 we find a = 1 = b^k = 2⁰, so apply the second case of the theorem to determine

T(n) = $$\Theta$$(n⁰log_bn) = $$\Theta$$(lg n).

Let's apply the Master theorem to mergeSort.

T(n) = 2T(n/2) + cn, T(1) = 1.

We can use the special case where f (n) = cn^k with k = 1. Since a = 2 and b = 2 we find a = 2 = b^k = 2¹, so apply the second case of the theorem to determine

T(n) = $$\Theta$$(n¹log_bn) = $$\Theta$$(nlg n).

Finally, let's apply the Master theorem to the Tower of Hanoi. Notice that

T(n) = aT(n/b) + f (n)

is a different type of recurrence than the Tower of Hanoi recurrence

M(n) = 2M(n - 1) + 1;

the first involves division and the second subtraction on the size parameter n. A ``trick'' called reparameterization can be used to make this perceived different in form disappear. Reparameterization is such an important idea throughout mathematics we should not denigrate it calling it a ``trick;'' it is a method.

Here's how it works. Let's pretend that m = r(n) = bⁿ for some fixed base b and exponential parameter m. The function r(m) is the reparameterization. Notice that when n - 1 $\leftarrow$ n the new parameter is divided by b: m/b $\leftarrow$ m. Now let's also rename M(n) = M(log_bm) = $\hat{M}$(m)so that we have:

$$\hat{M}$$ = M(log_bm)

= M(n)

= 2M(n - 1) + 1

$$\hat{M}$$ =

a form where the Master theorem applies. In particular, a = 2 and k = 0 so that a = 2 > b^k = b⁰ = 1for any value of b; and the third case of the Master theorem yields

$$\hat{M}$$(m) = $$\Theta$$(m^log_b2) = $$\Theta$$(2^log_bm) = $$\Theta$$(2ⁿ),

which is what we expect.

Solving difference equations via characteristic polynomials

A recurrence of the form

a₀t_n + a₁t_{n - 1} + ^... + a_kt_{n - k} = 0

is called an n-th order homogeneous linear recurrence with constant coefficients. For example, the Fibonacci recurrence

F_n - F_{n - 1} - F_{n - 2} = 0,    F₀ = 0, F₁ = 1

is a second order homogeneous linear recurrence with constant coefficients. The Tower of Hanoi recurrence

M_n - 2M_{n - 1} = 1

is a first order non-homogeneous linear recurrence with constant coefficients.

Linear recurrences

By renaming t_n = xⁿ we can turn linear recurrence with constant coefficients into a polynomial:

a₀xⁿ + a₁x^{n - 1} + ^... + a_kx^{n - k} = 0

something with which we are familiar. For example, the Fibonacci recurrence becomes

xⁿ - x^{n - 1} - x^{n - 2} = x^{n - 2}(x² - x - 1) = 0.

This equation is satisfied if x = 0 which is a trivial solution of little interest, or if

x² - x - 1 = 0.

The polynomial p(x) = x² - x - 1 is called the characteristic polynomial of the Fibonacci recurrence. In general,

p(x) = a₀x^k + a₁x^{k - 1} + ^... + a_k

is the characteristic polynomial of the recurrence

a₀t_n + a₁t_{n - 1} + ^... + a_kt_{n - k} = 0.

Now, theoretically, every polynomial of degree k can factored as a product of linear terms, specifically the fundamental theorem of algebra states:

p(x) = a₀x^k + a₁x^{k - 1} + ^... + a_k = a₀$$\prod_{i=1}^{k}$$(x - r_i)

where r₁,..., r_k are the roots or zeros of the polynomial. Of course, the roots may be complex numbers and some may occur more than once. In practice, we can not factor a general polynomial of degree 5 or greater. Fortunately, most recurrences we will come upon will be first or second order, which we certainly can factor.

Let r be any root of the characteristics polynomial. Then t_n = rⁿ is a solution of the recurrence

a₀t_n + a₁t_{n - 1} + ^... + a_kt_{n - k} = a₀rⁿ + a₁r^{n - 1} + ^... + a_kr^{n - k}

= r^{n - k}(a₀r^k + a₁r^{k - 1} + ^... + a_k

= 0

More importantly, if r₁ and r₂ are both roots, then c₁r₁ + c₂r₂ is a solution of the recurrence for any coefficients c₁ and c₂. (This is the linear property of the recurrence.)

Consider the Fibonacci characteristic polynomial x² - x - 1. Use the quadratic formula, we find roots

r₁ = $${\frac{1+\sqrt{5}}{2}}$$    and    r₂ = $${\frac{1-\sqrt{5}}{2}}$$

The root r₁ is know as the golden mean and often denoted by $\phi$

$$\phi$$ = $${\frac{1+\sqrt{5}}{2}}$$ $$\approx$$ 1.618033988749 ^...

Let's call the other root $\hat{\phi}$. Note that

(x - $$\phi$$)(x - $$\hat{\phi}$$) = x² - ($$\phi$$ + $$\hat{\phi}$$)x + $$\phi$$ ^. $$\hat{\phi}$$ = x² - x - 1

So the roots satisfy the interesting relations

$$\phi$$ + $$\hat{\phi}$$ = 1    and    $$\phi$$ ^. $$\hat{\phi}$$ = - 1.

But to the point, the Fibonacci numbers are given by

F_n = c₁$$\phi^{n}_{}$$ + c₂$$\hat{\phi}^{n}_{}$$ = c₁$$\left(\vphantom{\frac{1+\sqrt{5}}{2}}\right.$$$${\frac{1+\sqrt{5}}{2}}$$ $$\left.\vphantom{\frac{1+\sqrt{5}}{2}}\right)^{n}_{}$$ + c₂$$\left(\vphantom{\frac{1-\sqrt{5}}{2}}\right.$$$${\frac{1-\sqrt{5}}{2}}$$ $$\left.\vphantom{\frac{1-\sqrt{5}}{2}}\right)^{n}_{}$$

for some constants c₁ and c₂, which can be found by using the initial conditions, to wit,

$$\left(\vphantom{\frac{1+\sqrt{5}}{2}}\right. {\frac{1+\sqrt{5}}{2}} \left.\vphantom{\frac{1+\sqrt{5}}{2}}\right)^{0}_{} \left(\vphantom{\frac{1-\sqrt{5}}{2}}\right. {\frac{1-\sqrt{5}}{2}} \left.\vphantom{\frac{1-\sqrt{5}}{2}}\right)^{0}_{}$$ F₀ =

= c₁ + c₂

= 0,

$$\left(\vphantom{\frac{1+\sqrt{5}}{2}}\right. {\frac{1+\sqrt{5}}{2}} \left.\vphantom{\frac{1+\sqrt{5}}{2}}\right)^{1}_{} \left(\vphantom{\frac{1-\sqrt{5}}{2}}\right. {\frac{1-\sqrt{5}}{2}} \left.\vphantom{\frac{1-\sqrt{5}}{2}}\right)^{1}_{}$$ F₁ =

$${\frac{c_1 + c_2}{2}} {\frac{\sqrt{5}(c_1 - c_2)}{2}}$$ =

$${\frac{\sqrt{5}(c_1 - c_2)}{2}}$$ =

= 1.

From which we can conclude

C₁ = $${\frac{1}{\sqrt{5}}}$$    and    c₂ = - $${\frac{1}{\sqrt{5}}}$$.

Non-homogeneous recurrences

Now let's consider non-homogeneous the Tower of Hanoi recurrence

M_n - 2M_{n - 1} = 1

We can derive a homogeneous recurrence by subtracting two successive versions of the recurrence:

M_n - 2M_{n - 1} = 1

M_{n - 1} - 2M_{n - 2} = 1

M_n - 3M_{n - 1} + 2M_{n - 2} = 0.

The characteristic polynomial of this last equation is p(x) = x² - 3x + 2 = (x - 2)(x - 1), and we can conclude that M_n = c₁1ⁿ + c₂2ⁿ = c₁ + c₂2ⁿfor some constants c₁ and c₂ which can be determined from the initial conditions M₀ = 0 = c₁ + c₂ and M₁ = 1 = c₁ + 2c₂. That is, c₁ = - 1 and c₂ = 1.

More generally, if the non-homogeneous recurrence has the form

a₀t_n + a₁t_{n - 1} + ^... + a_kt_{n - k} = cⁿq(n)

where c is a constant and q(x) is a polynomial of degree m, then we can use

(a₀x^k + a₁x^{k - 1} + ^... + a_k)(x - r)^{m + 1}

as the characteristic polynomial and the recurrence has solution

t_n = $$\sum_{i=1}^{k}$$c_ir_iⁿ + $$\sum_{j=0}^{m}$$d_jn^jrⁿ,

for some constants c_i, d_j which can be found from initial conditions.

Repeated roots

The above discussion of non-homogeneous recurrences led to characteristic polynomials with repeated roots. A root r has multiplicity m if the characteristic polynomial can be factored as

p(x) = a₀$$\prod_{j=1}^{m}$$(x - r)$$\prod_{i=1}^{k-m}$$(x - r_i)

where r_i $\neq$ r for any other roots r_i, i = 1,..., k - m.

When a root r has multiplicity m, we can show that

t_n = rⁿ(d₀ + d₁n + ^... + d_{m - 1}n^{m - 1}) = rⁿq(n)

satisfies the recurrence relation. In particular, we only need to show this for t_n = n^jrⁿwhere j can be any natural number from 0 to m - 1, and then appeal to linearity. To do this, note that since

p(x) = a₀$$\prod_{j=1}^{m}$$(x - r)$$\prod_{i=1}^{k-m}$$(x - r_i)

x^{n - k}p(x) and first m - 1 derivatives of x^{n - k}p(x)will vanish to zero at x = r. We won't do the calculus and algebra here, but trust me, it works.

Solving difference equations via generating functions

Generating functions offer a powerful tool for solving recurrences. The idea is simple, but can involve some convoluted symbol manipulation. We start with an infinite sequence

$$\langle$$a₀, a₁, a₂, ...$$\rangle$$

and define the associated generating function

 

$$\sum_{n=0}^{\infty}$$ (9)

From preliminary lectures sums and series we know the generating functions of a number of sequences:

1.

The Alice sequence $\langle$1, 1, 1,...$\rangle$ has generating function

A(z) = $$\sum_{k=0}^{\infty}$$z^k = 1/(1 - z).

2.

The Gauss sequence $\langle$0, 1, 2, 3,...$\rangle$ has generating function

G(z) = $$\sum_{k=0}^{\infty}$$kz^k = z/(1 - z)².

3.

The Harmonic sequence $\langle$1/1, 1/2, 1/3, 1/4,...$\rangle$ has generating function

H(z) = $$\sum_{k=0}^{\infty}$$z^k/(k + 1) = $${\frac{1}{z}}$$ln$$\left(\vphantom{\frac{1}{1-z}}\right.$$$${\frac{1}{1-z}}$$ $$\left.\vphantom{\frac{1}{1-z}}\right)$$.

A two-step process solves difference equations using generating functions. The difference equation defines a sequence $\langle$a₀, a₁, a₂,...$\rangle$ of unknown terms.

1.

Translate the recurrence into an equation involving the generating function of the sequence.

2.

Expand the equation into an infinite series using using tools from algebra and analysis.

The coefficients of the series are the sought for terms of the sequence.

Let's apply this method to the Tower of Hanoi recurrence. The sequence $\langle$M₀, M₁, M₂,...$\rangle$has terms M_n representing the number of moves when n disks are stacked on the first needle. The recurrence M_n = 2M_{n - 1} + 1, n > 1 is multiplied by zⁿ and summed from 1 to $\infty$.

$$\sum_{n=1}^{\infty} \sum_{n=1}^{\infty}$$ (10)

The sum on the left defines the generating function M(z). Using algebra, we manipulated the sum on the right to obtain

$$\sum_{k=0}^{\infty}$$ (11)

Thus we derive the equation

M(z) = z/(1 - z)(1 - 2z), (12)

finishing the first step.

The second step is to expand this expression into a series. We have:

M(z) = z/(1 - z)(1 - 2z)

= 1/(1 - 2z) - 1/(1 - z)

$$\sum_{n=0}^{\infty} \sum_{n=0}^{\infty}$$ =

$$\sum_{n=0}^{\infty}$$ =

$$\sum_{n=0}^{\infty}$$ =

That is, M_n = 2ⁿ - 1.

Reparameterization

Consider a recurrence in the form that appears in the Master theorem

T(n) = aT(n/b) + f (n).

To use generating functions in the solution of this type of equation, we can re-parameterize letting n = b^m with new parameter m and T(n) = $\hat{T}$(m). Then

$$\hat{T}$$ = T(n)

= aT(n/b) + f (n)

$$\hat{T} \hat{f}$$ =

A collection of recursive algorithms

To test our mettle, we will use our difference equation solution techniques on a collection of recursive algorithms. We start with some classic algorithms that sort elements selected from a sample space with a total (also called linear) order.

Sorting

With some loss of generality, we will assume the sorting problem we are to solve has an instance described by

1.

A finite alphabet of symbols, called A, with cardinality 2 or greater.

2.

A sample space S of words. Most often S will be all words over A whose lengths are bounded, say by an integer k.

3.

A finite collections of words independently and uniformly selected from S.

4.

A linear order on the words, that is, a relation < such that for any words u, v, w,

• One of: u < v, u = v, or u > v holds (the trichotomy property)
• If u < v and v < w, then u < w (the transitivity property)

A lexicographical order that satisfies these properties can be defined on words.

Mergesort

The code for mergeSort() is in another file which you can visit should you forget the details of how it is implemented.

Analysis of mergeSort().

Under the usual uniform-cost assumption, each move of a word from one array to another has unit cost and each comparison between words has unit cost. Other incidental operations are not counted.

Since, on the initial call, there are n words to move from words[] into tempWords[] and then back into words[], there are 2n String moves. Also there are n String comparisons, on the initial call, as k goes from first=0 to last=n-1. Thus, the recurrence relation for the time complexity T(n) of mergeSort() is

 

T(n) = 2T(n/2) + 3n. (13)

A reasonable initial condition is T(1) = 1, which takes into account the cost of a subroutine call which does no real work.

From the Master theorem, we can conclude that mergeSort() has time complexity

 

$$\Theta$$ (14)

that is, mergeSort() has a best, worst, and average case time complexity that grows as nlg n.

Space complexity

What is the space complexity S(n) of mergeSort()? In the merge section of the code an new array of size last-first+1 is declared. This is largest when last=n-1 and first = 0. We must assume that a smart compiler reuses and reclaims space and so the space complexity is O(n).

Another issue is stack space required to support recursion. A pointer to the words to be sorted and the range being sorted and merged must be stored, that is, three items. Since the recursion depth is O(lg n), we find that the additional space complexity due to recursion does not alter the big-O space complexity mentioned above.

Quicksort

The code for quickSort() is in another file which you can visit should you forget the details of how it is implemented.

  
Analysis of quickSort().

That partition requires O(n) operations is clear from analyzing the single for loop. In fact, using n + 1 as the cost of partition will make the analysis smooth, so we assume the difference equation for quickSort() is

 

T(n) = T(p) + T(n - p - 1) + n + 1, T(1) = 1 (15)

In the worst case partitioning will create an empty array on one side of pivot and an n - 1 array on the other side, that is, p = 0 (or p = n - 1). If this occurs on every call then

T(n) = T(n - 1) + n + 1, T(1) = 1.

Solving the difference by summation factors is easy. Since T(n) - T(n - 1) = n + 1 we can form the telescoping sum

$$\sum_{i=2}^{n}$$T(i) - T(i - 1) = T(n) - T(1) = $$\sum_{i=2}^{n}$$(i + 1) = (n + 1)(n + 2)/2 - 3.

Or T(n) = (n² + 3n - 2)/2.

What are the instances where this worst case behavior occurs?

Intuitively, in the best case, pivot will partition the array into (nearly) equal sized sub-arrays, that is, p = n/2. When this happens at each recursive stage, the recurrence for quickSort() is

T(n) = 2T(n/2) + n + 1.

The Master theorem implies that T(n) = $\Omega$(nlg n).

The average case time complexity.

In the average case, any pivot from p = 0 to p = n - 1 can occur. We will assume each of these n cases occur with equal probability 1/n. Thus, the recurrence is:

T(n) = n + 1 + $$\sum_{p=0}^{n-1}$$[T(p) + T(n - p - 1)]/n,

or

T(n) = n + 1 + 2/n$$\sum_{p=0}^{n-1}$$T(p).

There are two tricks to simplify this equation

1.

Eliminate denominators: multiply both sides of the equation by n.

nT(n) = n² + n + 2$$\sum_{p=0}^{n-1}$$T(p)

2.

Eliminate summations: subtract two successive equations. Replace n by n + 1,

(n + 1)T(n + 1) = n² + 3n + 2 + 2$$\sum_{p=0}^{n}$$T(p),

and subtract the previous equation from this forming:

 

(n + 1)T(n + 1) - nT(n) = 2(n + 1) + 2T(n) (16)

Now let's pretend G(z) is the generating function for the sequence

$$\langle$$T(0), T(1), T(2),...$$\rangle$$

where T(0) = 0, that is

G(z) = T₀ + T₁z + T₂z² + ^... = $$\sum_{n=0}^{\infty}$$T_nzⁿ

where we write T_n for T(n).

Terms of the form nT(n) in equation 18 suggest differentiation of the generating function. Notice that

$$\sum_{n=0}^{\infty} \sum_{n=0}^{\infty}$$ G'(z) - zG'(z) =

$$\sum_{n=0}^{\infty} \sum_{n=0}^{\infty}$$ =

$$\sum_{n=0}^{\infty}$$ =

$$\sum_{n=0}^{\infty}$$ =

$${\frac{2}{(1-z)^2}}$$ =

Thus,

G'(z) = $${\frac{2}{(1-z)^3}}$$ + $${\frac{2}{1-z}}$$G(z)

Or, multiplying by (1 - z)² and rearranging terms

(1 - z)²G'(z) - 2(1 - z)G(z) = $${\frac{2}{1-z}}$$

But the left-hand side above is the derivative of

(1 - z)²G(z)

so integrating both sides

(1 - z)²G(z) = - 2ln(1 - z) + C

where C = 0 since G(0) = T₀ = 0It follows that

$${\frac{-2}{(1-z)^2}}$$ G(z) =

$$\sum_{i=1}^{\infty} \sum_{j=1}^{\infty} {\frac{z^j}{j}}$$ =

$$\sum_{n=1}^{\infty} \sum_{k=1}^{n} {\frac{n-k+1}{k}}$$ =

$$\sum_{n=1}^{\infty}$$ =

Therefore, the average case time complexity of quickSort() is

 

$$\Theta$$ (17)

Multi-Precision Integer Multiplication

Now let's consider a different type of problem. When the product c = a ^. b of two positive integers exceeds a computer's word size we can accepts an approximate answer $\hat{c}$ or devise a multiple precision approach to record the exact product. The use of large integers in encryption algorithms makes multi-precision arithmetic problems interesting.

Let's pretend we're programming a computer with positive integer register word size of n bits. We'll also pretend that positive integers a and b lie in the range from 0 to 2ⁿ - 1. Of course, their product c can be as large as 2²ⁿ - 2^{n + 1} + 1 and require as many as 2n bits to represent exactly.

To get started, we'll partition a positive integer a into most and least significant half-words. That is, write a in its radix 2 form:

a = a^{n - 1}2^{n - 1} + a^{n - 2}2^{n - 2} + ^... + a₁2¹ + a₀2⁰

= 2^n/2(a^{n - 1}2^{n/2 - 1} + ^... + a_n/2) + (a_{n/2 - 1}2^{n/2 - 1} + ^... + a₀)

= 2^n/2aMost + aLeast

Multiplication of two n bit integers a and b can then be written as

a*b = (2^n/2aMost + aLeast)*(2^n/2bMost + bLeast)

= 2ⁿ(aMost*bmost) +         2^n/2(aMost*bLeast + aLeast*bMost) + aLeast*bLeast

= 2ⁿ(aMost*bMost) +         2^n/2((aMost - aLeast)(bLeast -         bMost) + aMost*bMost + aLeast*bLeast) +         aLeast*bLeast

If we let T(n) denote the cost of multiplying 2 n bit numbers then the first expansion of the product produces the recurrence

T(n) = 4T(n/2) + cn,

and the last, more complex, expansion of the product produces the recurrence

T(n) = 3T(n/2) + dn.

That is, the first expansion has 4 multiplies of n/2 bit numbers, two bits shifts, and 3 adds of at most 2n bit numbers, the second expansion requires only 3 multiplies of n/2 bit numbers, the same shifts, but 6 adds. The shifts and adds can be done in linear time.

Locating the max and min of an Array

Given an array of words, we'd like to find the minimum and maximum words, that is, if the words were ordered, the minimum would be first and the maximum would be last. A brute force algorithm simply scans the array and maintains information about the minimum and maximum words. It is easy to count that the worst case time complexity of naiveMinMax() is O(n); in the worst case 2n - 2 compares are made.

The code for the brute force scan of an array to find the minimum and maximum is in another file which you can visit should you forget the details of how it is implemented. Let's determine the number of comparisons in the average case for this algorithm. For each iteration of the loop at least one compare is always made and a second compare is made if and only if word[i] is greater than all of the previous words. We'll assume all the words are distinct to rule out the ``or equal'' case. Over all n! permutations of words, it is easy to convince oneself that word[i] will be greater than the previous i - 1 words with probability 1/i. To see this, notice word[2] will be greater than word[1] 1/2 of the time, word[3] will be greater than word[1] and word[2] 1/3 of the time, and so on. In general, i words can be arranged in i! ways and out of these (i - 1)! arrangement will have the maximum word last.

We can compute the average case complexity by adding, from i = 1to i = n, the probability of making one compare (1) times the cost of one compare (1) plus probability of making a second compares (1-1/i) times the cost of the second compare (1).

$$\sum_{i=1}^{n-1}$$(1 + (1 - 1/i)) = 2n - 2 - H_{n - 1} = 2n - 2 - $$\Theta$$(lg n),

where H_{n - 1} is the n - 1-st harmonic number.

Next, let's develop a recursive solution to the min-max problem.

public String[] recursiveMinMax(String[] words, int low, int high) {
  String[] minMax = new String[2]; 
  
  
  
  return minMax;
}

For a one element array the minimum and maximum are identical.

=
  if (low == high) {
    minMax[0] = words[low];
    minMax[1] = words[low];
    return minMax;
  }

For a two element array, one compare in made and based on the outcome the minimum and maximum words can be set.

=
  if (low == high - 1) {
    if (words[low].compareTo(words[high]) $<$ 0) {
      minMax[0] = words[low];
      minMax[1] = words[high];
    }
    else {  
      minMax[0] = words[high];
      minMax[1] = words[low];
    }
    return minMax;
  }

Now when there are more than two elements, we'll divide the array in half (roughly), determine the minimum and maximum of both halves, compare the two minimums to determine the global minimum, and compare the two maximum to determine the global maximum. = int middle = (low+high)/2; String[] minMaxLow = recursiveMinMax(words, low, middle); String[] minMaxHigh = recursiveMinMax(words, middle+1, high); if (minMaxLow[0].compareTo(minMaxHigh[0]) $<$ 0) { minMax[0] = minMaxLow[0]; } else { minMax[0] = minMaxHigh[0]; } if (minMaxLow[1].compareTo(minMaxHigh[1]) $<$ 0) { minMax[1] = minMaxHigh[1]; } else { minMax[1] = minMaxLow[1]; }

From the syntax of the algorithm, we can derive the recurrence The initial conditions

T(1) = 0    and    T(2) = 1

reflect that zero and one word comparisons are made for 1 and 2 element arrays. For larger array, there are two recursive calls with (roughly) one-half the elements, and then two word compares are made. Thus, we have the recurrence below.

 

T(n) = 2T(n/2) + 2, T(1) = 0    and    T(2) = 1
(18)

which we can show has solution T(n) = 1.5n - 2.

Testing the Algorithms

Not trusting our ability to write error-free code we'll write some test cases. The code for the test cases is in another file which you can visit if you are interested in the details.

Problems

Problem 1:

Problem 1: How much space is used in the recursion stack needed to support this implementation of the algorithm? Find an alternative (iterative) algorithm that simulates the task in constant space.

Problem 2:

Problem 2: Solve the following recurrences using induction: find a proposed solution by sampling and verify it by induction.

1.

The number of nodes in a perfect binary trees of height n: t_n = t_{n - 1} + 2ⁿ, t₀ = 1.

2.

The maximal number of regions in a plane cut by n lines: r_n = r_{n - 1} + n, r₀ = 1.

Problem 3:

Problem 3: Solve the following recurrences using back substitution.

1.

The number of nodes in a perfect binary trees of height n: t_n = t_{n - 1} + 2ⁿ, t₀ = 1.

2.

The maximal number of regions in a plane cut by n lines: r_n = r_{n - 1} + n, r₀ = 1.

Problem 4:

Problem 4: Solve the following recurrences using summation factors.

1.

The number of nodes in a perfect binary trees of height n: t_n = t_{n - 1} + 2ⁿ, t₀ = 1.

2.

The maximal number of regions in a plane cut by n lines: r_n = r_{n - 1} + n, r₀ = 1.

Problem 5:

Problem 5: Use mathematical induction to prove the three cases of the Master Theorem.

Problem 6:

Problem 6: Use back substitution to prove the three cases of the Master Theorem.

Problem 7:

Problem 7: Use summing factors to prove the three cases of the Master Theorem.

Problem 8:

Problem 8: Use the Master Theorem to find the time complexity of the following algorithms:

1.

Find max-min of an array: T(n) = 2T(n/2) + c, T(1) = 1

2.

Multi-precision integer multiplication: T(n) = 3T(n/2) + cn, T(1) = 1

Problem 9:

Problem 9: Use generating functions to find the solution of the following difference equation:

1.

Nodes in a complete binary tree: t_n = t_{n - 1} + 2ⁿ, t₀ = 1.

2.

Maximal regions in a plane cut by n lines: r_n = r_{n - 1} + n, r₀ = 1.

Problem 10:

Problem 10: Use generating functions to find the time complexity of the following algorithms:

1.

Binary search: T(n) = T(n/2) + c, T(1) = 1

2.

Find max-min of an array: T(n) = 2T(n/2) + c, T(1) = 1

3.

Mergesort: T(n) = 2T(n/2) + cn, T(1) = 1

4.

Multi-precision integer multiplication: T(n) = 3T(n/2) + cn, T(1) = 1

Problem 11:

Problem 11: Given a binary tree T one can traverse it in pre-order, in-order, or post-order. Write recursive algorithms for these and analyze their time complexities.

Problem 12:

Problem 12: Solve the two recurrences using the Master theorem. Argue that the first recurrence is no better than the naive O(n²)algorithm for multiplication, the the second is asymptotically better.

Problem 13:

Problem 13: Develop a data representation of multi-precision integers and implement integer addition, (fast) multiplication using it. For additional credit, implement addition and multiplication modulo n. These operation form the basis of many public-key encryption algorithms.

Problem 14:

Problem 14: Show that the proposed solution to 20 is correct.

Problem 15:

Problem 15: The exact recurrence that counts the number of comparisons is given below. Write a program that will compute 1.5n - 2, C(n) and the error C(n) - 1.5n + 2 for all n up to some fixed value. Use this information to derive an exact solution to the exact recurrence.

 

C(1) = 0    and    C(2) = 1 (19)

$$\mbox{if $n$\space is even}$$ C(n) = (20)

$$\mbox{if $n$\space is odd}$$ C(n) = (21)

This problem can be found in [5].

@ Analyze the time complexity of these traversal algorithms.

Bibliography

1

J. L. BENTLEY, Programming Pearls, Addison-Wesley, 1986.

2

C. A. R. HOARE, Quicksort, Computer Journal, 5 (1961), pp. 10-15.

3

R. SEDGEWICK, Implementing quicksort programs, Communications of the ACM, 21 (1978), pp. 847-857.

4

R. SEDGEWICK, Algorithms in C, Addison-Wesley, 1988.

5

D. R. STINSON, An Introduction to the Design and Analysis of Algorithms, The Charles Babbage Research Center, P. O. Box 272, St. Norbert Postal Station, Winnipeg, Manitoba R3V 1L6, Canada, second ed., 1987.