Asymptotics in the Analysis of Algorithms

by William Shoaff with lots of help

You can download a postscript version of this file (which is prettier) at

http://www.cs.fit.edu/%7Ewds/classes/algorithms/Asym/asymptotics.pdf

Introduction

We need to formalize the ideas that an algorithms has running time or storage requirements that are ``never more than,'' ``always greater than,'' or ``exactly'' some amount. Recall that we use T(n) for the running time or time complexity and S(n) for the storage requirements or space complexity of an algorithm on an instance of size n. And recall that the time and space may vary for different instances of the same size, e.g., it may take less time to sort an already sorted list.

The phrase ``never more than'' expresses an upper bound on T(n)or S(n). There are several ways you could choose to express this:

1.: The algorithm never take more than (some function of n of) operations.
2.: The algorithm has running time that is always less than some function of n.
3.: The algorithm's running time is in the order of some function of n.
4.: The algorithm's time complexity is big-O of some function of n.

You can make similar statements about space complexity.

The phrase ``always greater than'' expresses a lower bound on T(n)or S(n). There are several ways you could choose to express this:

1.: The algorithm always take at least (some function of n of) operations.
2.: The algorithm has running time that is always greater than some function of n.
3.: The algorithm's time complexity is big-Omega ( $\Omega$ ) of some function of n.

You can make similar statements about space complexity.

There are certain technical features which much be placed on this function referred to above. We won't try to state all of them. Let's give the function a name, call it f (n).

Properties of complexity functions

A proper complexity functions (or running time function) has the following properties:

1.: f : $\mathbb{N}$ $\rightarrow$ $\mathbb{R}$ ⁺ -- a complexity function f maps natural numbers into non-negative real numbers.
2.: f (n + 1) $\geq$ f (n) $\forall$ n -- a complexity function f is non-decreasing.

There are really some more stringent requirements, but we do not need to worry about them here.

You are familiar with the common functions that will be useful complexity functions.

the constant function f (n) = c;
the logarithmic function f (n) = lg n;
the linear function f (n) = n;
the linear-log function f (n) = nlg n;
the quadratic function f (n) = n²;
the cubic function f (n) = n³;
the general power function f (n) = n^k;
the exponential function f (n) = 2ⁿ;
the factorial function f (n) = n!;
and the function f (n) = nⁿ.

And you should be familiar with the growth rate of these functions. The table below illustrates their growth.

**Table:** Sample growth rates and running times ( $\mu$ sec)
growth rate	n = 1	n = 10	n = 100	n = 1000
lg n	0	3.32	6.64	9.96
n	1	10	100	1000
nlg n	0	33.2	664	9966
n²	1	100	10000	10⁶
n³	1	1000	10⁶	10⁹
2ⁿ	2	1024	10³⁰	10³⁰⁰
nⁿ	1	10¹⁰	10²⁰⁰	10³⁰⁰⁰

O-notation

O-notation is used to denote upper bounds. We write

T(n) = O(f (n))

to denote that T(n) never takes more than roughly f (n)operations. This is stated in English as

``T(n) is in the order of f (n)'', or ``T(n) is big-O of f (n)''

We need to make this informal notion more precise.

Definition 1 Let T(n) be the time complexity of an algorithm and let f (n) be a proper complexity functions. We write

T(n) = O(f (n))

if and only if there exists a natural number N > 0 and a constant c > 0such that

T(n) $\displaystyle \leq$ c^.f (n) $\displaystyle \forall$ n > N.

This can be interpreted as saying

``T grows slower than a constant time f for all sufficiently large n'', or
``The graph of T lies below the graph of a constant time f for all sufficiently large n.''

The notation O(f (n)) actually denotes a set; the set of all functions with growth rate less than or equal to f (n).

O(f (n)) = {T : $\mathbb{N}$ $\displaystyle \rightarrow$ $\mathbb{R}$ ⁺| $\displaystyle \exists$ c > 0 $\displaystyle \ni$ $\displaystyle \forall$ n $\displaystyle \in$ $\mathbb{N}$ , T(n) $\displaystyle \leq$ cf (n)}

$\Omega$ -notation

$\Omega$ -notation is used to denote lower bounds. We write

T(n) = $\displaystyle \Omega$ (f (n))

to denote that T(n) always takes at least roughly f (n)operations. This is stated in English as

``T(n) is $\Omega$ of f (n)''

We need to make this informal notion more precise.

Definition 2 Let T(n) be the time complexity of an algorithm and let f (n) be a proper complexity functions. We write

T(n) = $\displaystyle \Omega$ (f (n))

if and only if there exists a natural number N > 0 and a constant d > 0such that

d^.f (n) $\displaystyle \leq$ T(n) $\displaystyle \forall$ n > N.

This can be interpreted as saying

``T grows faster than a constant time f for all sufficiently large n'', or
``The graph of T lies above the graph of a constant time f for all sufficiently large n.''

The notation $\Omega$ (f (n)) actually denotes a set; the set of all functions with growth rate less than or equal to f (n).

$\displaystyle \Omega$ (f (n)) = {T : $\mathbb{N}$ $\displaystyle \rightarrow$ $\mathbb{R}$ ⁺| $\displaystyle \exists$ d > 0 $\displaystyle \ni$ $\displaystyle \forall$ n $\displaystyle \in$ $\mathbb{N}$ , cf (n) $\displaystyle \leq$ T(n)}

$\Theta$ -notation

$\Theta$ -notation is used to denote both upper and lower bounds. We write

T(n) = $\displaystyle \Theta$ (f (n))

to denote that T(n) always takes roughly f (n)operations. This is stated in English as

``T(n) is $\Theta$ of f (n)''

We need to make this informal notion more precise.

Definition 3 Let T(n) be the time complexity of an algorithm and let f (n) be a proper complexity functions. We write

T(n) = $\displaystyle \Theta$ (f (n))

if and only if there exists a natural number N > 0 and constants c, d > 0such that

d^.f (n) $\displaystyle \leq$ T(n) $\displaystyle \leq$ c^.f (n) $\displaystyle \forall$ n > N.

This can be interpreted as saying

``T grows at the same rate as some constant time f for all sufficiently large n'', or
``The graph of T lies between the graphs of df and cf for all sufficiently large n.''

The notation $\Theta$ (f (n)) actually denotes a set; the set of all functions with growth rate less than or equal to f (n).

$\displaystyle \Theta$ (f (n)) = {T : $\mathbb{N}$ $\displaystyle \rightarrow$ $\mathbb{R}$ ⁺| $\displaystyle \exists$ c, d > 0 $\displaystyle \ni$ $\displaystyle \forall$ n $\displaystyle \in$ $\mathbb{N}$ , df (n) $\displaystyle \leq$ T(n) $\displaystyle \leq$ cf (n)}

Properties of the asymptotic notations

The O, $\Omega$ , and $\Theta$ notations are called asymptotic notations since they apply for large values of n. Below the bound N, the inequalities stated in the definitions need not hold.

Theorem 1 Let f, g and h are running time functions. Then the following properties hold:

1.: f (n) = O(f (n)) (O is reflexive)
2.: f (n) = $\Omega$ (f (n)) ( $\Omega$ is reflexive)
3.: f (n) = $\Theta$ (f (n)) ( $\Theta$ is reflexive)
4.: If f (n) = O(g(n)) and g(n) = O(h(n)), then f (n) = O(h(n)) (O is transitive)
5.: If f (n) = $\Omega$ (g(n)) and g(n) = $\Omega$ (h(n)), then f (n) = $\Omega$ (h(n)) ( $\Omega$ is transitive)
6.: If f (n) = $\Theta$ (g(n)) and g(n) = $\Theta$ (h(n)), then f (n) = $\Theta$ (h(n)) ( $\Theta$ is transitive)
7.: f (n) + g(n) = O(max{f (n), g(n)})
8.: f (n) + g(n) = $\Omega$ (max{f (n), g(n)})
9.: f (n) + g(n) = $\Theta$ (max{f (n), g(n)})
10.: If f (n) = O(g(n)) and g(n) = O(f (n)) and then f (n) = $\Theta$ (g(n)).

Limits

Limits play an important role in working with asymptotic relations.

Definition 4 We say f (n) converges to a, or, the limit if f (n) exists and equals a as n approaches infinity provided that $\forall$ $\epsilon$ > 0, $\exists$ N > 0 $\ni$

| f (n) - a| $\displaystyle \leq$ $\displaystyle \epsilon$ .

We write this as

$\displaystyle \lim_{n\rightarrow \infty}^{}$ f (n) = a.

We say f (n) diverges to infinity if $\forall$ $\triangle$ > 0, $\exists$ N > 0 $\ni$

f (n) $\displaystyle \geq$ $\displaystyle \triangle$ .

We write this as

$\displaystyle \lim_{n\rightarrow \infty}^{}$ f (n) = + $\displaystyle \infty$ .

Theorem 2 Let f and g be running time functions. Then the following properties hold:

1.: If $\lim_{n\rightarrow \infty}^{}$ ${\frac{f(n)}{g(n)}}$ converges to a > 0, then f (n) = $\Theta$ (g(n))
2.: If $\lim_{n\rightarrow \infty}^{}$ ${\frac{f(n)}{g(n)}}$ = 0, then f (n) = O(g(n)) but g(n) $\neq$ O(f (n)).
3.: If $\lim_{n\rightarrow \infty}^{}$ ${\frac{f(n)}{g(n)}}$ diverges to infinity, then g(n) = O(f (n)) but f (n) $\neq$ O(g(n)).

Theorem 3 (l' Hôpital's Rule) Suppose that

$\displaystyle \lim_{n\rightarrow \infty}^{}$ f (n) = $\displaystyle \lim_{n\rightarrow \infty}^{}$ g(n) = + $\displaystyle \infty$

(or alternatively, both converge to zero), then

$\displaystyle \lim_{n\rightarrow \infty}^{}$ $\displaystyle {\frac{f(n)}{g(n)}}$ = $\displaystyle \lim_{n\rightarrow \infty}^{}$ $\displaystyle {\frac{f'(n)}{g'(n)}}$

That is, the limit of ratios of the functions is the limit of the ratios of their derivatives.

Examples of working with asymptotics

Example 1:

Show f (n) = 5n = O(n).

We need to find a constant c > 0 and an natural number N > 0 such that 5n $\leq$ cn for all n $\geq$ N. Clearly this is not hard, one choice is c = 5 and N = 1.

Example 2:

Show f (n) = 6n² + 3n - 2 = O(n²).

We need to find a constant c > 0 and a natural number N > 0 such that 6n² + 3n - 2 $\leq$ cn² for all n $\geq$ N. There are several ways to approach this problem.

1.
Solution 1. For for 3 $\leq$ n, we know 3n $\leq$ n² and -2 $\leq$ n². Therefore

6n² + 3n - 2 $\displaystyle \leq$ 6n² + n² + n² = 8n²
for all n $\geq$ 3. So we can pick c = 8 and N = 3.
2.
Solution 2. The inequality 6n² + 3n - 2 $\leq$ cn² is equivalent to (c - 6)n² - 3n + 2 $\geq$ 0. Picking c = 7 allows us to simply factor the quadratic as (n - 2)(n - 1) which will be greater than 0 provided n $\geq$ 2.

Example 3:

Show f (n) = lnn = O( $\sqrt{n}$ ).

We need to find a constant c > 0 and a natural number N > 0 such that lnn $\leq$ c $\sqrt{n}$ for all n $\geq$ N. This looks a little more intimidating, so we appeal to some theorems.
l'Hôpital's rule looks promising. Recall,

$\displaystyle {\frac{d}{dn}}$ ln n = $\displaystyle {\frac{1}{n}}$ and $\displaystyle {\frac{d}{dn}}$ $\displaystyle \sqrt{n}$ = $\displaystyle {\frac{1}{2\sqrt{n}}}$
Therefore

$\displaystyle \lim_{n\rightarrow \infty}^{}$ $\displaystyle {\frac{\ln (n)}{\sqrt{n}}}$ = $\displaystyle \lim_{n\rightarrow \infty}^{}$ $\displaystyle {\frac{2\sqrt{n}}{n}}$ = $\displaystyle \lim_{n\rightarrow \infty}^{}$ $\displaystyle {\frac{2}{\sqrt{n}}}$ = 0.
Therefore, appealing to Theorem 2,

lnn = O( $\displaystyle \sqrt{n}$ ) but $\displaystyle \sqrt{n}$ $\displaystyle \neq$ O(ln n).

Example 4:

Show f (n) = log_an = $\Theta$ (log_bn), that is, all logarithmic functions grow at identical rates.

We need to find constants c, d > 0 and a natural number N > 0 such that dlog_bn $\leq$ log_an $\leq$ clog_bn for all n $\geq$ N.
The simple trick is to know the conversion formula

log_an = $\displaystyle {\frac{\log_b n}{\log_b a}}$ ,
which is obvious once you rename things, say

x = log_an and y = log_bn.
Then, by the definition of logarithms

a^x = n = b^y
or, taking logarithms (base b) of the left and right side of the equation

log_b(a^x) = xlog_b(a) = yb
or, going back to original names

log_anlog_ba = log_bn,
from which the conversion formula is immediate What this says is, we can pick d = c = ${\frac{1}{\log_b a}}$ and N = 1.

General rules

1.: If p < q then n^p = O(n^q) and n^q = $\Omega$ (n^p).
2.: If P(n) = a_kn^k + a_{k - 1}n^{k - 1} + ^...a₁n + a₀ is a polynomial of degree k, then P(n) = O(n^k).
3.: Logarithms log_an, a > 1 grow slower than any power n^k.
4.: Exponentials aⁿ, a > 1 grow faster than any power n^k.

Problems

Problem 1:

Show that if f (n) = $\Theta$ (g(n)) then g(n) = $\Theta$ (f (n)).

Problem 2:

Show that if f (n) = O(g(n)) then g(n) = $\Omega$ (f (n)).

Problem 3:

If f (n) = $\Omega$ (g(n)) and g(n) = $\Omega$ (f (n)) and then f (n) = $\Theta$ (g(n)).

Problem 4:

Place the following functions in O order (from the slowest growing to the fastest growing).

lg(lg(n))	2^lg(n)	( $\displaystyle \sqrt{2}$ )^lg(n)	n²	n!
(3/2)ⁿ	n³	lg²n	n^{lg n}	log(log(n))
n*2ⁿ	lg n	1	ln n	2^{lg n}
eⁿ	5n³ + 4n²	$\displaystyle \sqrt{n^2}$	lg(2ⁿ)	2^{ln n}
4^{lg n}	(n + 1)!	n² + n + 3	n	2ⁿ

Problem 5:

Prove that $\lfloor$ n $\rfloor$ = $\Theta$ (n).

Problem 6:

lg n! = O(g(n)) for a common function g(n). What is g(n)?

Problem 7:

Do all exponentials grow at the same rate? That is,

Is aⁿ = $\Theta$ (bⁿ) for all a, b $\geq$ 1?

Justify your answer.

Problem 8:

Define the iterated logarithm, lg^*n, as follows.

lg⁽ⁱ⁾n = $\displaystyle \left\{\vphantom{\begin{array}{lr} n & \mbox{if $i=0$} \\ \lg(... ...undefined} & \mbox{if $i>0$ and $\lg^{(i-1)}n \leq 0$}. \\ \end{array}}\right.$ $\displaystyle \begin{array}{lr} n & \mbox{if $i=0$} \\ \lg(\lg^{(i-1)}n) & \... ... \mbox{undefined} & \mbox{if $i>0$ and $\lg^{(i-1)}n \leq 0$}. \\ \end{array}$ $\displaystyle \left.\vphantom{\begin{array}{lr} n & \mbox{if $i=0$} \\ \lg(\... ...undefined} & \mbox{if $i>0$ and $\lg^{(i-1)}n \leq 0$}. \\ \end{array}}\right.$

The iterated logarithm function is defined as

lg^*n = min{i $\displaystyle \geq$ 0 : lg⁽ⁱ⁾n $\displaystyle \leq$ 1}

What are lg^*2, lg^*4, lg^*16, lg^*65536, lg^*(2⁶⁵⁵³⁶)equal to?

William Shoaff
1999-06-07