In the literature, the so-called RK4 is given as following: we first define the following coefficients
then
This is the most important iterative method for the approximation of solutions of ordinary differential equations
This technique was developed around 1900 by the German mathematicians C. Runge and M.W. Kutta. In order to study its stability, we use the model problem
In other words, we replace 
Applying the above discussion to RK4 method, we see that
which implies
which yields
Thus
![\displaystyle\frac{{{y_{i+1}}}}{{{y_{i}}}}= 1+\frac{{\Delta t}}{6}\left[{\lambda+2\lambda\left({1+\frac{1}{2}\Delta t\lambda }\right)+2\lambda\left({1+\frac{1}{2}\Delta t\lambda\left({1+\frac{1}{2}\Delta t}\right)}\right)+\lambda\left[{1+\Delta t\lambda\left({1+\frac{1}{2}\Delta t\lambda\left({1+\frac{1}{2}\Delta t}\right)}\right)}\right]}\right]](http://l.wordpress.com/latex.php?latex=%5Cdisplaystyle%5Cfrac%7B%7B%7By_%7Bi%2B1%7D%7D%7D%7D%7B%7B%7By_%7Bi%7D%7D%7D%7D%3D+1%2B%5Cfrac%7B%7B%5CDelta+t%7D%7D%7B6%7D%5Cleft%5B%7B%5Clambda%2B2%5Clambda%5Cleft%28%7B1%2B%5Cfrac%7B1%7D%7B2%7D%5CDelta+t%5Clambda+%7D%5Cright%29%2B2%5Clambda%5Cleft%28%7B1%2B%5Cfrac%7B1%7D%7B2%7D%5CDelta+t%5Clambda%5Cleft%28%7B1%2B%5Cfrac%7B1%7D%7B2%7D%5CDelta+t%7D%5Cright%29%7D%5Cright%29%2B%5Clambda%5Cleft%5B%7B1%2B%5CDelta+t%5Clambda%5Cleft%28%7B1%2B%5Cfrac%7B1%7D%7B2%7D%5CDelta+t%5Clambda%5Cleft%28%7B1%2B%5Cfrac%7B1%7D%7B2%7D%5CDelta+t%7D%5Cright%29%7D%5Cright%29%7D%5Cright%5D%7D%5Cright%5D&bg=ffffff&fg=000000&s=0 "\displaystyle\frac{{{y_{i+1}}}}{{{y_{i}}}}= 1+\frac{{\Delta t}}{6}\left[{\lambda+2\lambda\left({1+\frac{1}{2}\Delta t\lambda }\right)+2\lambda\left({1+\frac{1}{2}\Delta t\lambda\left({1+\frac{1}{2}\Delta t}\right)}\right)+\lambda\left[{1+\Delta t\lambda\left({1+\frac{1}{2}\Delta t\lambda\left({1+\frac{1}{2}\Delta t}\right)}\right)}\right]}\right]")
which is nothing but
Therefore, the stability condition is given as follows
and 
are two sequences. Then,![\displaystyle\sum_{k=m}^n f_k(g_{k+1}-g_k) = \left[f_{n+1}g_{n+1} - f_m g_m\right] - \sum_{k=m}^n g_{k+1}(f_{k+1}- f_k)](http://l.wordpress.com/latex.php?latex=%5Cdisplaystyle%5Csum_%7Bk%3Dm%7D%5En+f_k%28g_%7Bk%2B1%7D-g_k%29+%3D+%5Cleft%5Bf_%7Bn%2B1%7Dg_%7Bn%2B1%7D+-+f_m+g_m%5Cright%5D+-+%5Csum_%7Bk%3Dm%7D%5En+g_%7Bk%2B1%7D%28f_%7Bk%2B1%7D-+f_k%29&bg=ffffff&fg=000000&s=0 "\displaystyle\sum_{k=m}^n f_k(g_{k+1}-g_k) = \left[f_{n+1}g_{n+1} - f_m g_m\right] - \sum_{k=m}^n g_{k+1}(f_{k+1}- f_k)")
.
, it can be stated more succinctly as![\displaystyle\sum_{k=m}^n f_k\Delta g_k = \left[f_{n+1} g_{n+1} - f_m g_m\right] - \sum_{k=m}^n g_{k+1}\Delta f_k](http://l.wordpress.com/latex.php?latex=%5Cdisplaystyle%5Csum_%7Bk%3Dm%7D%5En+f_k%5CDelta+g_k+%3D+%5Cleft%5Bf_%7Bn%2B1%7D+g_%7Bn%2B1%7D+-+f_m+g_m%5Cright%5D+-+%5Csum_%7Bk%3Dm%7D%5En+g_%7Bk%2B1%7D%5CDelta+f_k&bg=ffffff&fg=000000&s=0 "\displaystyle\sum_{k=m}^n f_k\Delta g_k = \left[f_{n+1} g_{n+1} - f_m g_m\right] - \sum_{k=m}^n g_{k+1}\Delta f_k")
,
.![\displaystyle\sum_{k=m}^n f_k(g_k)-g_{k-1}) = \left[f_{n+1}g_n - f_m g_{m-1}\right] - \sum_{k=m}^n g_k(f_{k+1}- f_k)](http://l.wordpress.com/latex.php?latex=%5Cdisplaystyle%5Csum_%7Bk%3Dm%7D%5En+f_k%28g_k%29-g_%7Bk-1%7D%29+%3D+%5Cleft%5Bf_%7Bn%2B1%7Dg_n+-+f_m+g_%7Bm-1%7D%5Cright%5D+-+%5Csum_%7Bk%3Dm%7D%5En+g_k%28f_%7Bk%2B1%7D-+f_k%29&bg=ffffff&fg=000000&s=0 "\displaystyle\sum_{k=m}^n f_k(g_k)-g_{k-1}) = \left[f_{n+1}g_n - f_m g_{m-1}\right] - \sum_{k=m}^n g_k(f_{k+1}- f_k)")
.![\displaystyle\sum_{k=m}^n f_k\Delta g_{k-1} = \left[f_{n+1}g_n - f_m g_{m-1}\right] - \sum_{k=m}^n g_k \Delta f_k](http://l.wordpress.com/latex.php?latex=%5Cdisplaystyle%5Csum_%7Bk%3Dm%7D%5En+f_k%5CDelta+g_%7Bk-1%7D+%3D+%5Cleft%5Bf_%7Bn%2B1%7Dg_n+-+f_m+g_%7Bm-1%7D%5Cright%5D+-+%5Csum_%7Bk%3Dm%7D%5En+g_k+%5CDelta+f_k&bg=ffffff&fg=000000&s=0 "\displaystyle\sum_{k=m}^n f_k\Delta g_{k-1} = \left[f_{n+1}g_n - f_m g_{m-1}\right] - \sum_{k=m}^n g_k \Delta f_k")
.
such that 
, for a given function 
. Such an ![[a, b]](http://alt1.mathlinks.ro/latexrender/pictures/5/d/3/5d34cf1804e09e81ad12a25ceed08832df2abebd.gif)
, with 
and 
of opposite sign. By the Intermediate Value Theorem, there exists a number 
in 
with 
. Although the procedure will work when there is more than one root in the interval 
, we assume for simplicity that the root in this interval is unique. The method calls for a repeated halving of subintervals of 
and 
, and let 
be the midpoint of ![[a,b]](http://alt1.mathlinks.ro/latexrender/pictures/1/4/7/1475c39442659c6febe03b2e1809dbde6ec5fb5a.gif)
; that is, 
, then 
, and we are done. If 
, then 
has the same sign as either 
or 
. When 
, and we set 
and 
. When 
, and we set 
and 
. We then reapply the process to the interval ![[a_2, b_2]](http://alt2.mathlinks.ro/latexrender/pictures/b/e/c/becd98dfe69317aff9592944785a82078d91e3c3.gif)
.
if 
. In this section we consider the problem of finding solutions to fixed-point problems and the connection between the fixed-point problems and the root-finding problems we wish to solve. Root-finding problems and fixed-point problems are equivalent classes in the following sense:
or as 
. Conversely, if the function 
has a zero at 
, the Newton’s method generates the sequence 
by
, we have 
. The approximation 
is chosen in the same manner as in the Secant method, as the 
and 
. To decide which secant line to use to compute 
, we check 
. If this value is negative, then 
. If not, we choose 
determines whether we use 
. In the latter case a relabeling of 
denote the solution of the initial value problem 
as the difference between 
and the specified boundary value 
. 
has a root, and that root is just the value of 
which yields a solution 
of the boundary value problem. The usual methods for finding roots may be employed here, such as the bisection method or Newton’s method.
is the solution to the initial value problem
is the solution to the initial value problem
then 
for all 
. Thus 
for all 
for all 
, 
. This is self-satisfied.
, 
.
.
, and 
plotted in the first figure. Inspecting the plot of 
and 
. Some trajectories of 
are shown in the second figure.
and 
(approximately).
equal to 
, 
, 
(red, green, blue, cyan, and magenta, respectively). The point 
is marked with a red diamond.
is the function to be minimized, 
(
) are the inequality constraints and 
(
) are the equality constraints, and 
and 
are the number of inequality and equality constraints, respectively.
and the constraint functions are 
and 
. Further, suppose they are continuously differentiable at a point 
. If 
) and 
(
, 
(
for all 
) is positive-linear dependent if there exists 
not all zero such that 
)
for equalities and 
for inequalities than it doesn’t exist a sequence 
such that: 
therefore 
and 
thus 
.
and 
for all 
active in 
are continuously differentiable convex functions and the equality constraints hi are affine functions. It was shown by Martin in 1985 that the broader class of functions in which KKT conditions guarantees global optimality are the so called invex functions. So if equality constraints are affine functions, inequality constraints and the objective function are continuously differentiable invex functions then KKT conditions are sufficient for global optimality.
is 
.) Given this definition, each coefficient, 
, 
with 
.
subintervals by the following interior points 
. 
. T
, which will be denoted by 
, is called the step size. The value of 
at step 
, that is, 
. 
. T
, we can compute the value of 
by using the following scheme 
, 
, constants.
, 
,…,
. 
. We denote by 
the following number 
, having all 
for all 
. To do this, for each 
, 
. More precisely, we need to compute 
and for each 
, with 
.
and 
by using the following scheme 
decomposition (also called a 
is a decomposition of 
where 
is an orthogonal matrix (meaning that 
) and 
is an upper triangular matrix (also called right triangular matrix). This generalizes to a complex square matrix 
matrix 
, as the product of an 
unitary matrix 
rows of an 
is an 
upper triangular matrix, 
is 
is 
, and 
the thin 
and we require that the diagonal elements of 
(
if 
, 
and 
decompositions 
being a lower triangular matrix.
, one can solve the Least Square Problem 
where 
and 
with 
. To be precise, by mean of ![Q = \left( {\begin{array}{*{20}{c}} {{{\left[ {{Q_1}} \right]}_{m \times n}}} & {{{\left[ {{Q_2}} \right]}_{m \times \...](http://alt2.mathlinks.ro/latexrender/pictures/d/1/7/d177fe503a9df3ceca3374824c92382e0a3a61d6.gif)
. Thus 
. 
. Assume that after finishing the Gram–Schmidt process, we obtain 
. Therefore, we put
.
, we obtain 
from 
, that is, 
, we firstly project 
onto 
data points were obtained, 
, 
, 
, and 
(shown in red in the picture on the right). It is desired to find a line 
that fits “best” these four points. In other words, we would like to find the numbers 
and 
that approximately solve the overdetermined linear system 
![S(\beta_1, \beta_2)= \left[6-(\beta_1+1\beta_2)\right]^2 +\left[5-(\beta_1+2\beta_2) \right]^2 +\left[7-(\beta_1 + 3\beta_2)\...](http://alt1.mathlinks.ro/latexrender/pictures/8/1/9/81906c08b73f52943931ea68a297e3ac824946b9.gif)
in respect to 
, 
of the line of best fit. The residuals, that is, the discrepancies between the 
, 
, 
, and 
(see the picture on the right). The minimum value of the sum of squares is 
, with 
where 
which fit the equations “best”, in the sense of solving the quadratic minimization problem 
are linearly independent, given by solving the normal equations
is an 
, which is either the field of real numbers or the field of complex numbers. Then there exists a factorization of the form 
is an 
is 
denotes the conjugate transpose of 
in non-increasing fashion. In this case, the diagonal matrix 
while matrix 
. Finally, matrix 
and 
. For the first one, that is, 
or you can perform the same routine as above. That is 
.
. We split into three cases
is a small perturbation on 
is a small perturbation on 
in the sense that the following system 
holds true. This together with the fact that 
gives 
. By some simple calculation, one has
.
is a small perturbation on 
holds true. This together with the fact that 
. By some simple calculation, one has
holds true. This together with the fact that 
. By some simple calculation, one has
is called the condition number of the matrix 
. It is easy to see that 
. If 
, the system 
