Solutions of F(x)=x and F(F(x))=x

Let us assume that $F(F(x)) = x$ has a solution $d$ then $F(F(d)) = d.$ Let $F(d)=e;$ then $F(e)=d,$ thus, $(d, e)$ and $(e, d)$ are points on the curve $y = F(x).$

If $e = d$ then $F(d)=d$ which is a contradiction.

If $e \ne d$ then $(d, e)$ and $(e, d)$ lie on different sides of $y = x$ thus continuity of $F$ implies that $F$ must cross the line $y = x$ somewhere, say at $x=k,$ making $F(k)=k$ which is again a contradiction. It follows that $F(F(x)) = x$ has no solution.

Note: The above argument can be applied to any continuous function $F.$ A more rigorous proof can be given based on the above argument.

By the stipulation of the problem the roots $x_1,$ $x_2$ of $f(x)=F(x)-x$ are complex and conjugate: $x_{1}\in\mathbb{C}\setminus\mathbb{R}$ and $x_{2}=\overline{x_{1}}.$ So $f(x)=a(x-x_{1})(x-x_{2});$ and, hence,

$\begin{align} F(F(x))-x &= F(F(x))-F(x)+F(x)-x\\ &=a(F(x)-x_{1})(F(x)-x(_{2})+a(x-x_{1})(x-x_{2}). \end{align}$

But

$\begin{align} F(x)-x_{1} &=F(x)-x+x-x_{1}\\ &=a(x-x_{1})(x-x_{2})+(x-x_{1})\\ &=a(x-x_{1})(x-x_{2}+1). \end{align}$

By analogy, $F(x)-x_{1}=a(x-x_{1}+1)(x-x_{2}).$

It follows that $F(F(x))-F(x)=a(x-x_{1})(x-x_{2})[a(x-x_{1})+1][a(x-x_{2})+1]$ and, therefore,

$F(F(x))-x=a(x-x_{1})(x-x_{2})\big([a(x-x_{1})+1][a(x-x_{2})+1]+1\big)$

which clearly has no real solutions. This is because

$ [a(x-x_{1})+1][a(x-x_{2})+1]=[a(x-x_{1})+1]\overline{[a(x-x_{1})+1]}. $

with the discriminant essentially different from $b^2-4ac.$

In the notations of the previous proof, either $f(x)\gt 0$ or $f(x)\lt 0,$ for all $x\in\mathbb{R}.$ Assume the former. Then, in particular, $f(F(x))\gt 0,$ for all $x\in\mathbb{R}.$ Even more so $f(F(x))+f(x)\gt 0.$ But

$\begin{align} f(F(x))+f(x)&=[F(F(x))-F(x)]+[F(x)-x]\\ &=F(F(x))-x, \end{align}$

implying $F(F(x))-x\gt 0,$ for all $x\in\mathbb{R}.$ The case $f(x)\lt 0$ is treated similarly. Thus for no real $x,$ $F(F(x))=x.$

This is a simplified paraphrase of the previous proof, with a tint of geometric appeal.

Either $F(F(x))\gt x$ or $F(F(x))\lt x,$ for all $x\in\mathbb{R}.$ This means that the graph of $y=F(x)$ remains either above or below the main diagonal $y=x.$ Assume the former: $\forall x\in\mathbb{R},$ $F(x)\gt x.$ Then $F(F(x))\gt F(x)\gt x,$ implying that $y=F(F(x)),$ stays over the diagonal. Similarly, in the case $F(x)\lt x,$ the graph of $y=F(F(x))$ remains below the diagonal.

Note: This solution holds for any function $F,$ not necessarily quadratic, whose graph is entirely in one of the halfplanes defined by the diagonal $y=x,$ nor necessarily continuous.

Here's an example of a discontinuous function for which the statement does not hold:

$f(x)=\begin{cases} \sqrt{2} & \mbox{if } x\in\mathbb{Q},\\ 0 & \mbox{if } x\in\mathbb{R}\setminus\mathbb{Q}. \end{cases}$

Clearly, $f$ has no fixed point. However $f(f(0))=f(\sqrt{2})=0.$

The problem has been posted in a comment on a page at the site; Solution #1 came as a response to that comment. Solutions 2 and 3 are due to Leo Giugiuc as is the example of the discontinuous function in Solution 4. I am grateful to Leo for helping me with Solutions 2 and 4.