Dear Alex and Tuan,

>A, B are two points on a circle (O). There are two arcs AB
>and two circular segments AB but we choose the small one.
>How to cut from this circular segment one right triangle
>with maximum area?

This is a very interesting problem.
The solution exhibits significant interest, with many symmetries, alignments and circles worthy of further exploration.

Consider the minor arc APB of circle with centre O and diameter AC = 2.r. Let AB=2.h and let the perpendicular from O to S on BC =k. Let APQ be the right triangle within the minor segment, with Q on BC.
(see Fig 1).

Using Cartesian Coords with origin at A and x-axis along ASQB:

Equation of arc APB is

y = ((r^2-(x-h)^2)^1/2 + k

Area tAPQ = x.y/2 + y^3/(2.x)

Differentiate and set dA/dx = 0 and solve for x

The resulting equation is a sextic, so no easy closed form solution is likely.
However, for example, if h=5, k=3, then r=sqrt(34) gives
x=6.283 and theta=23.161 degrees.

A better alternative is to use Polar Coords, with the Pole at A, and theta=0 along x-axis. Then C=(2.r, -thetaC) where tan(thetaC) = k/h
and P=(rho,theta) with rho = 2.r.cos(theta+thetaC)

The structure of the solution is shown in Fig.2
Observe that 2.r.sin(theta+thetaC) = rho (from tAPC),
PC.cos(theta).sin(theta).2 = DE = rho (from kite EPDC),
rho = 2.t.cos(theta) (from tAPE),
PD = PE = EQ = QD = t (from rhombus EPDQ),
AE = EP (from tAPE),
ED = 2.t.cos(theta) = rho (from //gram APDE).

So, formally, in Polar Coords: Let AE = EQ = PD = t
. project APD onto AB so t.(1 + 2.(cos(theta)^2) = 2.h ...(1)
. from tAPC 2.r.cos(theta+thetaC) = 2.t.cos(theta) ...(2)
Solve (1) and (2) simultaneously for t and theta.

OR use rho.tan(theta+thetaC) = PC = rho/sin(2.theta)
.: tan(theta+thetaC) = 1/sin(2.theta)

The resulting equation is again a sextic, with either of the above approaches, but with only even powers present, it may be solved as a cubic, giving some advantage over the Cartesian approach. However the real advantage of the polar solution is the way in which it reveals the underlying structure of the solution geometry.

( Figures to follow, in a separate post ).

Regards, Peter Scales.

Peter Scales
Peter Scales