Let's distinguish between rotational kinematics and rotational dynamics. Both topics are discussed in sophomore level mechanical engineering texts. Sorry I can't give you the name of a specific text, but that's the general area to look at.Rotational kinematics can be reduced to a fairly simple formal procedure, which I should warn you is NOT covered well in many of the engineering texts I mentioned above. Since dynamics never needs derivatives of position higher than the second derivative, many books give rather "crippled" formulas which are inflexible in application and don't extend to higher derivatives.
I can give you a better way right now:
Let a stationary coordinate system fixed to the ground be numbered zero, and let its unit basis vectors be i0, j0, and k0. Let coordinate system 1 have basis vectors i1, j1, and k1, let its origin be at r10 and let it rotate around this origin with angular velocity w10. The subscript "10" indicates "system 1 relative to system 0." Continue, defining system 2 located at r21 rotating with w21 relative to system 1, and in general system n located at rn n-1 rotating with wn n-1 relative to system n-1.
Now all you have to do is notice five things:
1) The first subscript of a vector tells you what it is pointing at, and the second tells you what coordinate system it is specified in.
2) Displacements are cumulative, so that the origin of system n is located at rn1 = r10 + r21 + ... +rn n-1 relative to system 0,
3) Rotations are cumulative, so that system n is rotating with a total angular velocity of wn1 = w10 + w21 + ... +wn n-1 relative to system 0.
4) Because the basis vectors are rotating, every time you take a time derivative, you must use the product rule, treating the basis vectors as variables.
5) The time derivative of a basis vector v rotating with velocity w is always the cross product w X v.
If you apply these rules consistently, you can compute velocities and accelerations as seen in the base system (system 0) for an object described by any number of nested coordinate systems. Just remember to label each vector by what it points to and which origin it comes out of, and the rules do the rest.
Incidentally, all of these rotations and cross products can be written as the action of matrices on the object's position vector, just as Alex mentioned above. I have not found the quaternion formalism particularly more or less advantageous in practice than the matrix method. On the plus side, the notation is more compact, but on the minus side, you have to learn to understand quaternions first. But on the plus side again, quaternions are really very interesting and well worth looking at just for the pleasure of learning about them.
Now as to dymanics, that is another problem. Kinematics can specify acceleration, but it cannot connect it to applied force. For that you need the moment of inertia matrix I and the equation torque = Ia, where a is the angular acceleration. (Note that since I is a matrix, the vector a need not be parallel to the torque.) This equation is not always the most useful one, because I may be changing. In fact, since I depends on the shape of an object, when the object rotates, I must rotate along with it, in the sense that the principal axes of inertia (which are along the eigenvectors of I) are tied to the shape of the object. Therefore, I is usually changing, and a better formulation is that torque = dH/dt, where H is the angular momentum Iw.
Usually what is done is to attach a coordinate system to the rotating object, with principal axes used as the coordinate axes. Then in this coordinate system, I is constant and w is zero, so it appears that dH/dt must be zero...but wait, we have nonconstant unit coordinate vectors, so the time derivative acts on them, and if you follow the five rules above, you can find the torque on the object, as seen by the stationary coordinate system 0.
I think this answers most of what you wanted to know, but for more detail, consult mechanical engineering textbooks with the words kinematics or dynamics in their titles.
Hope this helps!
--Stuart Anderson