How do you install a PVC clean out into a cast drain system in a house?

With a Saddle Clamp. Looks like an 8 to 10 inch PVC T split lengthwise and held on with 2 U bolts. Place it where you want the clean out, mark around the inside of the pipe where the clean out will be and drill a series of 1/8th inch holes around the circle as close together as possible. When all the holes are drilled, elongate the holes to connect one to the next one. When most of them are connected tap on the plug and it will break out. Catch it before it goes down the drain. You can drill 1 hole in the center before you start and put a screw in it to hold on to the plug. Seal the saddle clamp to the pipe with Plumbers Putty and secure it with the U bolts. Install the PVC clean out plug for the size pipe you have.

You may already have a clean out in the main stack before it leaves the house. These are usually rusted solid. You can take out the old one and install a PVC one in it's place.

Saddles are not allowed in civilized countries.

Only a non skilled would dare suggest this type of fitting as a saddle does not allow for a 45 degree angle for proper flow