it is probably the bearings going out in the ac compressor pulley or belt slippage. the reason it is different when it is on is because the belt is not under "load" or has resistance.
The Air conditioning compressor is likely seized. There is a clutch in the pulley that engages the compressor when the A/C is turned on. You can easily check the compressor (with engine off make sure...
Without the year, make or model it could be any of the following..1. Relay.2. Sensors.3. Clutch hub worn out.4. A/C - heater controller.5. PCM or ECM bad.It would be best to scan the vehicles...