That all depends on what kind of invitation (that is, the occasion or event), what sort of relationship you have with the host and what sort you would like to continue to have, and your reason for declining.
A lifelong friend inviting you to his father's wedding, your son's fiancee's parents inviting you to dinner, a coworker inviting you to a lingerie party, and a neighbor inviting you to her church would all call for different responses unless your answer is an enthusiastic "Yes, thank you!"