Answer:
You should politely decline, giving whatever reason you choose to give.
You could use a generic reason, such as:
I make it a practice to not write recommendation letters for anyone who has worked with me.
You could deflect the request by suggesting that someone else is more qualified to write a strong recommendation.
If your letter would be negative rather than positive, and you feel confident with being honest, you could gently communicate that you would not want your negative comments to hurt the person's chances of getting the position / getting into school / etc.
It's stressful to ask for letters of recommendation, so deal with the request promptly. You can be friendly, polite, but clear about declining the request. If the person hounds you to write the letter, you could also use the badgering as the reason you've decided to not write the letter.