It really would depend on what exercise you're actually doing with the dumbbell. 20 lbs is pretty heavy for a person just beginning weight lifting. Be sure that you've been trained how to properly do whatever exercise. Maintaining proper form during an exercise is critical to increasing muscle mass- if you aren't exercising the proper muscle group, you can't expect to see gains.
You might want to consider purchasing an adjustable dumbbell, which will let you adjust the weight you're using to a particular exercise. Also, as you improve, you'll be able to increase the weight you're using without having to go out and buy another dumbbell.