Answer:
This is a rather broad topic and it really just depends on what your interests are.
In general, avoid coins advertised on TV, coins from private mints (like the Franklin Mint) and coins from tiny nations you haven't heard of. These generally sell for initially high prices but sell only for its metal content. If you like those designs, simply buy it secondhand from a coin shop at a fraction of the issue price.
But as for what you can collect, it is limitless. Myself, I specialize in British Coins, with some ancient, medieval, world and a bit of US coins. It also depends on your budget, with a high budget, expensive coins like a full set of Gold Sovereigns, rare US pattern coins or early proof sets are collectable. If you have a moderate budget, consider coins like US silver dollars, Standing Liberty Quarters or British maundy sets. If you are on a tight budget, consider common-date Mercury dimes, even in mint state they shouldn't cost more than $20 a piece, or assemble a set of wheat pennies, aside from a few key dates (1909 S, 1914 D) they are all very affordable. Or consider just getting various world coins from a local dealer, they are very cheap. Coins with no precious metal content usually sell for 6 coins per dollar, making having a wide variety of coins from nearly every nation possible. Despite what you might think, ancient coins are easily affordable. Common bronze coins in decent shape might only cost $15, while some silver coins might cost around $30 in decent shape.
Bottom line: collect what you want. There's an old saying: buy the book first. Of course, due to the internet, you don't need to shell out the money for a book (though there are some books that are essential if you decide to seriously collect, namely the red book for US coins and Spinks for British Coins). But research before you buy and shop around.