I started out with:
Boss 202, Dr.Groove Drum machine that kicks out some cool beats & bass lines for a cool price. I bought it on Ebay for around $125 and it's hella easy to use. But like the homeboy stated, you are limited. It has around 227 sounds.
Roland MC-303, GrooveBox that I bought for around the same price as the Boss on Ebay also. I got this for all the bass synths and keyboard sounds. It also fuctions as a drum machine, but changing the paramitters for that function is limited. I hooked up a cheap midi equipped keyboard to play the sounds as the 303 has a small number of keys.
Digitech Vocal 300 & Sure Mic with a pop filter. I got the Digitech for vocal fx, and Sure mic because of its durability. A pop filter is a must!
Roland VS-840GX to record your music. Also bought off of Ebay. Not too complicated to use. Records to zip disks. You have to convert files to .wav then download to your computer.
I bought this setup all together for around $850 including all cables.
Option #2
I recently bought FL Studios V.5 XXL Edition and havent touched my other equipment since. You will never run out of sounds, because you could always buy new sample disks. I made a beat within 3hrs and burned it to disk. I havent gotten all into it yet to record vocals, but it is very easy to use. You will need a computer with a sound card, midi controller, mic, skillz & patience.
Good Luck!