The major issues hear: CPU power, amount of RAM and speed of it, harddrive performance, ASIO soundcard, and software.
check out www.pricewatch.com for cheap computers or go around to yard sales and pick up non working computers for parts. I just built a dual core intel machine with 2 gb of ram for under $150 using spare parts I found.
Make sure that your cpu is at least dual-core. You need at least 1-2 gb of ram. The higher the number the better. Like 800 is better than 400. You need at least 160gb of harddrive space with a seperate harddrive for recording if you can afford it.
For soundcard get anything that has an ASIO driver. Any studio soundcard will do. Check out M-Audio soundcards. Firewire is better than USB and some people prefer PCI soundcards. Don't buy soundblasters.
Many soundcards come with recording software. If you don't have any DAW software you might want to buy a soundcard that comes with Cubase 4 Le. It's a great piece of software for free with good soundcards such as the Presonus Firebox.
The only softwares that i've really liked have been Cubase/Nuendo and ProTools. Sonar, Reaper, and Ableton Live are all good but they just weren't for me. Remember that Reaper is a great value.
For making instrumentals outside of recording software I like FL Studio and Reason.