Nothing wrong with having an idea of what sounds you might want to use but.......
I almost never try to get all my sounds together before I start a track. Once I start the creative process, my work takes it's own form and direction. Depending on how creative your mind is and how much musical talent you actually have, it could be real easy to wind up not using little or none of the sounds you thought you were going to. Just get the ideas out, you can always come back later and change some sounds if necessary.
I used to make drum kits specifically for each track before I started actually making the track. I figured out that was a serious waste of time. It's way more productive & effecient to take an entire day where all I do is get drum/percussion sounds together, set them up in kits (usually 76 sounds per kit) and then dump 5 or 6 new kits into my gear. Then you spend your time finding sounds you would use rather so when you go to do a new track, you know you can go straight to those kits and won't have to do much looking to find the right sounds.
Also..... Understand that different people work at different speeds and it's all dependant on the persons creativity, talent and ability to translate ideas into a finished product. (By the way, the biggest joke in the world is the millions of people who swear they would be the next Dr. Dre if they just knew how to play the keyboard...you know, because they have all these ideas for HOT BEATS in their head.... *LAUGH*) Anyways.. If it takes you a week, it takes you a week. If it takes you 30 minutes, it takes you 30 minutes. The only thing I can really suggest is to keep practicing. Most people become more effecient the more they practice.