Folder thumbs are created with a "folder.jpg" file. These thumbs are applied to folders (and depending on your media type, the entries inside the folders as well).
So to try and recap what you've told us so far,
You've got some folders and you're trying to apply icons to them, but only a few of these folders are coming up with the custom thumbs?
What's happening to the items inside the folders?
You can create a debug log by holding X+Y as XBMC loads. It'll be generated in your install folder. Note that while running in debug mode you'll get to view a running tally of how much RAM XBMC has spare, keep an eye out to see if that dips a bit low. You can dump a copy of the log onto
PasteBin.
The icon cache settings deal with how large much space is set aside per icon, to my understanding. Hence the amount of memory XBMC will be trying to use is cache size times the amount of icons. You're more likely to have issues by cranking that figure up, so take it back down to the default setting for now.