Just to reiterate, definitely rent a game prior to purchasing if you worried about spending. Or, you can download some demos through xbox live. The best demo I have seen so far is Need for Speed.
For background info, I am a PC gamer stepping into the world of consoles again after a few years of departure. My opinion may be bias since I skipped the xbox 1 years.
However, after much research on all the games available for the 360, I purchased Kameo, Perfect Dark Zero, Project Gotham 3 and Quake 4. So far, I have played all, but Quake 4 (only because I play it on my PC all the time).
I would say the best non-sports games out there in my opinion (in order):
COD2 (I have this on the PC, so I am not buying it...but it still tops the list)
PDZ
PGR3
Kameo
Quake 4 (I'm dedicated to ID Software...had to buy this for the PC and 360

)
NFSMW
As you can see, all of those games have great replay value, except Kameo. Condemned also looks like an excellent game, but I'm not all up on short replay. I like to play games that you can keep on playing once you have beaten single player mode.
There are so many mixed reviews about these games, so you should try before you buy. However, I am extremely happy with my game choices because I know they will keep me busy until the next wave of games are released.