Mysteries. Thrillers. Suspense. What's the difference?
Well, Nathan has answered this before, but in short:
• Thrillers are action-oriented;
• Suspense novels are danger-oriented, but not necessarily action-oriented;
• Mysteries, are, well... mystery-oriented, regardless of whether there's any action or danger involved. Is there a riddle to be solved? A question to be answered? &c.
Hopefully all is clear now. Next piece of business: how is this genre selling?
If you're interested in the UK fiction market—which isn't really substantially different from the US market, at least in this category—things are looking up for mystery/thriller (at least, as of last year). As you can see here (scroll down), mystery sales account for almost a quarter of all adult sales (units), so as long as your writing is solid and you've got a more-or-less original idea, you should be okay. As I've already mentioned, escapist fiction is the ideal market right now, so mystery/thriller/suspense (including all its crossovers, such as sci-fi thriller and romantic suspense) is a great place to be in 2009. And honestly, from what I hear, it's also a great place to be over the next few years, so if you're writing in this genre: carry on.
So again, with all the usual disclaimers (not legal advice, no warranties, exchanges, refunds, substitutions, &c), my guess is that romantic suspense, police/procedural thriller, mystery/thriller/suspense with female protagonist/detective, sci-fi/military thriller, and historical mystery (mayhap steampunk?) will lead the pack, although I honestly expect the rest of the sub-genres to continue to do well. Anything with sci-fi/fantasy/romance crossover (to super capitalize on the escapist/happy ending fiction theory) is, I think, a good idea (provided your book is engaging and well-written).
Now: what's the average advance?
As we saw yesterday and the day before, it's difficult to pin this sort of thing down, but from what I've observed in national account adult sales, the average advance for mystery/thriller/suspense is roughly in line with the average adult novel advance, which is to say, around $10,000 or so (in my experience, closer to $8,000). You could make as little as $1,000 on your debut novel and as much as a couple million; it all depends. Again, if you have an agent, I'd certainly ask him or her about this.
Finally, one major caveat: if your mystery/suspense/thriller is coming out this September (particularly in the second half of the month), you might be a little overshadowed. Otherwise, though, you should be fine.