• You have an idea for a book that would only be targeted at an extraordinarily small "market," i.e. your family. If you want to bind your great-grandmother's recipes into a cookbook, create a collection of stories for your children, &c, and you only need a few dozen copies, self-publishing is for you.
• For whatever reason, you have no interest in selling your work and merely want to see it in print before you die.
• Alternately, you have no interest in selling your work and merely want to disseminate it widely on the Internet as a (fre)e-book. (If this is the case, though, you might not even really need the self-publishing company, unless you need their website to legitimize your book.)
• You do not have enough copies of other peoples' books to keep your coffee table level.
I consider the following reasons for self-publishing to be very bad:
• Your book has been rejected by every agent and his/her mom, so now you're going to show them/the world/your own mom/&c that you really are a published writer.
• You believe you can sell more books on your own than you could through a traditional publisher, so you're going to forgo the whole system.
• You say you have no interest in selling your work and merely want to disseminate it widely on the Internet, but secretly believe as soon as it's out there you'll start getting phone calls from all those silly agents and editors, offering seven figure advances and instant literary stardom. Later, Brad Pitt will call to politely ask if he might be considered for the role of your protagonist once the details of the movie deal(s) are all hammered out.
• You believe your book is too literary for 99.9999% of agents/publishers and won't sell within the traditional publishing framework because you and your book are just too darn smart.
Before I go much further, I want to make this clear: I think the traditional system is flawed. All systems are necessarily incomplete. (That's a math joke, folks. I don't really think Gödel's incompleteness theorems apply to books. Man, if only you'd read my self-published book, 1010010010101111 Binary Math Jokes—which, by the way, is way too intellectual for the average agent, editor, or reader—you'd get that.)
All joking aside, though, just because the system isn't perfect doesn't mean you're better off avoiding it altogether. Consider these stats (and also these) over at How Publishing Really Works, courtesy of this SFWA article. Compare that to the sales of the average traditionally published book—around 12,000 copies—and you'll understand my general skepticism. Very occasionally, a self-published novel will be something that was somehow overlooked by the publishing industry as a whole and is actually quite good and/or salable. 99%+ of the time, however, these books are either written by the functionally illiterate, are tangled messes of inane plot and one-dimensional characters, do not appeal to the vast majority of readers, are way too long or way too short, or some combination of all of these. In short, most self-published novels are crap.
You might argue that most traditionally published books are crap, too, and if that's the case, you could very well be that guy who believes he and his book are too smart for the entire world. Whether or not this is true, it is a sad and inescapable fact that the market for your book is a subset of all the people in the entire world, so you're S.O.L. even if you and your book really are that smart, which is unlikely. I mean, really, how many Prousts can there be?
So, in summary:
• If you just want a couple dozen copies of your book for family and friends, my recommendation is: self-publish.
• If you ever want to earn money from your book, my recommendation is: do not self-publish.
• If you've tried and tried and done absolutely everything humanly possible and still can't sell your novel, it's probably not very good. Lock it in a drawer and write a better one.
The publishing industry is a creaking, hulking, slow-moving, kerosene-burning juggernaut of 19th-century jerry-rigged methods and models all built pick-a-back one atop another, but it does adapt and is your best bet for getting an audience and a halfway decent check for your writing. Unless you're one of those very few who are better off self-publishing (as described above), get back to work and write something engaging that any agent or editor would be proud to show the world.