The Book Industry Study Group (of which the vast majority of publishers, via the Association of American Publishers, are members) employs a system of BISAC (Book Industry Standards and Communications) codes, which provide a framework (both conceptual and technological) for identifying a title's genre.
The most recent BISAC code list can be found here; keep in mind that whatever you're writing is going to have to be coded under at least one of these categories (many books have a secondary or even tertiary BISAC code), so try not to go too bananas with your proposed genre (e.g. "it's a literary romantic psycho-suspense thriller with elements of science fiction Western").
I suppose you could go in under NON-CLASSIFIABLE if you felt you had to, but 1.) that's not super helpful, 2.) it won't make your book even remotely easy to find in book stores, and 3.) your code will be "NON000000," which is one of the more depressing-looking ones. It's almost as if the book is crying out for a better BISAC code ("No! NOOOOOOOOO").
So, for example, if you're writing paranormal romance, your BISAC code will be FIC027120 (FICTION / Romance / Paranormal). If you're writing a book about stamp collecting, you'll get ANT044000 (ANTIQUES & COLLECTIBLES / Stamps). They get pretty specific, so it's unlikely you'll write something that can't easily be mapped to the system.
If you're writing something that could just as easily be one genre as another (say, science fiction or thriller), it's no big deal to have one as your primary BISAC and the other as your secondary (though the primary will pretty much determine who buys the book at the larger accounts like Barnes & Noble and where the book will eventually live in the store).
All the details of BISAC coding will be handled by your publishing house, but if for some reason you're concerned about it, you can always ask via your agent. (Remember, kids: always ask your agent first!)
And now you know!