Some things that came to mind as I listened to Billy Collins talk about being a poet and writing in general:
Poets start from scratch more often. Whether that's good or bad I can't say, but the novelist has a thread to follow for months, even years, while the poet faces a blank page every day or two.
Poets are allowed to break the rules of writing (such as they are). Playing with language in encouraged. Novelists have more of a job to do, a goal to reach, less time for wordplay, even with more words.
Calling yourself a writer is difficult, possibly more so for poets than for other writers. We feel compelled to wait until someone else defines us with terms like "author" and "poet" before we are comfortable using them.
And finally, I got the same sense from Collins that I get from others I would label "true" writers: they can't not write. It isn't the money (HAH!) the fame, the chance to become Poet Laureate, or even the satisfaction of saying, "I'm a novelist/poet/whatever." It's a lot like an addiction: we recognize the perils, but the attraction overcomes all.