It all has to do with the character of Ennis Del Mar. His character works on many levels - you can sense the lonely child he was and the taciturn old codger he will become, the tragedy of taking the wrong path, the slow crush of predestined failure - but the level that got to me is the way he's locked inside himself and barely able to connect with the world.
Whether this is a universal condition or a personality trait, I don't know. But for some of us who feel that way, being close to people can be a lot more difficult than being alone. Closeness just heightens our sense of never being close enough, and our sense of yearning for something unattainable.
The irony of Ennis Del Mar is that while he suffers from an inability to be close, he achieves a closeness that most people are never able to achieve.
The last time a movie got to me this strongly was Naked Lunch. I still haven't sorted out what there was about the character Bill Lee (played by Peter Weller) that left me with another sort of yearning. That film too made me realize that something is missing from my life - perhaps it is a connection to the subconscious.
(It's just coincidence that the actor who played Ennis Del Mar died last week. I hadn't even known who Heath Ledger was.)