No, it doesn't suck. If you show them love through their life, then saying "I love you" won't do shit. It's not the words, it's the sentiment. I would think that it "sucks" to be concerned with words you didn't say rather than the feelings you DID give.
And if they died on bad terms with you, I'm sure they'll understand in the afterlife (if you believe in that) that you did show/feel love towards them. If you don't believe in an afterlife (like me), then they're unconscious, unaware, and unaffected by what happened in life, in which case it's not really something to worry too much on since YOU are the only one who carries the "burden" and can resolve it through understanding.