Trying is good, but no reason to force it or feel silly for not falling for those "secondary" ones. Or even the fifth.
If I were packing some Beethoven for a desert island, I think I could grab Fidelio, the sixth and ninth symphonies, the piano concertos, the string quartets ... and call it a day without regret at not hearing the rest again. Okay, maybe a few sonatas. But those others are the cream of the crop for me.