I don’t understand why you think that is a contradiction. You both agree there’s a spectrum between the two. Technically, if you’re not 100% authoritarian you have a greater-than-zero alignment with libertarianism.
Now, if you’re trying to say landing somewhere in the middle of the spectrum means you’re neither, then I tend to agree with you (labels suck). However, I’d take it a step further and say that nobody is going to be the 100% perfect embodiment of either end of the spectrum, and therefore, no true authoritarian or libertarian exists. I think, to say either one of you is wrong is just arguing semantics.
I’d personally rather align more with libertarianism than authoritarianism. But hey, different strokes for different folks.