“If you say you believe in the gender binary, then you’re agreeing with conservatives.”
The fact that two groups with wildly different political ideologies can sometimes align on a single broad issue isn’t a particularly groundbreaking insight, and using guilt-by-association tactics to smear feminists as right-wingers is a dishonest and obnoxious tactic. Any serious activist will find themselves in agreement with “the other side” at some point; communists and libertarians tend to agree on the importance of gun ownership, liberal feminists and men’s rights activists tend to agree on the value of legalizing prostitution, neo-Nazis and anti-fascists tend to oppose Israeli settlements, and so on. But this isn’t evidence of some deep ideological similarity between any of these two groups.
But even more importantly, those who talk about “believing in the gender binary” simply misunderstand the difference between descriptive and prescriptive claims. The nature of a particular gender system is an objective reality, structured by a particular society’s social relations. It’s not something you “believe in” or “don’t believe in” – rather, it’s something you acknowledge or don’t acknowledge. When a radical feminist or other gender-critical person asserts that gender is based on sex, they aren’t expressing how they wish gender was, or how they think gender ought to be. They’re simply describing the way they believe gender does work, whether we like it or not. In that context, criticizing gender-critical people for “believing in the gender binary” is like criticizing anti-racists for “believing in white supremacy.” It’s a simple confusion between recognizing the way a social structure works and endorsing the way a social structure works.