This week, the U.S. House of Representatives finally passed the Violence Against Women Act (VAWA), which reauthorizes funding for the fight against domestic violence in the United States. The bill passed after a prolonged partisan fight over specific protections for Native-American women and lesbian, bisexual, and trans women. Also this week, the conservative justices on the U.S. Supreme Court questioned the need for the promotion of equal voting rights, and, in particular, the continued need for oversight of equal rights in states that historically have discriminated against African Americans.

Each story elicited a pundit storm of outrage over the partisan divide on discrimination on the basis of sex, race, or sexual orientation. Many politicians and justices quoted in the press perpetuated the notion that liberals care about discrimination while conservatives do not.

But that would be an oversimplification.

At the heart of the discussion about the need for both VAWA and the Voting Rights Act is a fundamental disagreement about what governments should do about discrimination, and, even more so, what they shouldn’t do.

That difference of opinion is what led Justice Antonin Scalia to refer to the Voting Rights Act—a law that was conceived of as a tool to overcome racial entitlement among whites—as perpetuating a sense of racial entitlement among non-whites. By portraying people of color as receiving, or rather demanding, special treatment, Scalia converted the legal protection of equal rights into a situation of enforced discrimination.

It is also why some Republicans cited “inclusiveness” as the reason they supported explicit benefits for generally underserved populations in VAWA, while other Republicans claimed to vote against these benefits for the very same reason. While the former acknowledge that some women need to be explicitly named in order to be visible to policymakers and service providers, the latter promote the notion that by treating everyone the same, we will somehow magically be equal.

Simply put, the difference is not so much whether someone cares about discrimination but rather if they choose to see the full range of its reach. With some notable exceptions, most people across the political spectrum recognize flagrant forms of racism, sexism, and homophobia. The real divide is on how much we believe can and should be done to overcome historical disadvantage and internalized prejudice. Should the government allow quotas in universities to promote race and gender equality? Should states actively promote a diverse workforce?

International human rights standards are clear that affirmative action can only be legitimate while it serves a purpose; when a situation of historical disadvantage has been overcome—that is, when those who were meant to benefit from affirmative action genuinely are equal—the special measures must go.

The question, of course, is how to determine the exact moment when everyone truly has equal opportunities. This is a question that necessarily will have different answers for different people. Recent studies suggest that discrimination is still a reality for many of the subgroups that benefit the most from both the new incarnation of VAWA and the Voting Rights Act: Blacks, Latinos, working women, and Native Americans. Still, the American public just reelected a Black president, and the minority leader of the House of Representatives is a woman. In other words, systemic inequalities persist even though some people manage to escape their consequences. In this situation, perhaps the best test of whether temporary special measures are still warranted is conversational. When we stop talking about how strange it is that President Obama and Representative Pelosi got to where they are, there will be equal opportunity for all.

Of course, the courts cannot use conversation as a legal test to determine when to mandate an end to temporary special measures. Conversation can, however, be a rule of thumb for the rest of us until such time where it is no longer remarkable to find African Americans or women in positions of power.

In the meantime, the onus should be on the government, including members of Congress and justices of the Supreme Court, to prove when affirmative action has run its course, both when it comes to the prevention of domestic violence and voting rights.