I understand that if that is what you think the phrase means, you should be offended. But I don't think most people feel that it means there is "racial inequality where there isn't."
To me, and Urban Dictionary.com, playing the _____ card simply means to "call someone racist, even if they aren't, just to get away with something."
An example is in the movie "This is 40" where the dad has been mooching of his sons family and the wife calls him out on it. The dad says that her real problem is that she hates Jews, even though it is obviously not the issue.
View: https://www.youtube.com/watch?v=Z1QoJof7IAs
Anyway, don't want to derail the thread. But since I was the one who brought the phrase in here, I'd say something.
Carry on, no offense meant.