In a way I would agree with you. However, our society is already divided up along those lines. For example, do you go down to the slums to hang out? What about the high society part?
I choose to no associate with people who engage in certain behavior that is often times attributed with ones economic status (position, level, class, not sure what would be a good word for it).
So I guess I am ok with someone saying they draw the line at avoiding those that engage in or associate with people who do drugs, have unprotected sex, don't get tested, etc.
Is it right? Probably not. Is it the world we live in? Unfortunately so.