Living in a place as homogeneous as Japan brings out some interesting psychoses in people, and one could fill a book studying what goes on in the minds of foreigners living here. I do believe every gaijin secretly wishes he were the only non-Japanese person in the country, and there's a strange tension that arises when foreigners encounter each other for the first time, as each tries to mentally categorize the other. Many foreigners living in Japan are actively hostile towards the "gaijin talents" who are able to appear on TV only (they say) because they happen to have learned the language well. This is nothing more than jealousy, although I do truly hate that guy Dave Specter, who dyes his black hair blonde to look more like a proper American as he dishes out Hollywood gossip on the morning "wide show." Then there are the "Three Stages of Eye Aversion" that gaijin seem to go through when encountering other foreigners. First, you can't keep from stealing glances at the strange foreign guy riding the train -- his very presence disturbs the glorious harmony in the train car, although for some reason yours doesn't. Then you realize how silly you're being, so you so hello to him, but then you find out that not everyone from Sri Lanka speaks English. Thoroughly embarrassed now, you vow to ever meet eyes with or talk to another foreigner again.
Some gaijin avoid looking at other foreigners.