It seems like a cultural difference and not something that should be taken as an offense. It's an introverted behavior, and aren't Japanese people, and Asian cultures at large, generally considered more introverted than their western counterparts? U.S. people, especially, might be the most extroverted in the world, and thus the least capable of understanding this.