some people who are building cant help but rub others up the wrong way about it - EG they bang on about how much their marble tiles from Italy are costing, or their ebony floors, or how their home theater has cost a fortune in soundproofing for their 2000w hummel and klein audio set up.
Sometimes those people deserve a punch in the face.
As to those who are really casual about it, i get it, sometimes people are just jealous, and sometimes are disappointed that their life choices haven't put them where you are. I try to read my audience and adjust the conversation accordingly. Friends thinking about embarking on the journey are curious, poorer friends don't want to hear about it.
if youre getting snide comments or jealousy try and asses what you've just said, or how you've been acting. Its quite often the source of why people respond the way they do (not the other way around). It can be very sobering when you get called out on it.
I think just talking about it can annoy people. When you are building your so caught up in it that it occupies your life and you have nothing else interesting to say. People who aren't building aren't able to engage in your conversation.