Don't worry about the wording. People say thinigs in e-mails that don't really represent the way they feel. Not everyone words things perfectly.
Last time I built, the builder actually encouraged me to go on-site & check the work so things could be fixed quickly before too much incorrect work was done.
I would have thought that the block of land is yours, and the house is on your block, so they can't stop you from goinf on your own property, but I'd carefully check your contract with the builder.
Relax and enjoy your build.