It does depend to an extent on the wording of your contract. Mine was different, but ehhc's for example allows for the "building period" to start when the builder actually starts on site, so the "building period" could therefore start earlier than having to wait the 20 day.
In this case, the builder could claim that they were ready to start on the 23rd but were delayed until the 31st (or whenever it was). Either way, if it is all inside the magical "20 days", then it doesn't really make a difference if they actually planned to start early or not - as long as the delay is valid (ie, it was actually too wet to start) then your 180 days starts from the 31st August.
Just out of curiosity, what was the date at which the 20 days started? That is, when did you have all of your stuff to the builder, and they had the approvals in place?