HI - oh dear you poor thing. I so feel for you. Doesnt help you i know
Your neighbour is a fool - you have photographic evidence that the fence was in a bad state before you started the process (I am sure you have many more photos that in some way show the state of the fence when you lived in the old house) Might pay (check legalities first) to send a copy of one or two photos to the neighbour with a brief letter explaining.
Why do fences cause so much issues!!