Courteous letter to neighbor... Need advice?

I would tell him that you are going to have your fence replaced and that you are going to have the property line checked before you do so. Because you know that he is using part of your property I would tell him that the utility company already informed you of where his ends and yours starts. Tell him you have plans for that area and you are not trying to inconvenience him. Tell him when you plan on having the work done and leave it at that. Putting in a date he has to remove the garden by sounds nasty. Sign and leave your # in case he has any ?'s.