Forestry expert witness Russell E. Carlson, RCA, BCMA, Tree Tech Consulting, writes on a residential boundary line tree dispute:
In general, boundary line trees are considered to be the property of both adjacent owners, and cannot be harmed without the consent of both. As to the tree on your mother’s property, the neighbor cannot cause direct harm by going on to her property.
The grey area is when the neighbor cuts roots. If he stays entirely on his property, he may have the right to do as he wishes. If he has a valid reason, such as building a pool or an addition to his house, and that causes injury, there may not be recourse. However, if his intent is to destroy the trees, and that can be documented and proven, the situation would probably be different. Again, this is very much dependent on the laws in your jurisdiction, and a lawyer is necessary to help sort it out.
One thing to encourage is mediation or arbitration. Mediation is simply a meeting of the interested parties, with a mediator to help facilitate the discussions. The mediator can be anyone the parties agree to, although there are specially trained mediators available. At least one well qualified arborist should be present to discuss the arboricultural issues. This is the best way to start, if the neighbors can agree to meet face to face to discuss it rationally.
Another optioin is to have an attorney send a letter to the neighbor advising against any action against the trees until some effort has been made to mediate or negotiate the problem. I suggest going light on any threats of legal action, but make it clear that that may be an option. You may need to do this quickly, if there is a chance he will go ahead without more warning.
The most important thing is to make contact soon, letting the neighbor know your opposition to harming the trees and asking for a meeting to discuss it. Next, contact a good lawyer, preferrably with some experience in tree law. There are referral services that can help you find one.