Was i wrong for what i did? please answer?

You did great. You should have called them sooner. For him to do that in front of the kids and not give a crap is really going to far. Also for him to be that crazed for nothing, means he's probably doing something behind your back. I'd just let him keep walking your kids deserve better.