Who's the liar? Who is the deciever? Who can I trust?

I would say, just relax. Enjoy your time with this man until he gives you a reason not to. You don't really have anything to lose because if you believe Matthew, then what? You will lose this guy and never know what could have been. Plus it seems to me that Matthew said this because he was angry that you chose his brother over him.