Why do I always feel so guilty for talking to them?

For one you are failing to realize that they bring up the past and some things from the past remind you of different things but if you feel that way then just let them be the past and leave them alone. you deserve to go with your life and be happy with this guy if he is the one and you do deserve to feel strongly for him but if they do make you feel this way then the best solution is to let them go