When John and Olivia rush over to the Baldwin Sisters' house to find Erin, John clearly indicates that they had not met Ashely and it was high time they did... and they are introduced to him formerly by Erin when they find them....but Ashley had spent time at the house in the "The Legacy" when he and Erin first fell in love.
It's very out of character for Ashley, a naval officer of his breeding, to run away from the dinner table without explanation when asked to say grace as a guest. He would have simply been forthright and said aloud he preferred not to due to his beliefs.