When David Merlin (David Niven) and Polly (Ginger Rogers) leave the New Years Eve party, the amount of confetti on his jacket changes dramatically from when he talks to the lady by the door and when he exits the building.
Since the film was released in August 1939, the New Year's Eve celebration must be 31 December 1938. A theater marquee is seen, showing Irene Dunne and Charles Boyer in Love Affair (1939), which was not released until April 1939 so could not have been shown at a theater the preceding New Year's Eve.
A woman trying to leave a baby at an orphanage is not proof that she is the baby's mother. At the very least, she would have to undergo a doctor's examination to determine whether she had given birth recently and to ensure that her and the baby's blood types were consistent (e.g., if Polly were type AB and the baby was type O, then it would be proven that they are not mother and daughter.
J. B. Merlin tells his chauffeur, "Follow my son and keep out of sight." Then Peter Merlin is shown walking down the lightly traveled street with a chauffeured, stately car slowly following him about 10 yards behind. Additionally, just to ensure Peter wouldn't be able to recognize him, J. B. covers the lower half of his face with his hand.