Consumer confidence fell in May, with a survey suggesting it could have been hit by the election of a Conservative government.