Although the impact of a potential Brexit looms large, it is perhaps a debate many retailers are unwilling to participate in