N Brown has suffered a fall in full-year profits amid increased investment in its infrastructure and one-off charges relating to financial services customer redress.