I had been having similar probs to you, i also have a servo (with dual line) system, now that they work (sorted today) they are great and compared to non servo brakes are far superior imo,
i dont know of any inherent probs with the servo's but i think you can get a service kit for them, if you think it might be at fault. just make sure you are getting a good vacum to it, the pipe can degrade and end up collapsing, you need to get a proper vacum pipe though if it does need replacing and i ensured i had a good seal on both ends with a union bolt.