I am able to right click on the calendar and delete it. But it comes back after some time. I have even removed my mail profile and created a new profile. It looks like this setting is somehow stored on the exchange server ...
Microsoft SharePoint Full Article Link - How do I delete linked SharePoint calendar from outlook?