Nice concept here. I think I understand where you want to go with this whole scrolling idea, but maybe 2 things to consider:
1) The default view is too 'zoomed in'. I think you can tweak it abit such that you can see more, but the camera scroll mechanic is still needed
2) I think it only makes sense for the camera to track your main character when you choose to focus on him.