Pastors need to be accountable to the elders in the church as well. I was in a church where the pastor wasn't a godly man. There's a good chance he wasn't even saved. A number of people witnessed a number of bad things and nobody put it all together until after he left. He didn't believe in outreaches so he watched attendance dwindle. When it got low enough he was suddenly "called" to another church. The church folded less than a year after that.