What is the problem here?

So today had an interesting ending. My Grandfather and I got into a fight because I am going to a church event in Milwaukee. He says " I am getting involved in the politics of the church." (wish I knew what that meant) and that all I do is work and go to church which is not even close to the truth. What is the problem with going to church? I asked him what is wrong with church I could be out drinking and doing worse things. He just said that he is not going to interfere and again laid another guilt trip in my lap. I just want to move out but have no where to go. Is it so bad that i want to attend church? everyone please pray for me.