I've got to echo what yaustar wrote. Stay away from doing something you really have no interest in.
As far as a programming goes. It depends on what you want to do. If you want to get into the industry as a graphics programmer, then you need all the math and problem solving skills you can get. But if you love physics and you'd rather work to be a physics programmer, then a minor in physics would be helpful.
The CS curriculum will introduce you to several mathematical concepts already. A minor in math will allow you to do a little more molding with your courses to be more in line with game programming.
But do what you want to do. You can always learn what math you need on your own (which is what I've been doing).