Do you meet, or are you on track to meet Digipen's admissions requirements
in mathematics? If yes, then you may be worrying needlessly. If no, then you have an additional step to take while following your dream.
Kaiser, I noticed you wrote that you are seriously very bad at math, and that you didn't really pay attention during secondary school. You now seem concerned that you will need a stronger background in mathematics to be competitive in the game programming field and are wondering if you should quit now.
Game programming is your dream. What do you think you should do?