It all depends on what type of game programming you want to do. If you want to do engine programming then you should probably know all the inner workings of how a scene gets created. There's higher level programming in games like a gameplay or ui programmer. They wouldn't need to learn so much low level stuff, but it's good to know as you can take certain limitations into consideration.
I use cosines and sines alot in the games I write. I haven't had a case where I had to bust out a Calculus book yet.. You should have a good grasp of alegra, geometry, trigonometry, vectors, and matrices. There's probably more if you want to do graphics programming.
From my experience in 3D games, vertex/pixel shaders are definitely used. You should familiarize yourself with them, but I wouldn't go overboard and write every shader out there. The shaders in games are usually grabbed from a reference book somewhere and tweaked to the designer's liking.
You shouldn't stress out to much about Math. I'm actually not very good at it, but I know enough to get by. In practice, you just look up whatever formula you need and use it. It helps to know the concepts, so you know what to look for in the first place though.