There are 3 things I find scary in games:
1) Ambient noises, bumps, and voices. I'm actually a fan of the Silent Hill movie and one of the most chilling portions in my opinion is hearing the scream of the janitor echo down the hall as Rose runs away. Those background bleeps and bloops can really set a tone.
2) Sudden, non-expected surprises. It doesn't have to be super fast or scream super loud music as long as it catches you at the right time. I would suppose an immersive environment has to do with part of this. Again calling on Silent Hil, in Homecoming I literally jumped out of my seat at the beginning when a wheelchair fell down some steps. I couldn't tell what it was. A wheelchair scared me...
3) The unknown is always scary. Notice a lot of monsters and creatures are very 'pieced together' in most games. It triggers something unconsciously so you see the thing but can't put your finger on what's wrong exactly...
A tricky thing that might work or might end up horrible, is tweaking gameplay so enemies disappear or move fast. Not knowing where an enemy can be terrifying (although it can also be really aggrivating if done inappropriately).