I'm not sure what exactly effects it but I know you have to be crouched (obviously) and the speed u are walking, light, cover and direction enemies are facing and their perception stat all seem to play a part. Sometimes i can walk circles around a enemy and they wont see me but other times it seems like they see me comin from a long way away so it's hit and miss. When u get stealth to a hundred and have perks that boost your sneak criticals and other stuff it really pays off. Stealthboys also help and having that mutant in your party makes them last 10 times longer.