With snow covering your driveway, you can't see what your shovel is hitting ... which could cause incidental property damage.
You definitely should clear the snow from your pea gravel driveway. But it's risky to go down to the drive, sending rocks ...