I used to file the nut shelf on my guitars, but on my latest guitar I used a belt sander to grind the nut shorter. I held the nut with vise grip pliers and used the rotating sander.
It worked OK, but the underside is not completely level, although it is stable when screwed in.
A Floyd nut is...