Ugh, these are tiring. If it doesn't make a difference then make a fretboard out of rubber. Then will it matter? What will happen if the fret is mounted into rubber? Guess what? You'll hear that difference through the amp, won't you? No sustain, dull highs, they'll all come through loud and clear.
As with everything, there is a difference. The net effect on your life is up to you.
1. Is the difference enough for you to hear and/or experience?
2. If so, do you care?
You can argue all day about whether you care, whether it makes a difference to your ears in your plugged-in tone, but of course it makes a difference. If you can't hear the difference between that fake "Ebonol" material (found in perfectly decent bowling balls) and Rosewood, just consider yourself lucky. Ignorance is bliss. You have the luxury of going through your playing career with less to worry about. Just get some good pickups and you're happy!