I guarantee you that even though I'm in my 50's, I have smaller hands/fingers than just about anyone (even female) It's bad.
But a good setup and decent technique can make up for that. What I struggle with is "what is acceptable fret buzz" and it varies. Billy Sheehan is very very low action but he's a good luthier in his own right. If you watch youtube vids, there is more fret buzz than I personally like.
The point is that "one does not simply pick up a bass and it's perfect"

Keep doing searches with buzz words like action/setup/etc and just check it out.