I use guitar polish on my fretboards and clean them every time I change strings. Regular maintenance prevents build-up. When I work on someone else's guitar that has build-up, I use a soft toothbrush and some guitar polish to remove it.
It really depends on where you live and the environment. Growing up on the Atlantic ocean we have a lot of humidity and tons of salt in the air. My house was only 2 miles from the beach. Salt degrades and dries out wood very quickly. I have always kept up on keeping my fretboards clean and treated.
Linseed oil (boiled or raw) is not a good idea, unless you want to "varnish" the surface of your board. But if you want to clean/oil/preserve/protect your fretboard you should use another product.