A magnetic p'up, being an electromagnetic trasducer, produces a "tone footprint" as a result of the interaction of the elements that create the magnetic circuit.
This varies depending on the type of p'up (single coil, humbucking) and the alloys that make every component.
The one subject to the most evident audible change is the humbucking p'up, due to its inherent complex structure, depending on many factors, being one of them called Permeability. This is the capacity of the materials of stabily conduct magnetism through'em, deflecting it to a certain element in a certain way.
When in a HB you change the magnet, the result is a different magnetic circuit is created, which it'll take a certain amount of time to create a new, stable field, taking the longest time the weakest magnet type.
IME, it goes from a couple of days flat for a strong mag like an A8, and a week to ten days for an A3 mag to "settle".
So, yes. It does exist such a thing, and it's very real.
HTH,