That within the same exact pickup design, magnetic strength is what causes the tonal differences we hear, is *mostly* true. But there are some caveats. The big one is that the statement is made assuming identical magnetic permeability of the mags. In reality, the differing magnetic permeability of the alnico grades, which affects the shapes of the respective magnetic fields that they establish, affects how each grade magnetically couples with the strings.
A2/3/4 are more magnetically permeable than A5. This means that they want to "hold on to" the magnetic field more tightly; the field hugs the magnet more closely than it does with A5's. This creates slightly different (more "withdrawn") magnetic field shapes that A5, regardless of gauss. This results in increased inductance within the pickup itself, but in the end, reduced output vs. A5. This is because A5, with its larger field, is more strongly coupled to the strings, even though its lower permeability (i.e. less dense, more widespread magnetic field) results in coils with lower inductance. Thus, A2/3/4 will still be [just a bit] in a league of their own compared to A5; they will result in less output, *even if* A5 and, e.g. A2 are gaussed exactly the same.
This is all considering the case in which A2 and A5 are matched in strength. But in the real world, they are not. So you take the slight differences in output caused by the differences in permeability when strength is equal...and add in the fact that in real-world application, their strength is *not* equal, and you have an even larger tonal difference, caused simply by the increased magnetic strength. This leads to even more additional output (much more than the difference in permeability causes), which in turn dramatically affects how an amp responds to an A5 magnet, vs. the lower powered alnico varieties.
So...no, it's not only as simple as magnetic strength...but in the real world, it pretty much is. So much so, that we can say that it basically comes down to magnet strength.