I have an English "A" level Geology, and did the first 2 years of a Geochemistry degree before switching to biochemistry.
There is a very big difference between metal salts, and silicates. Solubility product.
Many metal salts, (ores), leach metal ions into water, not all, but a significant number. Things like Iron Pyrites, (fools gold), Chalcopyrite, Sphalerite, Cassiterite, Blende etc. will release some metal ions, but not that many. Most seriously polluting ore minerals, by the time they are found, have already lost the majority of the readily soluable minerals. The remaining stuff is still best avoided. The classic sign for ore minerals is "stones" that feel overly heavy for their size. A chunk of Galena will weigh a lot more then a similar sized piece of quartz for example. Many, Galena for example, (Lead Sulphide), also have a metallic look about them. Avoid heavy stones and stones that look like metal.
Metals bound into the lattice of silicate minerals are just that, bound. They will not leach out, at least, not in any reasonable time frame. Plagioclase certainly is an unstable mineral, but it decays, (weathers), over millions of years, and its products, (largely clay minerals like Kaolinite), are also harmless. Consider, granite is largely a mixture of quartz, fjeldspar and mica, (which decays in a similar manner to fjeldspar), and is regarded as perfectly sound for aquarium use.
Aluminium is one of the most common metals in the Earths crust, but almost all of it is bound in silicates where it is not economically extractable. The same is true of Iron and many other metals.
Minerals such as Amethyst are totally inert. If you want to liberate the colouring metals, it is necessary to use some really serious chemicals, or physical processes, (electric arc spectroscopy), to establish their presence.
As a footnote to the guy that bought some uncut emeralds, no, you have bought glass. When I was in Colombia in '96, about once an hour, some guy would siddle up and say he had uncut emeralds, bargain price, worth a fortune.....
As a second footnote, labradorite is a key mineral in the syenitic rock "larvakite", (spelling varies), which is used as a facing stone for a lot of buildings. It appears dark blue but at certain angles "flashes", the "schiller flash". At university, it was colloquially known as "Barclays Bankite" as Barclays Bank often used it in their facades.