If they are well “locked into” the surface they are unlikely to affect durability, although their appearance may be considered an issue. However, particles can be dislodged by Materials Handling Equipment (MHE) or other actions, leaving small surface voids. These voids can be drilled out and and filled with resin mortar.
Where soft particles, such as naturally occurring mudstone, chalk or lignite, are exposed in the surface, they can be removed by drilling and replaced with mortar as described above. The durability of the floor is unlikely to be affected.