You are actually right! Because neural networks are already robust against noise by design, one could approximate the matrix multiplication operation and work with results that are not numerically correct, but close enough so that the prediction of the neural network is not affected. This is a very active area of reserach, which spans from approximate hardware units to approximate algorithms.