Abstract: In the paper, Gaussian mixture model is used to describe the residuals of matches in the new robust algorithm for fundamental matrix estimation, and an improved split-merge EM (SMEM) algorithm is used to classify the matches, so that the false matches can be detected and rejected by the least mean absolute residual criteria. Finally, M-estimator is used to estimate the fundamental matrix. Our algorithm gives better result than random sample consensus (RANSAC) algorithm with higher efficiency in the large number experiments tested.