Abstract: Mainstream image dehazing algorithms rely heavily on high-quality pixel-level paired data during training, limiting their practical capabilities. Motivated by the fact that the memory mechanism can learn and store a large amount of valuable prior knowledge during the training process to assist the feature learning, and the haze-related degradation is mainly implied in the amplitude spectrum. This paper proposes an unpaired image dehazing method based on frequency memory-augmentation (FMA-Net), which achieves effective haze removal by augmenting the amplitude spectrum in the frequency domain. First, this paper proposes the memory feature learning module (MLM) used during training to update the memory features by pre-learning the amplitude spectrum features from unpaired haze-free images. Based on this, this paper also proposes a memory augmentation module (MAM), which uses a hard-attention mechanism to represent the relationship between the memories and the amplitude spectrum features of the haze image, and removes the haze by reconstructing a new amplitude spectrum. In addition, the MAM can be equipped with multiple scales to improve the algorithm’s ability. Experimental results show that the proposed FMA-Net outperforms the state-of-the-art algorithms in terms of both quantitative and visual quality on five benchmark test sets containing synthetic, artificial and real haze. Among them, the PSNR on two representative benchmarks, SOTS and HSTS-S, is improved by 0.29dB and 0.19dB, respectively.