Abstract: Concerning the mathematical model for evaluating reliability of digital watermarking system,a formula of error probability for a correlation-based watermarking detector is derived under a full frame DCT（discrete cosine transform） domain image watermarking schemes.Theoretical analysis indicates that error probability of detector is independent of image itself,but only relies on the length of embedding watermarks and embedding strength factor.Analysis of this paper reveals the factors that may influence the reliability of detection,this can help us optimize watermarking system.Corresponding experimental results are illustrated.