In this paper, a novel five-dimensional hyperchaotic system is proposed, which can generate multi-scroll chaotic attractors. The dynamics of the chaotic system are studied numerically and theoretically, such as equilibrium point, dissipativity and Lyapunov exponent. To get the encrypted image, diffusion matrix and replacement based on the chaotic system,matrix are designed, which are used for substituting and diffusing the images.Security analyses including key sensitivity analysis, key space analysis, statistical analysis, information entropy analysis and anti-cutting analysis are performed i...