Abstract: An improved parabola model and a group of restricted projecting conditions of fish-eye lens are proposed. For a scene, only two fish-eye photos taken forward and backward, along with the corresponding drafts given in advance to predict the horizontal and vertical silhouettes of objects, are enough to get the whole structure and 3D model of the scene by using an algorithm based on Gaussian local weight. Based on this model, not only the panorama can be acquired, but also the 3D view at any position and any direction can be got. This approach greatly simplifies the complicated calculation for photos taken by a camera with normal lens.