This paper presents a generalized architecture for the synthesis of application specific memory architectures for real-time image processing systems. The memory generation presented in this paper can handle both spatial and spatio-temporal memory models. The results show that the architecture efficiently solves the problems related to memory accesses for most of the available video image processing filters available at present.