There are several methods for clustering time course gene expression data. But, these methods have limitations such as the lack of consideration of correlation over time and suffering of high computational. In this paper, by introducing the non-parametric and semi parametric mixed effects model, this correlation over time is considered and by using penalized splines, computation burden dramatically reduced. At the end, using a simulation study the performance of the presented method is compared with previous methods and by using BIC criteria, the most appropriate model is selected. Also the proposed approach is illustrated in a real time course gene expression data set.