Performance of mutual fund selection methods is typically assessed using long samples (long time series). We investigate how well the methods perform in shorter samples. We carry out an extensive simulation study based on empirically motivated skill distributions. For both short and long samples, we present evidence of large differences in performance between popular fund selection methods. In an empirical analysis, we show that the differences documented by the simulations are empirically relevant.