When generating automatic sleep reports with mobile sleep monitoring devices, it is crucial to have a good grasp of the reliability of the result. In this paper, we feed features derived from the output of a sleep scoring algorithm to a 'regression ensemble' to estimate the quality of the automatic sleep scoring. We compare this estimate to the actual quality, calculated using a manual scoring of a concurrent polysomnography recording. We find that it is generally possible to estimate the quality of a sleep scoring, but with some uncertainty ('root mean squared error' between estimated and true Cohen's kappa is 0.078). We expect that this method could be useful in situations with many scored nights from the same subject, where an overall picture of scoring quality is needed, but where uncertainty on single nights is less of an issue.