Mental fatigue is a psychophysical condition with a significant adverse effect on daily life, compromising both physical and mental wellness. We are experiencing challenges in this fast-changing environment, and mental fatigue problems are becoming more prominent. This demands an urgent need to explore an effective and accurate automated system for timely mental fatigue detection. Therefore, we present a systematic review of brain-computer interface (BCI) studies for mental fatigue detection using artificial intelligent (AI) techniques published in Scopus, IEEE Explore, PubMed and Web of Science (WOS) between 2011 and 2022. The Boolean search expression that comprised (((ELECTROENCEPHALOGRAM) AND (BCI)) AND (FATIGUE CLASSIFICATION)) AND (BRAIN-COMPUTER INTERFACE) has been used to select the articles. Through the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) methodology, we selected 39 out of 562 articles. Our review identified the research gap in employing BCI for mental fatigue intervention through automated neurofeedback. We summarized the AI techniques employed to develop EEG-based mental fatigue detection are discussed. We have presented comprehensive challenges and future recommendations from the gaps identified in discussions. The future direction includes data fusion, hybrid classification models, availability of public datasets, uncertainty, explainability, and hardware implementation strategies.