Developing machine learning (ML) models to detect real time adverse events requires careful validation of proposed approaches. This article describes two ML models to detect diagnostic divergence (i.e., the deviation between predicted diagnosis and documented diagnosis, weighted by mortality) of pneumonia in the emergency department (ED). More than 6.5 million ED visits were analyzed by the models and 130 were analyzed by expert physicians for diagnostic divergence. Correlation between human and automatic reviewers was weak to moderate. The authors present potential reasons for this outcome and propose future research to improve ML accuracy.