There is concern around patients' increasing use of online symptom checkers for medical information and health care recommendations. This study used standardized patient cases to examine the accuracy of 23 publicly available services which aim to provide a diagnosis for specific symptoms or give a triage recommendation. The online services listed the correct diagnosis first in about one-third of instances and listed the correct diagnosis in the top 20 possible diagnoses in more than half of cases. Concerningly, symptom checkers provided varying triage recommendations, with appropriate advice ranging from 33% to 78% of evaluations. Certain symptom checkers encouraged users to seek care in cases where self-care was reasonable. These data do not support the use of online symptom checkers for diagnosis or triage and argue for use of simulation approaches to evaluate digital health tools. A related editorial calls for evidence on the actual use of such symptom checkers, followed by randomized trials to evaluate the effect on outcomes and costs prior to their widespread implementation.