Self-evaluation of animal welfare by farmers has recently been encouraged, e.g. by the Austrian organic farming association BIO AUSTRIA. However, rather little is known how a resource-efficient training of a large number of farmers could take place and which level of agreement might be achieved. For this purpose, in the present study, an online training-tool for the assessment of 10 animal-based parameters of dairy cattle welfare was established. Th is tool included online quizzes containing pictures or video clips of selected animal-based parameters which had to be assessed by the test persons. IOR as compared to a gold standard (calculated as Cohen’s Kappa κ) was investigated. Furthermore, it was of interest whether practice in terms of repeated trials leads to improvement. In total, 938 κ values from 111 users were obtained from the 10 different quizzes. The average agreement per quiz in round 1 reached values of κ ≥ 0.40 (n = 58–100 users). For the parameters cleanliness and diarrhoea, κ exceeded 0.40 for all test persons in round 1. Agreement was lowest for the parameters body condition, hairless patches and lameness. Retaking the quizzes (round 2, n = 14–24) led to significant improvement of agreement for all parameters, except for hairless patches and lameness. In conclusion, the results of this study are promising as regards the intended use of the training-tool. However, its potential to improve reliability of live on-farm assessments needs to be further investigated, e.g. with regard to transferability to live observations.