The explanations and reasons for decisions made by algorithms are firmly demanded in various applications. In case of biomedical applications concerning the health and/or treatment options for patients explainability and comprehensibility of algorithms is especially essential. This calls for systems that produce human-understandable knowledge and decision proposals out of the data such that domain experts can base their decision on these systems. There are different names for such systems: symbolic, knowledge-based, expert systems or recently “explainable AI” systems. Current systems of explainable artificial intelligence systems have either specific assumptions about the data or often do not follow the Grice’s maxims, i.e., the explanations are not meaningful and relevant to domain expert. Using the already researched insights about structures in data and swarm intelligence, an explainable machine learning system (XAI), allowing for user interaction and understandable both upon the Grice’s maxims and from the perspective of domain experts can be developed by our long years of experience in this field.