Table 1 Selected competencies for General and AI specialist-clinicians

Health Informatics Competencies

- Knowledge of data stewardship and patient privacy concerns.

- Ability to work with electronic medical records in recording and accessing patient information.

- Ability to engage with telemedicine systems.

- Awareness of the limitations of health data systems with respect to completeness and representativeness of data.

- Ability to adapt to and work with novel informatics interfaces and computer systems.

- Ability to use basic clinical decision support systems.

- Detailed understanding of clinical workflows, with the ability to appropriately design models to support given clinical use cases.

- Understanding of the social, economic, and political context of AI at the level of health research, health system structure, and health technology regulation.

- Organizational management skills to guide informatics implementation projects and workflows.

- Proficiency in data management skills, such as data cleaning and quality assurance

- Understanding of and ability to align work with common data interoperability standards.

Artificial Intelligence Competencies

- Understanding the applicability of a given AI technology in specific clinical contexts.

- Interpreting and explaining the output of a given AI model (or, in the case of unexplainable models, evaluating the empirical validation process of a given model).

- Knowledge of the limitations of a given AI model, with a particular emphasis on fairness and bias as well as differential model performance.

- Understanding the importance of human oversight and the limitations and failure cases of AI systems.

- Ability to recognize and mitigate AI failures as they arise in a clinical workflow.

- Detailed knowledge of model architecture, with assessments of the appropriateness of a specific technology for a given clinical task.

- Evaluating the performance and robustness of an AI model for a specific clinical problem.

- Evidence-based evaluation of AI-based tools, including trial design, implementation, and continuous monitoring.

- Generating and curating datasets for the purpose of

- Identifying limitations and biases in performance of algorithms towards marginalized groups and fine-tuning model performances by curating more representative datasets.

- Ability to interpret and communicate model performance metrics to non-technical stakeholders.

- Awareness of the legal and regulatory landscape for AI in healthcare, including liability concerns and approval processes.

Generative AI / LLM Competencies

- Baseline awareness of the inputs and architectures of LLMs.

- Skills in offering both initial and follow-up queries to LLM systems as appropriate in a given clinical context.

- Skills in prompt engineering, with awareness of the context-specificity and stochasticity of LLM outputs.

- Understanding of the “hallucination” phenomenon, and ability to be appropriately skeptical and verify LLM outputs where necessary.

- Integrating information from a diverse range of sources (including LLM summaries or differential diagnostic predictions alongside traditional info such as patient demographics, clinical presentation, and investigation results) in the context of a patient-centered clinical encounter.

- Collaboration with colleagues from other medical specialties to identify opportunity and limitations for further development of LLMs within clinical contexts.

- Ability to generate and incorporate human feedback and clinician / patient preference information in fine-tuning models.

- Detailed generation and evaluation of prompts, alongside sensitivity analysis for the impact of subtle prompt fluctuations on outputs

- Understanding of cutting-edge technical developments in generative AI (such as retrieval-augmented generation (RAG) or long context window techniques).

- Ability to design and implement human evaluation studies to assess clinical impacts of LLM-generated content.

- Understanding of the ethical and legal implications of using generative AI, including with respect to intellectual property, liability, and privacy.