The German Federal Financial Supervisory Authority (Bundesanstalt für Finanzdienstleistungsaufsicht – BaFin) has issued a guidance paper on supervisory principles for the use of big data and artificial intelligence (BDAI) in decision-making processes by financial institutions.1 The principles are intended to promote the responsible use of big data and artificial intelligence2 and help control the associated risks.
The principles are driven by the need to draw a difficult distinction between BDAI processes and processes driven by conventional statistics given the mere technical definition of artificial intelligence (AI) employed by BaFin, which is in line with the understanding of the Financial Stability Board’s (FSB) definition of AI. From a risk standpoint, BaFin identifies three characteristics of particular relevance to modern BDAI methods:
- High complexity: Algorithms used are frequently much more complex than conventional statistical processes, which renders them opaque.
- Short recalibration cycles: Recalibration cycles are getting shorter due to the combination of algorithms that are constantly learning with the fact that new data becomes available on an almost daily basis. As a result, the boundaries between calibration and validation are increasingly blurred.
- High automation: Use of BDAI methods is leading to an increase in automation that makes it ever easier to scale processes, with the impact of the individual algorithms being amplified. Accordingly, the principles shall apply primarily to those algorithms which exhibit these three distinct features that differentiate them from mere statistical IT applications.
In defining appropriate principles as precisely as possible, BaFin follows a “two-phase approach” dividing the algorithm-based decision-making process into the following two simplified phases: development and application.
The development phase examines how the algorithm is selected, calibrated and validated. In particular, institutions embedding BDAI into their decision-making processes must observe the following:
- Data strategy and data governance: Institutions must have a verifiable process (data strategy) which guarantees the continuous provision of data and defines the data quality and quantity standards to be met and which must be implemented in a data governance system and responsibilities must be clearly defined.
- Compliance with applicable data protection requirements: Data protection requirements for the use of data should already be taken into account when planning algorithmic decision-making processes. In particular, disclosure requirements vis-à-vis data subjects must also be observed.
- Ensuring accurate, robust and reproducible results: By ensuring that results can be reproduced, the results can be understood and verified at least to a certain degree by individuals within the company and by external parties. As regards reproducible results, users should be, for example, able to reproduce results in a subsequent test performed by an independent third party.
- Documentation to ensure clarity for both internal and external parties: Institutions must observe that the selection of the model must be documented (Step 1), the model calibration and training must be documented (Step 2) and the model validation must be described (Step 3). However, in certain circumstances, a clear distinction may not be made for the documentation of the selection, calibration and validation of the model, as in some instances the quality of a model can only be determined after the initial calibration and validation of the model.
- Appropriate validation processes: Every algorithm should go through an appropriate validation process before being included in operations performed or at least to be examined by an independent function or individual that is not involved in the original modelling process. In determining regular, appropriate intervals, according to BaFin it is essential that institutions set out factors that will lead to the ad hoc validation of the algorithm and thus potentially lead to the algorithm being recalibrated or an alternative algorithm being selected (systematic change in input data, external (macroeconomic) shocks, changes to the legal requirements).
- Using relevant data for calibration and validation purposes: The data must be relevant and representative for the application in question. For instance, it must contain information on all relevant sub-groups, as imbalanced data in the calibration or validation process otherwise may lead to modelling bias. Bias must be prevented as soon as data is prepared, eg in the data aggregation phase (key principle of “preventing bias”).
In the application phase, the results of the algorithm must be interpreted and included in decision-making processes. This can either be done automatically or by involving experts. A functioning mechanism comprising elements such as sufficient checks and feedback loops for the development phase must be established in all cases.
- “Putting the human in the loop”: Employees should be sufficiently involved in the interpretation and use of algorithmic results when reaching decisions depending on how mission-critical the decision-making process is and the risks this entails.
- In-depth approval and feedback processes: When using algorithm-based results in decision-making processes, the situations involving a more in-depth approval process should be clearly defined in advance in a risk-oriented manner (eg threshold-based processes).
- Establishing contingency measures: Institutions should set out measures so business operations can continue to run if problems arise in algorithm-based decision-making processes (at least for mission-critical applications).
- Ongoing validation, overall evaluation and appropriate adjustments: Ideally, an internal or external audit should be performed to examine the regular evaluation and adjustment process, ensuring that the functionality and risks of the algorithms in practice are evaluated independently. Risks associated with the use of algorithms can be reduced by involving an additional independent internal or external control function.
Aside from these two phases, BaFin defines “overarching” principles such as the necessity for a clear responsibility structure and adequate risk and outsourcing management.
These principles establish preliminary ideas for minimum supervisory requirements relating to the use of artificial intelligence and form the basis for discussions with various stakeholders. At the same time, these principles already serve as guidance for entities under BaFin’s supervision. However, BaFin explicitly emphasizes that it is essential to note that the principles do not rule out the fact that certain regulated activities may already be subject to stricter regulations or administrative practices (such as data protection requirements). In such cases, compliance with these rules takes precedence.