As someone who has designed and implemented automated descision systems for the financial sector:<p>Many of these systems are rule based, not black box NN. The reasons are many. The first descision such a sytem makes is wether the case is suitable for automated descision. If not, the case manager wants all the data and the exact reason why the case needs manual attention.<p>Rules regarding descisions change constantly, mostly in line with (aggregate) risk positions and the financial business climate. Specifics are tuned on types of cases, exposure to certain asset classes in the portfolio etc., not just on the basis of a global parameter.<p>Risk rules and models vary significantly between different regions and countries. So does the availability of data. This wouls significantly fragment the training data.<p>Rules can deal easily with niche cases incorporating specific explicit world knowledge. E.g. dentists require significant credit at the start of their carreer for building out their practice and the assets are harder to market in case of loan default. Would the (fragmented) training dataset have enough nuance to cover all such niche rules? What if they change?<p>My personal feeling is some of the recent regulation should be more focussed on systems and outcomes, and try less to tie that to specific technology.<p>Suppose I take a room of 50 low paid employees. I project a past finance case onto the screen, and they get to vote yes or no. The result is then showm so they know what the right answer eas. They earn a bonus in line with how many they get right. Once sufficiently trained, they move to production and real case descisions are based on a simple majority of their votes.<p>No AI system to be found here. Just human 'experts' making joint descisions, right? The AI act does not apply.