Data problems are very likely to occur in the planning and development stage of algorithms

1.4.2 The data challenge As has been illustrated above, the quality, accuracy, validity and reliability of algorithms depend on the quality of the input provided.109 Several problems may arise in this respect.

First, an otherwise correctly designed algorithmic system may be fed with incorrect information. To give a simple example: if a sensor dysfunction wrongly leads to detecting the violation of a traffic rule,110 a rule-based algorithm will not be able to identify the mistake itself and will simply generate a decision to impose a fine based on the assumption that a traffic offence has been committed. However, humans would consider that outcome to be unfair, because the car driver is being fined for an offence that she has not committed.111 Another inaccuracy that may occur is that the data used to train an algorithm is unrepresentative of the general population, inadequately deals with outliers or does not include particular minority groups.112 This type of inaccuracy may easily lead to discrimination, since this may result in an algorithm automatically reflecting the imbalanced data on which it has been trained. The classic example is that of the facial recognition software that performs less well on Black women’s faces than on White women’s and Black men’s faces because Black women are under-represented in the dataset used to train the algorithm.113 Finally, and very relevant from a non-discrimination perspective, if a self-learning algorithm is fed with unbalanced or biased data it is very likely to generate equally unbalanced and biased output based on its detection of correlations and patterns in that data.114 As Barocas and Selbst have explained, ‘approached without care, data mining can reproduce existing patterns of discrimination, inherit the prejudice of prior decision makers, or simply reflect the widespread biases that persist in society’.115 Often this is summarised as ‘rubbish in, rubbish out’ or ‘bias in, bias out’.116 As was noted in the AI Now report from 2016, ‘there is the risk that AI systems trained on this data will produce models that replicate and magnify those biases. In such cases, AI systems would exacerbate the discriminatory dynamics that create social inequality, and would likely do so in ways that would be less obvious than human prejudice and implicit bias’.117 This is exacerbated if a self-learning algorithm is using the output it has generated based on flawed data to further ‘improve’ itself. In that case a feedback loop can be created that reinforces already existing patterns of structural discrimination by ‘reifying’ and further enacting discriminatory correlations.118 Because of the human factor, discussed in section 1.4.1 above, such data problems are very likely to occur in the planning and development stage, for example when selecting the data that is to be used or when preparing or labelling the data. Clearly it is imperative to be aware of the quality, accuracy and reliability of the data used for labelling, training, feedback and learning. In this report, this is described as the data challenge.