Avoiding the Fate of Icarus

14 min readOct 1, 2019

by Gabriel Straub and Bill Thompson

A bronze figurine of Daedalus. Who landed safely. — Daedalus — who landed safely. Photo by Petre Stojkovski — http://media.a1.com.mk/media/golemi/otkritie-dedal.jpg, FAL, https://commons.wikimedia.org/w/index.php?curid=8602817

Having argued that data scientists must think carefully about their work and its impact, here we outline some steps that can be taken. We believe we can ameliorate the risks posed by the application of data science by being open about our higher level goals, taking accountability for the outcomes of our work, and being transparent about the choices and tradeoffs we make.

We will look at this in the context of the BBC, as our current employer and an example of an organisation that is obliged to open about its internal deliberations in many areas.

1. Be explicit about what fair means to you

Every organisation has a set of higher level goals that drive it, beyond the need to have enough income and structure to remain in existence. In fact the goals of the BBC, which was created as a private company in 1922 and then turned into a public service corporation in 1927 through a Royal Charter, are explicitly stated in a document written on vellum and signed by Queen Elizabeth II — other organisations may lack the clarity that this process offers.

The BBC is a values-driven organisation, charged with putting broadcast technology at the service of the people. It emerged from a belief that broadcasting technology was too powerful to be left to commercial interests and should be managed in the interest of all, and during the last near-century it has established effective independence from political influence too. It seeks to broadcast ‘All that is best in every department of human knowledge, endeavour and achievement…. The preservation of a high moral tone is obviously of paramount importance’ and has developed rigorous editorial guidelines: “The BBC strives to be fair to all — fair to those our output is about, fair to contributors, and fair to our audiences. BBC content should be based on respect, openness and straight dealing.”[1]

Crucially, the BBC’s values and editorial guidelines apply just as much to the technologies it uses or builds as they do to the programmes and services themselves. Just as the Today news programme on BBC Radio Four must be impartial, and the BBC Bitesize education website must consider the needs of the young people who use it, so the machine learning tools they develop or deploy must take account of their our core principles and values, especially fairness.

In the ML context, fairness for the BBC also extends to three other key areas:

Independence: it will be increasingly vital that people can find information and recommendations they can trust and that they can be sure are free from commercial or political agenda.
Impartiality: the BBC should promote services that are built to minimise the bias (implicit or explicit) that can arise from training machine learning on data that reflects existing prejudice or has been developed by designers that fail to reflect the diversity of society.
Universality: the world must avoid an AI future that is limited only to the wealthy or well educated few, or even one in which AI services are limited to a small number of companies who have exclusive access to the data.

Of course the BBC also has a duty to the market, and to treat both suppliers and other players fairly and in a non-discriminatory way. This is reflected in the Market Impact Assessments that are carried out on BBC activities by the UK communications regulator Ofcom[2].

In the context of machine learning systems, the implications of the universality of the BBC’s approach to fairness go very deep. With customers, it has to consider how reasonable it is to differentiate outcomes based on customer attributes. While a company focused purely on profit may segment audiences to maximise income, without considering whether doing so might disadvantage certain groups, for example by denying them access to certain services or special offers, the BBC cannot do this.

This means that a decision on whether to include the gender of a user as part of a cold start recommendation would be made in consultation with editorial colleagues and not simply on the basis of anticipated benefit to the BBC.

This also affects supplier relationships. Since the BBC’s aim is to represent all audiences it has targets that shape what output and services are commissioned, and targets for the diversity both of its workforce and of those who appear in programmes or are reflected in online offers. Tools and services that are brought into the BBC must reflect these goals and help deliver them, not undermine them.

The need to avoid ‘filter bubbles’ and echo chambers is also important, to the degree that the BBC is not just optimising its recommendation engines for accuracy but also taking into account other dimensions such as novelty, diversity, coverage and serendipity[3].

The explicit commitment to fairness directly affects all aspects of the BBC’s work in machine learning, to the degree that it is working on developing a specific set of metrics and processes that build on the editorial guidelines to guide its approach to machine learning. There is also work on techniques that can shape outcomes towards fair ones — for example having one algorithm try to optimise for a business outcome (e.g. predicting a salary) while a second algorithm tries to predict user attributes (e.g. gender or race) based on the predictions of the first. The overall system then optimises for an algorithm that creates the best business outcome that does not allow the second system a good prediction of user attributes[4].

How to do this in your organisation

First, be clear what user attributes matter for you from a fairness perspective — define these and write them down. What would be unacceptable outcomes (e.g. difference in acceptance of job applications between males and females; difference in success in mortgage applications between whites and blacks)?

In doing this, make sure that your thinking covers both customers and suppliers. For example a former employee of Etsy told me that when his team made changes to recommendation algorithms they would check what impact that had on the distribution of items displayed by different dimensions such as size of seller.

As you start to develop or enhance your internal algorithms, try to determine how you will make decisions about whether or not they are fair. What metrics can you use? How can you account for other factors? Use this model to develop an explicit approach to fair ML that covers your whole organisation. what processes can you put in place to ensure algorithms in production are being fair?

Finally, it’s important to keep track of the decisions that are being made algorithmically, for example you can use http://deon.drivendata.org/ to create checklists that get added to each repository.

For reference, the UK government has developed a data ethics workbook that teams are expected to work through when doing a data project (

https://www.gov.uk/government/publications/data-ethics-workbook/data-ethics-workbook)

2. Take accountability for the algorithms inside your organisation

The second important area to consider is your own accountability, and a willingness to accept that there will always be trade-offs and interests to be balanced. One option is to make a public statement to this effect, as the BBC did in 2017 when it submitted written evidence to the UK House of Lords Select Committee on Artificial intelligence[5] promising responsible technical development and future investment to develop a public service approach to algorithms.

This implies conforming to both the spirit and letter of relevant laws and regulations. It is apparent that many organisations have asked their designers to build systems that, while not contravening the recent EU General Data Protection Regulations (GDPR) do as much as they can to undermine the principles of informed consent, and we will see the same happening when it comes to algorithmic transparency and explainability as legislatures around the world start to deal with the deeper implications of the widespread adoption of machine learning.

In order to be properly accountable enterprises deploying machine learning need to give customers the ability to tell them when algorithms do not work well, so that when developing recommenders customers have the opportunity to report that the recommendations were not good enough. This is already done by Amazon, which tells you what purchases are used in its algorithms and allow you to remove these purchases from consideration and indicate which recommendations you are not interested in.

However Amazon does not provide you with explicit information or control over individual results. In this, as in many other cases, companies are not really giving people control over what / how data is collected and used about them, but rather the illusion of control. While the majority of Americans agree that it is very important for them who can get access to their data, many struggle to understand the nature and scope of data collected about them[6].

Similarly, though Facebook tells you what kind of advertising they think you might be interested in and offer the ability to correct this information if you disagree, they do not give users a sufficiently full picture of their data collection practices to allow truly informed use of their services.

We see this in the large number of instances where people assume that Facebook listens to all of your conversations[7]. That they do not listen to your conversations but gather a whole lot of other data that users are probably not aware of only makes the situation a little bit better[8].

If not understanding data collection policies makes it impossible for people to assert real agency in their engagement with online services, then a lack of understanding of how machine learning systems operate on that data can only make it worse and increase the barriers between services and their users, removing the possibility of meaningful informed consent to the data processing that fuel algorithmic services. One way to counter this, in addition to better education, is internal accountability — for the organisation itself to take its responsibilities to users seriously. Doing this creates internal pressures to be fair and reasonable that can mitigate the desire to maximise profit, engagement or ad exposure.

At the most basic level, giving a user the ability to turn off any personalisation can be an attractive option and it is one that the BBC offers: audiences can turn off all personalisation and data collection without having to go into a complex settings menu. However while this may work for media organisations and recommendation systems there are many contexts within which users have no choice about engaging with a machine learning system — the face detection systems at national borders, or diagnostic systems used in healthcare are just two examples. In these places an acknowledgement of a duty of care and a willingness to be held accountable can go a long way towards reassuring users that their interests are being represented as the systems are developed and deployed.

How to do this inside your organisation

When it comes to turn this into action inside your organisation there are three important principles to take into account.

The first, which may seem obvious but sadly is not so, is to make sure you understand the algorithms enough to control them. This means not relying on black box promises but having people in your organisation who understand statistics and machine learning well enough to challenge claims and investigate them. At the extreme, you could consider keeping some of your users or audiences outside the scope of your ML systems so that you can observe unintended long term effects, but this may not be feasible.

Second, where you are using algorithms to make decisions on behalf of customers, make sure that they can provide proper and meaningful feedback. Give them the ability to easily tell you if you have inferred something wrong and to investigate the data and processes behind that inference. Then make sure that this information feeds back into your algorithm both in terms of assessing its quality and also for retraining.

Finally, offer your customers honest choices about how algorithms shape their experience of your service. Give them an easy and obvious opt out of algorithmic experiences where this is feasible. You may want to make it clear to them that this might make their experience less relevant and smooth, but give them a choice. and then make sure that choice is honoured across the whole portfolio of your products — don’t ask customers about this every time they interact with you.

3. Be transparent and open around the choices that are made.

As well as being open about what fairness means to the organisation, and accepting that they are accountable for your deployment of machine learning, practitioners can avoid flying to close to the sun by being transparent about the choices that they are making and encouraging wider debate and better public understanding of the issues raised.

The BBC is fortunate to have a platform that already reaches 92% of the UK adult population, and a news operation that is one of the most respected in the world, and it can use this influence to report on machine learning in a way that is accessible to the wider population[9]. It is also developing courses on machine learning not just for data scientists and engineers[10], but also for journalists and decision makers. These courses aim at equipping everyone at the BBC with a realistic view of the opportunities and threats of machine learning.

The work on enhancing understanding stretches to events including an ongoing very well-received series of ML fireside chats[11] and AI conferences[12] that bring together public service organisations, commercial institutions and academia around the biggest issues in this space, with the proceedings of these events available to the public.

However wider public awareness of what ML is and how it works is only one aspect of being transparent, serving to ensure that people are able to understand what organisations choose to say about their practices and choices. It must be accompanied by a genuine willingness on the part of those organisations actively involved in deploying machine learning to be examined, questioned, and challenged.

How to do this in your organisation

This will only really be possible if organisations speak in a language that can be understood.

Doing this is difficult, because many of the issues are very complex, but it is achievable. We have seen good examples when it comes to data collection policies — in 2012 the UK public broadcaster Channel 4 released its Viewer Promise that explained in simple English what data was collected and why. It was also a great opportunity to showcase their brand personality. (Unfortunately as part of their GDPR review they have now replaced this with a much less accessible privacy policy[13]). We need to develop similar practices around algorithmic transparency to reflect the new ways in which data is being exploited, and the new forms of data that can be of use to organisations as they deploy algorithms to help them make decisions in all operational areas.

One option that should be considered is a sandbox that allows users to explore the algorithms in use with dummy or test data. The sandbox could be part of the settings page and allow user to change their key information (say age or gender) and then display a new set of advertising or recommendations based on that information. While this approach is probably not feasible for behavioural data (since few people would probably be willing to pretend to consume content they are not interested in) it should work well for demographic and similar data.

For behavioural data users could be given the opportunity to see the world through the eyes of someone else. Either a specific friend or individual (if that person is comfortable about sharing their data) or an ‘average’ person of a certain location or demographic. This would allow users to put themselves into the algorithmic shoes of someone else. it would also allow external researchers to better validate claims of algorithmic bias.

Lack of agency is a serious issue. While it might mean something as simple as not being able to find the product you are looking for at a price you know your neighbours are getting, on the more serious side, it can mean ML restricting your choices on who to date, where to work, how long to stay in prison and these are fundamental parts of how we define our self worth. Understanding how the algorithm works is one way to address this, not least because it creates the potential for people to call for change.

And we should not neglect the needs of technical stakeholders. Here we should provide a fair opportunity to assess algorithms. This does not necessarily mean exposing the full source code, as providing the assessment framework, exploratory statistics on the training data and publishing the assessment results would give external parties the opportunity of investigating algorithmic fairness without getting access to sensitive information. Obviously this would need to be done in such a way as to avoid third parties being able to reconstruct key attributes of the data or the algorithms (e.g. through differential privacy), but the risk of this should be balanced against the imperative towards openness.

Delivering this inside your organisation will take effort and commitment. First, you must take an active part in the debate and make sure that your colleagues and customers understand where machine learning is helping your organisation. The more specific and open you can be, the more useful the debate becomes.

Making this happen will not be straightforward: while few organisations would not subscribe to statements such as “we use machine learning in a responsible way”, they will find it hard to be specific about what responsible means or willing to discuss situations where they have not been responsible. However, as with company value statements, the best debates focus on trade-offs and support decision making.

Finally, try to ensure that you use language that is understood by the different stakeholders. Don’t hide behind technical or legal terms but also don’t assume that the subject is too complex to be understood by your audience.

Authors

Gabriel is the Head of Data Science and Architecture at the BBC where his role is to help make the organisation more data informed and to make it easier for product teams to build data and machine learning powered products.

He is an Honorary Senior Research Associate at UCL where his research interests focus on the application of data science on the retail and media industries. He also advises start-ups and VCs on data and machine learning strategies.

He was previously the Data Director at notonthehighstreet.com and Head of Data Science at Tesco. His teams have worked on a diverse range of problems from search engines, recommendation engines, pricing optimisation, to vehicle routing problems and store space optimisation.

Gabriel has an MA (Mathematics) from Cambridge and an MBA from London Business School.

Bill is a well-known technology journalist and advisor to arts and cultural organisations on matters related to digital technologies. He is a Principal Engineer in BBC Research & Development working on ways the BBC can deliver its public service mission online.

Bill has been working in, on and around the Internet since 1984, and was Internet Ambassador for PIPEX, the UK’s first commercial ISP, and Head of New Media at Guardian Newspapers where he built the paper’s first website.

He appears regularly on Digital Planet on BBC World Service radio and writes for a range of publications. Formerly a Visiting Professor at the Royal College of Art, he is an Adjunct Professor at Southampton University and a member of the Web Sciences Institute advisory board

He is a former member of the boards of Writers’ Centre Norwich, Britten Sinfonia, and the Cambridge Film Trust. In 2016 he was awarded an Honorary Doctorate of Arts by Anglia Ruskin University. He manages the website Working for an MP (w4mp.org).

Bill has an MA (Natural Sciences) and the Diploma in Computer Science from Cambridge.