From the depths of its winter in the 1990s and 2000s, machine learning has become the technology everyone wants to muscle in on. Xilinx is the latest large chipmaker to realign its business to technologies such as deep learning, driven by the thirst among large data-centre users to hoover up as much data as they can for analysis.
In describing how the company’s next generation of devices would support artificial intelligence (AI), Xilinx CEO Victor Peng says the world has seen “an explosion of data”. He adds: “We are only at the very early stages of AI. But it will disrupt multiple industries.”
Some of the disruption may be unintentional. Rich Caruana, senior researcher at Microsoft Research, has a salutory lesson on how machine learning can seemingly do all the right things and still come up with answers that are potentially fatal for the unlucky people who have the rules mysteriously go against them. He has described the problems at a number of seminars where he calls for better understanding of what AI is. As a member of a mid-1990s project to study machine-learning techniques for a pneumonia risk assessor, he stumbled across a problem that would have made the system unexpectedly dangerous.
“Our goal was not to diagnose pneumonia – we already know that. It was to figure out which of you are high-risk or low-risk. The idea is that if you’re low-risk the right treatment really is as an outpatient: antibiotics, chicken soup; call us in three days if you’re not feeling better. That actually is the safest care for you because hospitals are bad places to be if you don’t actually have to be there. High-risk though, you absolutely need to be in the hospital and maybe even in the intensive care unit [ICU]. Ten to 11 per cent of the patients in this data set died from their pneumonia. Pneumonia is a very serious thing if you’re a high-risk patient,” Caruana explains. “Our goal was to train the different machine-learning models that were available to us back in the mid-90s.”
By today’s standards, the neural network on which Caruana worked was relatively simple, but in a study of eight different statistical and machine-learning techniques at the University of Pittsburgh’s Center for Biomedical Informatics, “it turned out to be the most accurate model that any of us could train on this data for predicting pneumonia risk”, he notes.
The model was flawed. Not because the neural network did not work. Rather, it had learned its data a little too well. Caruana asked a friend to look at the model’s performance, and he spotted a correlation the model came up with that seemed to show asthma sufferers are less likely to die from pneumonia than the general population. “He frowned and he’s like: ‘Rich, what do you think this rule means?’.”
Doctors saw the issue with the correlation immediately. Asthma is a serious risk factor for pneumonia. The model had somehow learned the opposite of what it should. “But they said: ‘You know, it’s possibly a real pattern in the data’,” Caruana explains. “If you think about it, people with asthma are paying attention to how they are breathing. They will notice sooner than most that something is not quite right and that their inhaler or other treatment isn’t responding the way they expect. There’s nothing better than noticing the symptoms quickly and getting to care quickly.”
Doctors take into account the pre-existing condition as a serious risk factor and will often admit them to hospital – possibly moving the patient directly to the intensive care unit. The result? A category of high-risk patients get care early in the onset of the disease and so appear in the data as having a lower risk of dying.
“You might think that the reason why we didn’t ship the neural net is because there’s this asthma problem buried in the neural net,” Caruana says. “We didn’t use it because it’s a black box and ultimately could have other problems in it that we don’t know to fix. The concern was what the other unknown unknowns might be.”
Five things you should know about GDPR
1. It has a wide reach. If an organisation is doing business with customers who live in the EU and storing elements of their personal information, those organisations need to follow the GDPR regulations.
2. It makes you forget. Under the old rules, organisations had some discretion as to when and why they deleted personal data. Under the GDPR, EU citizens can demand you delete their data unless it is needed to fulfil a contract.
3. It has a really wide reach. If your organisation passes data on to others for processing and someone insists on their right to be forgotten, you need to find where that data went and insist on its removal.
4. It demands consent. Users have to agree to having their data collected and stored. You cannot store it unless they deliberately opt in.
5. It insists on secure storage. Anything considered personally identifiable information, and that appears to extend to tracking cookies served by a website, needs to be stored securely. And unless it is needed to perform services for that user, be deleted as soon as practical.
Caruana is far from alone in worrying about the problematic things machine-learning algorithms can pick up from their data. In a survey of people in six developed nations around the world, Accenture found only 34 per cent were confident or very confident that goverments would employ AI in an ethical and responsible way. Though they were less worried than the general public, fewer than half of the respondents working in government felt confident about responsible deployment.
Thanks to the explosion of interest in AI, which in turn is driving demand for online courses to train a new generation of software engineers in the technology, we now have numerous examples of machine learning picking up unwanted correlations. And fears are naturally growing about the consequences of hidden bias in AI and other statistically informed algorithms on the population at large.
Kate Crawford, principal researcher at Microsoft and founder of the AI Now Research Institute, explained in her keynote at the NIPS 2017 conference in Long Beach, California: “I think we’re at an extraordinary moment right now. We’re basically at an inflection point where the power and the reach of machine learning is rapidly expanding into many areas of everyday life from healthcare to education to criminal justice.
“Among the very real excitement about what we can do, there are also some really concerning problems arising. Forms of bias, stereotyping and unfair determinations are being found everywhere from machine-vision systems and object recognition to natural-language processing.”
Government bodies themselves are concerned. A recent UK report on AI and its impact by the Information Commissioner’s Office (ICO) argued it would have serious ramifications for data protection: “Implications arise not only from the volume of the data but from the ways in which it is generated, the propensity to find new uses for it, the complexity of the processing and the possibility of unexpected consequences for individuals.”
Already, data-protection law has implications for AI, although legal experts are divided on their practical ramifications. On 25 May, the General Data Protection Regulation (GDPR) comes into force across the European Union. A number of legal > < experts argue the resulting laws will contain a right to demand an explanation of any automated decision that has a significant effect on your life (see box: ‘Questionable rights’). Even if precedent-setting lawsuits do not agree, a second wave of legislation that follows a European Parliament resolution passed last year seems likely to push for organisations to justify the way their AI systems are designed to the public – at least in mainland Europe. The EU committee responsible is expected to publish its initial report in late April 2018.
The problem that faces the AI industry is that, especially with the newer and seemingly more accurate machine-learning technologies, generating a meaningful explanation is beyond it. As she introduced a paper on the explanation ramifications of the GDPR at the February’s FAT 2018 conference on AI fairness, accountability and transparency, Google Brain research scientist Been Kim quipped: “In the rise of complex models like deep learning even the creators of the model don’t really understand why things work well.”
She pointed to situations where in trying to determine how to structure a deep-learning stack, researchers have found simply removing some of the layers leads to a model that works better but with no indication of why it works better.
Deeply hidden secrets
Deep neural networks (DNNs) have demonstrated a striking ability to recognise objects in pictures, but unexpected problems have led researchers to question how they arrive at their results. Their failures can be spectacular, with people categorised as gorillas in one famous case.
The cause of that issue was unbalanced training that did not let the DNN try to learn a range of human faces across multiple races, but even seemingly well-trained networks can go badly wrong. They fixate on features that are almost invisible to the human eye but which let the DNN distinguish between different groups of images. This makes it easier to trick the networks into providing false results – a potential security hole flagged up in a report on malicious use of AI written by the Future of Humanity Institute.
Researchers at Kyushu University in Japan discovered late last year that modification of just one pixel in an image could upset neural networks trained to classify objects and animals. Instead of seeing a car, the network concludes it’s a cat. Two years earlier, a team from the University of Wisconsin created images humans perceive as pure noise that a neural network would label with 99.99 per cent confidence as recognisable objects.
One answer to understanding the mechanics of deep learning may lie in tracing the connections the networks make. Five years ago Matthew Zeiler and Rob Fergus, then both based at New York University, used visualisation to illustrate the way neurons connect to process images. They and others following the same approach use small changes to images shown to the network to work out how they activate different collections of neurons. Groups of neurons seem to zoom in on features such as eyes, disregarding the rest of the image as noise.
The use of saliency maps has backing from brain research. A brain-mapping study (above) at the University of California at Davis has shown how brain cells focus attention selectively on meaningful parts of an image. It mapped hundreds of images (far left) by eye tracking (centre left), ‘meaning’ (centre right) and salience or outstanding features (far right). Statistical analysis shows that eyes are drawn to ‘meaningful’ areas, not necessarily those that are most outstanding.
However, saliency methods have come under question. Working at Google Brain with colleagues from the Technical University of Berlin, Pieter-Jan Kindermans and Sara Hooker developed a way to confuse saliency analysers by injecting a picture of a cartoon cat’s face into multiple images used to train a network on one of the standard number-recognition tests for machine-learning techniques. Several saliency-mapping tools homed in on the cat’s face, not the numbers. But the underlying network still worked as it should, indicating the saliency analysis was not in fact homing in on the features that controlled the DNN’s output.
Deep learning uses huge numbers of inputs to produce a result. The neural networks inside these systems have orders of magnitude more tunable weights than the number of inputs, creating enormously complex structures with performance features that deny explanation under existing theories. Unexpected combinations of data that range from your browsing history to a pattern of answers on an application form could all conspire to paint you as someone not to be trusted when the conclusion has little basis in reality – and it may be extremely difficult to point to the cause of the misclassification.
Although fundamental research into the mechanics of deep learning is likely to lead to training methods that create more robust models, problems will still remain in the selection and use of training data. In Caruana’s example, the core problem lay in the difference between ideal data and available data and partly in the labels for the data. The data issue is a 21st-century version of Plato’s Allegory of the Cave. The AI system only sees shadows of reality in the data it’s fed. The reality is asthma sufferers are treated quickly and so do better because of the quality of care. The shadow says something slightly different. Caruana says, for the model, asthma is “a reasonable proxy for time to care”.
The obvious path is to take the asthma condition out. But this would be even more troublesome. Other variables that might indicate the presence of asthma would still lurk in the data and lead to a similar result. Instead, the data has to go into the model unadulterated and have its effects edited out.Transparent models
The question is how to take opaque models such as those produced by deep learning and make the undesired patterns stand out better or at least let users probe the model to work out why the model came up with an answer. In his work with colleagues at Microsoft Research, Caruana has been researching machine-learning techniques that are much better at revealing issues in the model. Users can probe combinations of variables to see how they affect the outcome and edit out undesired effects. He argues that for applications such as the pneumonia risk-prediction application, these generalised additive models yield results that are as good as far more opaque methods such as random forests and deep neural networks (DNNs).
If the GDPR does turn out to demand explanations that go beyond “the model was trained on relevant parameters and gave you a low score”, organisations may need to look at whether DNNs are appropriate for the task. “You might as well use a transparent model if it works just as well,” Caruana says.
US defence-research agency Darpa is keen to have access to a wide range of AI systems that can explain themselves to data analysts as well as robots that don’t go off the rails. The agency kicked off a five-year project to improve AI explainability in 2016.
Darpa programme manager David Gunning noted at the launch that there are likely to be significant discrepancies between performance and intelligibility across machine-learning techniques. DNNs appear to be one of the most inscrutable, at least today. Researchers have identified a number of avenues for shining a light inside the black box of DNNs, although the methods are revealing problems. But the intensity of current research suggests progress towards a better understanding of AI should emerge in the near future.
Alan Winfield, professor of robot ethics at the University of the West of England, points out: “Although there are some new and interesting aspects of AI they are still human-engineered systems and therefore, in principle, not inscrutable. There’s no reason that we cannot have transparency and explainability in principle.”
Future of Humanity Institute malicious AI report
Questionable rights
An update to the EU’s Data Protection Directive introduced in the mid-1990s, the General Data Protection Regulation (GDPR), sets new limits on what organisations can do with your personal data.
Buried in the text of the GDPR, which comes into force 25 May 2018, is what some legal researchers see as a right for EU citizens to demand an explanation from any organisation for a significant decision made about them carried out by a machine without human intervention. The caveats limit this right. It seems unlikely that you will be able to query Facebook or Google about their ad-serving algorithms. But if you are denied a loan, credit card or access to healthcare and you believe that the decision was entirely automated, you should be able to challenge it and ask why it was made in that way.
The issue is that the text about explanation is in a supplementary section of the regulation. Sandra Wachter and colleagues from the Oxford Internet Institute argued in a paper published last year that the structure of the regulation implies the EU held back from inserting a full right to explanation. But she says other parts of the GDPR could deliver a right to explanation. A clause in the main text calling for notification about the use of an EU citizen’s data should also contain meaningful information about the logic being used in the processing and its possible consequences in a general sense, but not the rationale for individual decisions.
Other legal researchers disagree about the reality of explanation. In a paper written for January’s FAT 2018 conference, Andrew Selbst of the Data & Society Research Institute and Julia Powles of Cornell University argued Wachter and colleagues took too narrow a view. “The information has to be meaningful to the data subject,” says Selbst. “Under human rights law you want to be able to see if you’ve been discriminated against and if you have to challenge it in court, that’s what the explanation is doing for you.”
In fact, firms may see benefits in explainability. Christina Demetriades, deputy general counsel at Accenture, explains: “Consumers will vote with their feet. Companies are going to be at a competitive advantage if they adopt explainable AI.”