04 Dec Securing artificial intelligence
Managing the risks associated with machine learning (ML) and artificial intelligence (AI) is a multi-dimensional challenge, says Dan Norman, and as organisations become more dependent on them, securing them will become business critical.
Over the last few years, artificial intelligence (AI) has received a tremendous amount of attention from the media, businesses and consumers alike. Its supposed potential to transform our lives and to provide innovative solutions to a range of challenges would have previously been the work of science fiction but is now becoming a reality for many. The acceleration and development of operational machine learning systems are helping organisations to digest, leverage and commercialise data in unparalleled ways, such as developing new drugs, analysing medical imagery in seconds or scheduling predictive maintenance on heavy machines. However, security practitioners are facing a significant challenge, as more organisations are starting to rely on these machine learning systems that they either have not secured or do not understand the security implications for.
Attention is required
Operational ML systems have become established in many organisations to improve efficiency, seek out innovation, and capitalise on the huge stores of data that modern enterprises generate and collect. The data is used to build a digital model of one aspect or process of an organisation. The model, and the algorithm driving it can provide a greater understanding of that aspect of the business, allowing the organisation to make better decisions about how and when to act. It is broadly accepted that ML models and the strategy they influence will demand constant attention because they can produce wildly different outcomes depending on very small changes – the so-called ‘butterfly effect’. They also need attention – not just from information security – from other stakeholders, especially those units or functions that will respond to the model’s advice. Stakeholders include information security, IT, operations, legal, data protection, data analytics and HR. In a large organisation that wants to make the most of its investment in ML systems, this roster of stakeholders and the oversight activity demanded will be considerable.
Understanding the lifecycle
A typical machine learning lifecycle covers gathering organisational requirements, ingesting data, modelling development/ testing, deployment and monitoring. Information security concerns are present at every stage of this lifecycle, and practitioners have an important role to play in handling the risks the organisation takes on if it commits strongly to being guided by the conclusions of the ML system.
Gathering organisational requirements
Every operational ML system is different, but it is important in every case that the organisation, and therefore the information security function, knows how and where the systems are in use, how many elements of the ML lifecycle are present, and how well they do their job. Key challenges at this stage may include a lack of skills and technical infrastructure to safely run and maintain operational machine learning systems; having governance structure in place to manage them; and local and international external regulations being fractured.
Machine learning models largely rely on accurate data, and lots of it, to function and generate a faithful model of the business process, activity, or phenomenon that they are being trained to deal with. Key challenges at this stage may include ingesting data without regard to its origin or its accuracy; not testing data for bias, leaking or exposing sensitive information; and risk of subtle disinformation campaigns to poison data sources.
Modelling development/testing
If data is the fuel that powers operational ML, the model it develops is the engine of the application. This step in the lifecycle typically requires many iterations as the model is slowly trained on high-quality data to return responses that align with the organisation’s aims for that ML system. Many ML projects fail at this stage because they are constructed haphazardly, deployed indiscriminately, and promoted deceptively. Key challenges at this stage include development pipelines; failure to adhere to secure code practices; and machine learning models providing little information about how decisions are made. Moreover, as political stances on machine learning evolve and change, sudden regulatory shifts may impact organisations.
Deployment
Once the model that defines the operational ML system has undergone sufficient testing, the moment comes for it to be deployed and for the organisation to adapt to its decisions and outputs – once those outputs are trusted and accepted. Information security should be one of the stakeholders involved in the conversation because many of the changes brought in will have an impact on the roster of policies and processes that serve the current incarnation of the organisation.
Key challenges at this stage include outages in infrastructure supporting the model that cause it to fail; attackers targeting the model to seek to perform espionage or sabotage; copyright claims being made against the organisation for using data sources without permission or an appropriate license; and customers complaining about outcomes and decisions being biased or unfair.
Monitoring
The dynamic nature of operational ML systems sets them apart from most other technologies that organisations deploy and use regularly. They need regular and constant curation throughout their design, development and deployment, and beyond that as they start to influence the way an organisation works. This oversight and curation process becomes increasingly important as ML systems spread inside individual organisations and permeates the sectors in which they operate. The monitoring and assurance activities are both the end and the beginning of the lifecycle. Key challenges at this stage include mismanagement during incidents and outages; over-automation of processes and loss of institutional knowledge; and potential regulatory demands for transparency and accountability.
Facing challenges head-on
Operational ML systems challenge organisations, and information security practitioners, in many ways. The initial challenge is just coping with the speed with which they are developing, spotting who is using them, defining a basis of governance for their use, monitoring potentially useful innovations and keeping an eye on how attackers are trying to adopt and abuse them. While many of the risks that emerge when an organisation adopts ML can be partially overcome by applying the knowledge and experience of practitioners, many others, and some of the most urgent and far-reaching, demand broader collaboration. Information security functions may start this debate but the ultimate resolution of these risks is likely to be far beyond the responsibilities of the security leader and their team.