In four years, AI has gone from being a niche conversation in aid circles to a central pillar of the humanitarian reset. With only 20 percent of humanitarian appeals funded and in the face of unprecedented pressure on human and financial resources, the sector is hoping technology will deliver efficiencies and scale.
However, in the rush to drive efficiencies, we risk racing to adopt AI faster than we can safely govern it. In the world’s most complex contexts, affected communities need humanitarian actors to embed responsible and participatory AI practices throughout the AI lifecycle. Not only because it is their right and our obligation to do so, but because it is more likely to be effective.
Applied to relevant, impact-assessed use cases, AI holds transformative potential. But just as no serious mountaineer attempts Everest without training, proper equipment, and local guides who know the terrain and when to turn back, humanitarian agencies need to incorporate AI literacy across teams, governance structures, and participatory approaches if they are to deploy AI responsibly.
That is why DAI and I have collaborated on this blog. With decades of experience navigating fragile and fast-moving situations, DAI understands that deploying AI safely and effectively requires a firm grasp of how technology, environment, and people interact. Here, we outline the key considerations.
What Are We Talking About?
Artificial intelligence is often spoken about as a single technology, but the term covers a range of distinct systems:
- AI types include generative AI, natural language processing, computer vision, and predictive analytics, among others.
- Applications built on those types include chatbots, translation engines, image classifiers, forecasting dashboards, and decision-support systems.
- A use case brings it to life—an example might be an AI model generating local flood forecasts to trigger pre-emptive cash programming for people at risk of displacement.
AI adoption across aid agencies ranges from “shadow AI” (staff using personal chatbots with no institutional oversight) to major institutional investments. While many humanitarians, like the broader public, largely associate AI with generative AI, attention is turning to other AI applications, from automation of internal processes to optimization tools for logistics and predictive modelling for aid allocation.
Where Risks Enter the System
Understanding AI this way helps identify where risks enter. The technology or system architecture may be subject to failure or adversarial attack. Data raises issues of privacy, ownership, bias, and quality, particularly where models trained predominantly in the Global North are deployed in contexts they were never designed for.
The deployment context matters: a system that performs well in one setting may cause harm in another. And human-AI interaction raises its own questions. A frontline worker entering sensitive beneficiary data may inadvertently create exposure risks; for instance, a system’s misclassification, flawed recommendation, or confidently wrong answer can cause direct harm to individuals with no means to challenge it.
Involve the Right People
The risks noted above are socio-technical, which means mitigating them demands more than technical expertise. Determining how a particular use case might undermine humanitarian principles in a specific context requires engaging multidisciplinary teams spanning technical, operational, and protection functions. It also demands meaningful engagement with affected communities, who must have genuine power to shape or reject an AI approach, not merely be consulted.
Not all AI systems or deployment contexts carry equal risk. Engaging communities and multidisciplinary teams throughout the AI lifecycle makes it easier to assess the cost of error if a system fails, generates inaccurate outputs, or skews decisions affecting people’s access to assistance. The measure of success is not whether an AI system works, but whether it works for the people it is meant to serve.
The Visibility Trap: Backend Systems and Hidden Risk
It is tempting to assume that internal AI systems are inherently safer than systems that communities interact with directly. That is not a given. Less-visible backend systems can avoid scrutiny while still processing sensitive data or shaping life-affecting decisions. Aid allocation algorithms that amplify historical bias, for example, could entrench the exclusion of marginalised groups without raising any internal flags.
Forecasting and prepositioning models illustrate this tendency well. False predictions that misdirect resources could result from biased or incomplete data, poorly chosen proxy indicators, or adversarial attacks (for example, manipulation of satellite imagery or conflict data to poison training datasets). Even well-trained models can suffer model drift when the operational context shifts but the system continues running on outdated patterns. This effect is particularly significant in conflict settings, where dynamics can change rapidly.
Mitigating these risks requires bias audits, use of multiple independent data sources, regular retraining, and staff AI literacy that empowers people to question AI recommendations rather than defer to them. Human oversight of high-stakes decisions informed by a sustained community feedback loop is a minimum, not optional, standard.
Data as Critical Infrastructure: Governance, Security, and Protection
Irrespective of AI, humanitarian agencies have a responsibility to respect individuals’ rights around data, get informed consent for its collection and use, and protect it.
Generative AI systems, however, particularly community-facing ones such as chatbots used to provide crisis-affected people with information, introduce distinct risks. Aside from hallucinations that can produce plausible but incorrect advice, such as inventing support that doesn’t exist, conversational systems might also be vulnerable to manipulation through prompt injection, which the U.S. National Security Institute has labelled “generative AI’s greatest security flaw.” Such attacks involve inserting malicious instructions within user input, tricking the system into ignoring original developer commands to perform unintended actions, such as leaking data, spreading disinformation, or revealing system prompts. This technique exploits the fact that the model processes instructions and user input through the same channel, without a hard enforcement boundary between them.
Data protection risks also arise where conversational systems capture sensitive disclosures, particularly in small or displaced communities where even partial information can enable re-identification. Community-facing conversational systems present the sharpest illustration of these risks, but data governance failures across any AI type—from predictive analytics to computer vision—can compromise protection in humanitarian contexts.
Mitigations require robust content design and controls, secure data transfer and access management, careful data minimisation—ensuring that community-facing AI systems are never trained or fine-tuned on individuals’ raw conversational data—and clear policies to ensure that vendor data retention is scrutinized and contractually governed. Organizations need a systems approach delivered through multidisciplinary teams, including community staff who understand local dynamics, to identify and mitigate technical, operational, and protection risks. A technology fix alone does not make an AI initiative safe.
Reaching the Summit
Understanding these challenges, the U.K. Government has funded the forthcoming SAFE AI Framework (Standards and Assurance Framework for Ethical AI)—led by CDAC Network in partnership with The Alan Turing Institute and Humanitarian AI Advisory—to support responsible and participatory AI in humanitarian contexts.
But responsible actors are getting ahead now by recognising that community participation and accountability to affected people are not boxes to tick; it is the mechanism through which contextualisation, risk identification, and course correction happens.
What makes AI initiatives safer is institutional commitment: AI policies treated as living documents, multidisciplinary ownership of risk, and communities engaged as genuine partners with veto rights throughout the AI lifecycle. Human behaviour changes, and the past is not always a reliable predictor, so AI systems trained on opaque, non-representative datasets cannot substitute for the judgement of people with lived experience.
Successfully integrating AI into humanitarian operations means being clear-eyed about its opportunities and limitations, building safety in by design, and keeping communities central. Accountability to affected populations is not only the right thing to do; it’s also how organisations will ensure AI implementations are effective. Convert these principles into practice, and the mountain will be easier to climb.
