ChecklistSocial Sector

AI and beneficiary data: a safeguards checklist for nonprofits

Before an AI tool reads case notes, eligibility records or feedback from the people your organization serves, work through four groups of checks: purpose and lawful basis, the impact assessment, vendor and model terms, and how decisions stay with people. Beneficiary data is often special category data about people who cannot easily say no, so the bar sits higher than for donor or staff data. The checklist below is written for that context.

Reviewed 10 October 2026 8 min read

On this page

Why beneficiary records need a stricter test than donor records
Purpose, lawful basis and impact assessment checks
Data handling and vendor terms to confirm in writing
Fairness, human decisions and accountability to communities
What to do when a safeguard check fails
Safeguards that slip when a deadline looms
A hypothetical refugee support charity screens a feedback tool
Questions and answers
Sources

Why beneficiary records need a stricter test than donor records

Beneficiary records describe people at their most exposed: health conditions, immigration status, ethnicity, religion, experiences of violence and sometimes biometrics captured at registration. Under the General Data Protection Regulation (EU) 2016/679, data about health, racial or ethnic origin, religious beliefs and biometrics used to identify a person is special category data, which may only be processed under one of the conditions in Article 9¹.

The relationship matters as much as the data. Someone registering for food aid, shelter or legal help is rarely in a position to refuse a form. The UK Information Commissioner's Office says freely given consent is harder to obtain where there is an imbalance of power², and the ICRC's Handbook on Data Protection in Humanitarian Action discusses why humanitarian organizations often rely on other legal bases, such as vital interest or public interest, rather than consent⁵.

AI adds three things to that picture: new copies of the data (prompts, embeddings, logs, fine-tuning sets), new parties (model providers and their subprocessors) and new ways to be wrong at scale. ColdAI designs beneficiary systems with privacy and security practices suited to vulnerable populations⁷, and this is the list we would work through first. It assumes the GDPR or UK GDPR applies; organizations elsewhere should map each item to their own law, but the questions rarely change.

Purpose, lawful basis and impact assessment checks

Settle these before anyone books a vendor demo.

0 of 7 checked

Write down the problem and who benefitsName the task the AI would change, the staff who would use it and how beneficiaries gain. If the main gain is a faster funder report, say so; it changes the proportionality test.
Test whether a non-AI option would doA better intake form, a rules-based filter or more staff time may meet the need with less exposure. Record why each was rejected.
Map every data field to the purposeList the fields the tool would read, flag special category data and any data about children, who merit specific protection under the GDPR¹.
Choose and document the lawful basisRecord the Article 6 basis and, for special category data, the Article 9 condition. Where people depend on the service, use consent only if refusing truly costs them nothing.
Run a DPIA before procurementArticle 35 requires a data protection impact assessment where processing is likely to result in high risk, including large-scale processing of special category data¹. The ICO's guidance treats vulnerable individuals and innovative technology as signs of high risk³.
Put exclusion risk inside the DPIAAsk who could lose out: people without phones, speakers of minority languages, people whose records are thin because they avoided services. Standard privacy templates rarely prompt for this.
Consult the DPO and, where possible, the communityTake the draft assessment to your data protection officer and to a community feedback panel while the design can still change.

Data handling and vendor terms to confirm in writing

Most exposure comes from what happens to records after they leave your own systems.

0 of 7 checked

Minimize before you sendStrip names, case numbers and identifying free text where the task allows. Pseudonymized data is still personal data under the GDPR¹, so this lowers risk without removing obligations.
No training on your data, in the contractThe agreement, not a settings page, should state that prompts, files and outputs are not used to train or improve the provider's models.
Data location and transfersConfirm where data is stored and processed, including backups and support access, and the transfer mechanism for any country outside your legal regime.
Named subprocessors and notice of changesGet the full subprocessor list and a right to object to additions. Article 28 sets the minimum contents of a processor contract¹.
Retention and deletion you can verifySet retention periods for prompts, logs and outputs, and ask how deletion is confirmed in caches, backups and vector indexes.
Offline and low-connectivity behaviorField apps that queue records on devices need encryption at rest, remote wipe and a limit on how long unsynced records may stay on a phone.
A rule on public AI toolsTell staff plainly whether case notes may go into consumer chatbots and give them an approved alternative; our guide to shadow AI governance covers the policy side.

Fairness, human decisions and accountability to communities

0 of 6 checked

Test performance across groupsEvaluate outputs separately by language, gender, age, disability and location where you hold that data lawfully. An overall accuracy figure hides the groups a tool fails.
Find out who is missing from the dataReference data drawn from past caseloads reflects who reached you before, not who needs help now.
Keep eligibility and assistance decisions with peopleEU GDPR Article 22 gives people the right not to be subject to solely automated decisions with legal or similarly significant effects¹. In the UK, section 80 of the Data (Use and Access) Act 2025 replaced Article 22 with Articles 22A to 22D, which restrict solely automated significant decisions based on special category data and require safeguards such as human intervention and a right to contest⁴.
Publish an appeal routeTell people how to ask for a human review, in their language, through a channel that does not need a smartphone.
Plan for breaches and access demandsAgree who decides when there is a breach or an authority asks for beneficiary records. The third edition of the ICRC handbook has chapters on cloud services and government access and on artificial intelligence⁶.
Tell communities how AI is usedExplain what the tool does, what it never decides and how to raise a concern, then feed complaints back into the assessment.

What to do when a safeguard check fails

If
The only lawful basis you can find is consent, and people rely on the service.
Then
Pause, or redesign the workflow so that declining has no effect on assistance.
Consent given under pressure is unlikely to be valid, and the loss of trust outlasts the project.
If
The vendor will not commit in the contract to no training on your data.
Then
Choose another provider or a deployment you control, such as a model hosted in your own cloud tenancy.
Policy pages can change without notice; signed terms cannot.
If
Group testing shows one community gets worse results.
Then
Limit the tool to tasks where staff check every output for that group, or stop until the gap closes.
A tool that works for the majority can still deepen exclusion for the people most in need.
If
The DPIA finds high residual risk you cannot reduce.
Then
Consult the supervisory authority before processing begins, as Article 36 requires¹.
Going ahead without that consultation is itself a compliance failure.

Safeguards that slip when a deadline looms

The DPIA is written after the pilot

Early signalThe assessment describes a system already in use and recommends no changes.

MitigationMake an approved assessment a gate in procurement and in grant budgets.

Free text leaks identities

Early signalA pseudonymized export still contains names and addresses inside case notes.

MitigationRun automated redaction and have a person spot-check samples before any transfer.

Feedback analysis drifts into profiling

Early signalSentiment scores begin appearing beside individual case files.

MitigationKeep feedback analytics aggregated and separate from case management.

Funder reports draw on personal stories

Early signalCase narratives reach reports without fresh, specific permission.

MitigationKeep a story-consent register and check it at every report; see grant reporting automation.

A hypothetical refugee support charity screens a feedback tool

Questions and answers

Can staff paste case notes into free AI tools?

Generally not. Consumer AI tools sit outside your processor agreements, may retain prompts and may use them to improve models unless a contract says otherwise. Case notes usually contain special category data about identifiable people. Give staff an approved tool covered by a processor contract with no-training terms, and write a short rule with examples of what may and may not be entered.

Do we need consent to analyze beneficiary feedback with AI?

Not necessarily, and consent is hard to rely on when people depend on you. Feedback analysis can often run under legitimate interests or, for public bodies and their delivery partners, a public task basis, supported by a clear notice. Remove identifiers before analysis where possible, keep results aggregated and make sure the impact assessment covers the AI provider.

Do the same data protection rules cover donor data?

The same law applies, but the risk is different. Donor data is rarely special category data and donors do not depend on you, so consent and legitimate interests are workable bases. Electronic marketing rules, such as PECR in the UK, also govern donor email and phone outreach. Beneficiary data needs the stricter controls on this page.

What if our organization works outside the EU and UK?

Map each item to the laws where you operate and where your data is processed. Many countries now have data protection laws built on similar ideas: purpose limits, sensitive data categories, impact assessments and processor contracts. Where local law is weaker, humanitarian and do-no-harm standards such as the ICRC handbook are a sensible floor.

Sources

Regulation (EU) 2016/679 (General Data Protection Regulation) — EUR-Lex · checked 10 October 2026
What is valid consent? — Information Commissioner's Office · checked 10 October 2026
Data protection impact assessments (DPIAs) — Information Commissioner's Office · checked 10 October 2026
Data (Use and Access) Act 2025, section 80: Automated decision-making — legislation.gov.uk · checked 10 October 2026
Handbook on Data Protection in Humanitarian Action, third edition — International Committee of the Red Cross · checked 10 October 2026
Handbook on Data Protection in Humanitarian Action, third edition (open access, table of contents) — Cambridge University Press · checked 10 October 2026
Social sector: use cases, delivery process and approach — ColdAI

More in Social Sector

Back to Social Sector

Next step

Get a second view on your AI data protection impact assessment

Send the draft DPIA, the vendor terms and a description of the beneficiary data involved. We will mark the gaps against this checklist and suggest design changes that lower the risk.

Share your DPIA draft

Why beneficiary records need a stricter test than donor records

Purpose, lawful basis and impact assessment checks

Data handling and vendor terms to confirm in writing

Fairness, human decisions and accountability to communities

What to do when a safeguard check fails

Safeguards that slip when a deadline looms

The DPIA is written after the pilot

Free text leaks identities

Feedback analysis drifts into profiling

Funder reports draw on personal stories

A hypothetical refugee support charity screens a feedback tool

Questions and answers

Sources

Related reading

More in Social Sector

Get a second view on your AI data protection impact assessment