Public Sector Data Migration: How to Prove Every Record Moved Before the Old System Is Switched Off

The replacement case management system is signed and the go-live date is set. Fifteen years of records still sit in the old system, in a structure nobody fully understands. A public sector data migration succeeds only when every record is mapped, reconciled three ways and signed off by its owner before the source is switched off.

Key takeaways

• A load that runs without errors proves volume, not accuracy; reconcile counts, control totals and a field-level sample.
• Decide migrate, archive or delete per dataset, using the retention schedule and the section 46 records management code.
• Agree sign-off criteria with the data owners, the Senior Information Risk Owner (SIRO) and the Data Protection Officer (DPO) before the first trial load.
• Keep mapping, cleansing and reconciliation under your control, even if the vendor’s import tool does the load.
• A data migration is finished when someone accountable signs a reconciliation, not when the load completes.

Why does public sector data migration go wrong?

Public sector data migration goes wrong because the old and new systems model the world differently, and many of the old system’s rules live only in staff memory.

The Department for Science, Innovation and Technology (DSIT) State of digital government review, published in January 2025, puts legacy at 28% of central government systems in 2024, up from 26% in 2023. Around 15% of survey respondents could not even estimate the size of their legacy estate.

The root cause is not the team. A legacy schema grows through years of patches. Status codes get reused, free-text fields carry structured data, and the person who knew what code 7 meant has retired. DSIT found only 27% of survey respondents believe their data infrastructure gives a comprehensive view of operations. A migration exposes every one of those gaps at once.

What does a silent migration error put at risk for a public body?

A silent migration error puts the public body’s UK GDPR accuracy duty, its records obligations and its service to residents at risk, usually after the source system has gone.

A poor public sector data migration fails quietly. The load runs clean. Months later, a status that meant “closed” in the source but “on hold” in the target turns up in a subject access request. By then the source is gone, and nobody can prove what the record said before it moved.

The exposure falls in three places:

• Accuracy. UK GDPR requires personal data to be accurate. A mis-mapped field makes thousands of records wrong at once.
• Records. Public authorities must manage records through their life, including disposal. An unrecorded switch-off breaks that chain.
• Service. Caseworkers who find wrong data stop trusting the new system and keep side spreadsheets, which defeats the replacement.

Waiting makes it worse. DSIT found 22% of legacy systems red-rated, a share that is rising, and every retirement takes away someone who remembers what the old codes mean.

How do you migrate public sector data with every record proven?

You migrate public sector data with every record proven by running seven phases, each with a named owner and an outcome, and by reconciling every trial load. Data quality is measured against the Government Data Quality Framework.

The diagram shows the flow, including the loop when reconciliation finds breaks.

1. Profile the source and the target. List every source table, field and code value, measure gaps and duplicates, and document the target’s model. Mark each dataset migrate, archive or delete. Outcome: a data inventory.
2. Map fields and transformation rules. Map each source field to a target field with its conversion rule: code lookups, date formats, split names. Outcome: a mapping specification signed by the data owner.
3. Cleanse and deduplicate. Fix defects at source, merge duplicates using agreed match rules, and leave expired records behind. Outcome: a log of every change.
4. Run trial loads and reconcile. Load a copy of live data into a test target and compare it with the source. Outcome: a reconciliation report for every run, with each break explained.
5. Sign off, or fix and reload. If totals match, data owners sign off. If breaks are found, correct the mapping or source defect, then reload and reconcile again. Outcome: written sign-off.
6. Cut over with a rollback plan. Take a full backup, freeze the source and run the rehearsed cutover with go and no-go points. Outcome: the target live, with a tested route back.
7. Decommission the source. Once the target runs cleanly, archive what must be kept, record where, and switch the source off. Outcome: a decommissioning record.

Data migration is the controlled transfer of data from a source system to a target system, with every record checked. The checking is the part most plans underweight.

How do you reconcile a data migration?

Reconcile three ways, because a record count proves volume, not accuracy:

1. Count records per entity. Residents, cases, addresses and contacts should match source totals after agreed exclusions.
2. Compare control totals. Sum numeric fields and check date ranges, so a shifted date or truncated value shows up.
3. Sample field by field. A caseworker who uses the records every day checks a random sample against the source screen.

The third check catches what the first two miss: data that loads without error but means something different in the target.

How do you choose between big-bang and phased data migration?

Choose big-bang when one source feeds the target and a short outage is acceptable. Choose phased when the service cannot stop or several sources are consolidated. Choose a parallel run when an undetected error causes direct harm to residents.

Criterion	Big-bang	Phased	Parallel run
Outage	One cutover window	Short window per phase	Little or none
Best fit	One source, few interfaces	Many interfaces, or several sources merged in turn	High-harm services where errors must be caught live
Main risk	One cutover must succeed	Data split across systems mid-way	Double running effort, and the two systems drifting apart

Apply these rules to your own estate:

• If you are consolidating systems after a merger of services or councils, migrate one source at a time and deduplicate against the target.
• If the go-live date is close, migrate active records and archive closed ones in a readable format.
• If nobody will own a dataset, do not migrate it until someone does.

The programme board wants the go-live date, the vendor wants its import tool used, and the DPO wants an impact assessment. Agreeing sign-off criteria before the first trial load turns that tension into a checklist.

Which UK GDPR and records duties apply to a public sector data migration?

A public sector data migration must meet the UK GDPR accuracy and storage limitation principles, and records duties when the source is switched off.

Regulation or standard	What it requires	How the migration addresses it
UK GDPR Article 5	Personal data “accurate and, where necessary, kept up to date”, and kept no longer than necessary	Cleansing fixes errors; reconciliation proves nothing changed; expired records stay behind
ICO guidance on DPIAs	An assessment before processing likely to result in high risk	The plan records early whether a DPIA is needed
Section 46 records management code	Public authorities manage records through their life, including disposal	Migrate, archive or delete decided per dataset
Digital continuity guidance	Information stays usable through technology change	Archived source data stays readable and documented

The Information Commissioner’s Office (ICO) guidance on the accuracy principle sets the bar: “You should take all reasonable steps to ensure the personal data you hold is not incorrect or misleading as to any matter of fact.” A signed reconciliation shows those steps were taken.

When is a full data migration not the right choice?

A full data migration is not the right choice when the source can stay as the system of record, or when the data is kept only for retention.

A common objection is that the old system should simply be replaced and everything moved. In public bodies, that carries risk many cannot absorb. Sometimes the better route keeps the source behind an integration layer, which our CRM and ERP integration team builds.

• If only the hosting changes, you need cloud migration and DevOps, not a data migration.
• If the process still runs on paper and spreadsheets, start with digital transformation.
• If there are only a few hundred records, rekeying by hand takes less effort than building load routines.

What should a Head of Data do first?

A Head of Data should first list every system and report that reads the source data, then name an owner for each dataset. That list sets the scope and exposes the ownership gaps that will block sign-off later.

Next, profile the three largest tables. Count nulls, duplicates and distinct code values. The result tells you how much cleansing lies ahead, and gives the SIRO and DPO something concrete to assess.

Frequently Asked Questions

Should we migrate all our historical data?

Migrate the records the new system needs to run the service, not everything the source holds. Mark each dataset migrate, archive or delete during profiling, guided by your retention schedule and the section 46 code. Archived data stays readable outside the new system, and expired records are not carried forward, which also meets the UK GDPR storage limitation principle.

How does a data migration fit with our existing systems and integrations?

A data migration must account for every system that reads or writes the source data. Reports, interfaces and file feeds are listed during profiling, then repointed or rebuilt for the target. The cutover runbook switches them in a planned order, so no feed writes to the old system after the freeze and no partner receives stale data.

What is the biggest risk in a public sector data migration?

The biggest risk is data that loads without error but means something different in the target system. A reused status code or a badly split free-text field passes technical checks yet fails residents. Field-level sampling by caseworkers who use the records every day catches it before sign-off.

Can the new system’s vendor migrate the data for us?

The new system’s vendor can usually load data, but its import tool expects data that already fits the target. The tool rarely profiles the source, interprets old codes or reconciles the result. Use it for the load step, and keep mapping, cleansing and reconciliation under the public body’s control.

Do we need a data protection impact assessment?

You need a data protection impact assessment when processing is likely to result in high risk to individuals, as ICO guidance explains. Large volumes of special category data, such as social care or health records, often meet that test. Your DPO decides, and the migration plan records the decision.

How do we start a public sector data migration?

Start by listing every system and report that reads the source data, then name an owner for each dataset. Then profile the largest tables. Thornacre’s data migration service can run this discovery with you; past projects are on our case studies page.

---

Bought a new system with old records still to move? Ask us for a data migration workshop with your data, IT and information governance leads. You leave with a scoped plan. Email hello@thornacre.com to arrange it.

This article is general guidance only; consult your legal or compliance team for your specific requirements.