Supporting the City of Surrey to move 20 million documents from Laserfiche to Microsoft 365

Image credit: surrey.ca

About the City of Surrey

The City of Surrey is a municipality in British Columbia, Canada. It is the largest city by land area and second most populated in Metro Vancouver.

The City employs over 4,689 staff as of 2023.

Technologies

SharePoint Online

Power Platform

Azure Function

Azure Cloud Services

ShareGate for migration

Foxit PDF Maestro Server for combining and converting files and OCR

The business challenge

The City of Surrey is one of the fastest growing municipalities in BC.

Over the years, digital records for Engineering & Planning (Projects), Planning and Development (Buildings), and Finance (Homeowner Grants) grew to about 6 terabytes of information and millions of documents.

These files were stored in a Laserfiche repository, and only accessible by designated city staff. The goal of this project was to move these files into Microsoft 365 for a modern user experience.

In addition to the growth in records volume, these files were not easily searchable, except by limited metadata. A key goal of moving these files into Microsoft 365 is to create a robust search experience that supports searching within the documents themselves to aid in finding the right content.

Solution highlights

Migration planning

The electronic records the City wanted to migrate were stored as individual TIFF files in a Laserfiche repository.

The City engaged Gravity Union to help migrate the following groups of documents:

  1. Homeowner grant information

  2. Planning and engineering building records including inspections, drawings, permits, applications, and legal documents

  3. Engineering projects records including correspondence and design files

Gravity Union first completed an initial assessment to evaluate the content, its use, and the volumes that need to be migrated. Gravity Union collaborated with the City on a vision for the migration and aligned on detailed requirements and high-level architecture for the new solution.

High-level architecture and process flow for the new solution

Design planning

Information architecture and metadata mapping was built into early prototypes to decide how to apply metadata to document sets in SharePoint.

The solution architecture required careful planning to accommodate the large existing volume of content along with continuing growth in the City.

Prototypes helped stakeholders visualize the solution and agree on how activities such as content holds and the process for uploading of new files would work.

A complex migration project

To make the files searchable (including the contents of the file), the migration process included clean up and translation steps. These steps convert the individual TIFF files in Laserfiche into combined and searchable PDFs.

Moving large volumes of data from on-premises to the cloud has complexities. As part of the migration, Gravity Union moved files to Azure storage to run functions that process the files and used the application, Foxit Maestro Server, to convert the TIFF files to PDFs with OCR. The effort required included steps to resolve issues with data integrity and quality so that the files are setup for use in Microsoft 365.

Example of processing steps run from code

Processed files in a SharePoint document library

Search as the front door

Gravity Union used a modern PnP (Patterns and Practices) search solution to enable a better search experience for the millions of documents.

The search landing page is the front-end experience for users. It is based on configuring the modern PnP search webparts to search by key metadata such as project number, portfolio ID as well as building address for building-related content.

The search results show the relevant matches to a query and a preview of the PDF document for quick verification, printing, downloading, or emailing.

Search result with preview of PDF document

Search filters

“The new solution significantly enhances property information access, boosting staff efficiency in application reviews and expediting building permit issuance within our community. The intuitive modern user interface facilitated a seamless transition for our staff, requiring minimal training.”

— Manager of Business Transformation, Planning and Development, City of Surrey

Content holds using a Power Automate solution

The City did not have access to full eDiscovery in M365 yet, so a simple process to identify content for holds was implemented for the short-term.

Gravity Union enabled a simple content holds process with Power Automate for building and project documents. A field is checked in the document library when a set of files is subject to a hold, and this initiates a set of processes run by Power Automate including updating the security settings.

When a user views documents under a content hold, they see a message informing them of the hold and directions of who to contact.

Behind the scenes, security settings are updated so that only the file registry team has access to edit the file and metadata. They can also take the necessary steps to process the hold. A SharePoint list is updated automatically to display the files subject to a hold.

SharePoint List displaying files that are subject to a legal hold

Benefits and Results

  • Process efficiency and fewer errors — Reduced the number of manual steps in processing documents using Power Automate and Azure Functions

  • Productivity improvement with searchable documents — By moving from TIFF files to PDFs and enabling OCR, documents are searchable, and it is easier for City staff to find information they are looking for

  • Reliability in content — Content and metadata is cleaned up resulting in a more authoritative source of content  

“It has been an amazing experience working with Gravity Union. They pushed us to think differently about how we work and what could be possible. Their responsiveness, leadership, technical expertise, and unwavering commitment were instrumental in successfully executing this intricate migration.”

— Program/Project Manager, City of Surrey

Norfolk County

We worked with an Ontario municipality to create a multi-year Records Management strategy and roadmap that considers investments in governance, scanning and digitization.

College of Surgeons and Physicians of Manitoba (CPSM)

See how we helped the College move towards paperless collaboration on SharePoint Online and Teams.

City of West Kelowna

The City of West Kelowna moves to Microsoft 365 as part of a broader paperless initiative.