Synchronising Active Directory User Attributes into SharePoint Online User Profile Properties: Part 2

I have received quite a bit of interest in the Active Directory / Office 365 User Profile synchronisation implementation that has been mentioned previously in this blog, as well as in some local SharePoint community presentations, so I thought it would be a good time to write it up in a bit more detail.

Goal:
Customise the Office 365 User Profile Syncronisation Service to synchronise additional attributes (e.g. Mobile phone number) from Active Directory into Office 365.

Constraint:
SharePoint Online does not allow us to change mappings for the built-in User Profile Syncronisation service.

Solution:
Build a custom automated connector that synchronises Active Directory and SharePoint Online User Profiles.

Approach:
Out of the various APIs offered by SharePoint, the only method that we found we could update User Profile information programmatically was by using the UserProfileService.asmx web service. Note that our investigations were conducted against a SharePoint 2010 based tenancy; new 2013 based tenancies offer REST web services which will probably be able to achieve the same goal.

The primary difficulty we found with using the web service was that many operations (including any updates to a SharePoint Online User Profile) would only succeed if the call to the web service was made whilst authenticated as the user who owns the profile being queried or updated! Not even when authenticated as a global tenant administrator and site collection administrator can you programmatically update someone else's User Profile, which is a bit strange given that SharePoint allows you to do exactly this through the tenancy administration interface.

The way we got around this authentication problem was by performing the User Profile update operation from within JavaScript code which is embedded into the Master Page of the My Site Host site collection. This custom JavaScript code runs every time a user loads any page within the site collection, and because it runs in the user's browser, the Client OM operations within the code run in the context of the current user, who is authorised to update their own User Profile. But how does the JavaScript code know what values to write into the User Profile?

This is where things get complicated! We set up a temporary data holding area (just a large custom SharePoint list) within the My Site Host. We built a custom on-premises Windows Service to query AD and push user data into the custom list on a periodic schedule. The data is then already sitting inside the My Site Host site collection and can be queried from our custom JavaScript code using the Client OM.

The following diagram helps to illustrate the process (click to expand):

Data flow of the custom AD / Office 365 connector

To follow this diagram, let's start with just the red arrows. On the left hand side of the brick wall (the corporate firewall) we have the on-premises elements of the solution: Active Directory (AD) and our Custom Windows Service. The Custom Windows Service initiates the transaction and uses the .NET Client OM to write data into the Custom List up in Office 365. This is as far as the data can travel whilst our automated processing is authenticating into Office using a shared service account.

From here, everything happens in the cloud; let's look at the yellow lines. Whenever any user loads a SharePoint Content Page in the My Site Host site collection, the browser executes some custom JavaScript code in the context of the current user. This code first uses the JavaScript Client OM to query the data from the Custom List that was populated earlier in the process. The JavaScript code then calls the SharePoint Web Service (UserProfleService.asmx) to push the data into the SharePoint User Profile.

The process of moving data from Office 365 User Profiles into Active Directory is basically the same, but  in reverse. For those User Profile Properties that are marked has being able to be read by Everyone, the Custom Windows Service can read the data directly via the UserProfileService.asmx web service and we can short circuit a lot of the process.

Summary:
Using the above approach, we built an automated process that is completely configurable regarding property mappings and synchronisation direction, that works for various data types, across both 2010 and 2013 versions of SharePoint Online. It's admittedly an awkward approach, but it appears to be the only way to achieve the desired outcome with the current limitations in SharePoint Online.

As a final note, this is a great example of the type of solution that could be achieved using the new SharePoint 2013 Provider Hosted Application model - provided of course that 2010 tenancy support is not required.