Synchronising Active Directory User Attributes into SharePoint Online User Profile Properties: Part 2

I have received quite a bit of interest in the Active Directory / Office 365 User Profile synchronisation implementation that has been mentioned previously in this blog, as well as in some local SharePoint community presentations, so I thought it would be a good time to write it up in a bit more detail.

Goal:
Customise the Office 365 User Profile Syncronisation Service to synchronise additional attributes (e.g. Mobile phone number) from Active Directory into Office 365.

Constraint:
SharePoint Online does not allow us to change mappings for the built-in User Profile Syncronisation service.

Solution:
Build a custom automated connector that synchronises Active Directory and SharePoint Online User Profiles.

Approach:
Out of the various APIs offered by SharePoint, the only method that we found we could update User Profile information programmatically was by using the UserProfileService.asmx web service. Note that our investigations were conducted against a SharePoint 2010 based tenancy; new 2013 based tenancies offer REST web services which will probably be able to achieve the same goal.

The primary difficulty we found with using the web service was that many operations (including any updates to a SharePoint Online User Profile) would only succeed if the call to the web service was made whilst authenticated as the user who owns the profile being queried or updated! Not even when authenticated as a global tenant administrator and site collection administrator can you programmatically update someone else's User Profile, which is a bit strange given that SharePoint allows you to do exactly this through the tenancy administration interface.

The way we got around this authentication problem was by performing the User Profile update operation from within JavaScript code which is embedded into the Master Page of the My Site Host site collection. This custom JavaScript code runs every time a user loads any page within the site collection, and because it runs in the user's browser, the Client OM operations within the code run in the context of the current user, who is authorised to update their own User Profile. But how does the JavaScript code know what values to write into the User Profile?

This is where things get complicated! We set up a temporary data holding area (just a large custom SharePoint list) within the My Site Host. We built a custom on-premises Windows Service to query AD and push user data into the custom list on a periodic schedule. The data is then already sitting inside the My Site Host site collection and can be queried from our custom JavaScript code using the Client OM.

The following diagram helps to illustrate the process (click to expand):

Data flow of the custom AD / Office 365 connector

To follow this diagram, let's start with just the red arrows. On the left hand side of the brick wall (the corporate firewall) we have the on-premises elements of the solution: Active Directory (AD) and our Custom Windows Service. The Custom Windows Service initiates the transaction and uses the .NET Client OM to write data into the Custom List up in Office 365. This is as far as the data can travel whilst our automated processing is authenticating into Office using a shared service account.

From here, everything happens in the cloud; let's look at the yellow lines. Whenever any user loads a SharePoint Content Page in the My Site Host site collection, the browser executes some custom JavaScript code in the context of the current user. This code first uses the JavaScript Client OM to query the data from the Custom List that was populated earlier in the process. The JavaScript code then calls the SharePoint Web Service (UserProfleService.asmx) to push the data into the SharePoint User Profile.

The process of moving data from Office 365 User Profiles into Active Directory is basically the same, but  in reverse. For those User Profile Properties that are marked has being able to be read by Everyone, the Custom Windows Service can read the data directly via the UserProfileService.asmx web service and we can short circuit a lot of the process.

Summary:
Using the above approach, we built an automated process that is completely configurable regarding property mappings and synchronisation direction, that works for various data types, across both 2010 and 2013 versions of SharePoint Online. It's admittedly an awkward approach, but it appears to be the only way to achieve the desired outcome with the current limitations in SharePoint Online.

As a final note, this is a great example of the type of solution that could be achieved using the new SharePoint 2013 Provider Hosted Application model - provided of course that 2010 tenancy support is not required.

Maximum of 10 fields in a Secure Store Target Application

The blog post title pretty much says it all - in all variants of the Secure Store Service Application: both SharePoint 2010 and 2013; on-premises or Office 365; you can't have more than 10 fields in a Target Application.

This seems a rather arbitrary and constrictive limitation - in the app I was working on we had a total of 11 fields that we wanted to secure so now we will have to create two Target Applications which is an annoying additional overhead.

Trying to do this through the UI gives an error with "The maximum field number was reached.", here's a screenshot of it in the newly release SharePoint 2013 version of Office 365:

This error dialog popus once when you hit "Add Field" if you already have 10 fields listed.

If you try via PowerShell, then you get this response:


New-SPSecureStoreApplication : Specified argument was out of the range of valid values.
Parameter name: applicationFields


I couldn't find this limitation documented anywhere other than in this Office 365 article - but of course, the limitation is broader than just SharePoint Online.

It's not a blocking issue because you can always create more Target Applications, but hopefully being aware of this limitation will allow you to plan out your Target Applications without having to figure out why your PowerShell deployment scripts don't seem to behave as expected.

Perth SharePoint User Group Presentation Deck

I'm thrilled to be presenting later today at the Perth SharePoint User Group on Cloud and Hybrid SharePoint Deployments: Enterprise User Profile Management. Here's a download link to the slide deck in case anyone wants to follow up on any of the resources or information presented:

http://sdrv.ms/YBbnfE

Thanks to the Perth SPUG team for the opportunity to present and to all the attendees!

Synchronising Active Directory User Attributes into SharePoint Online User Profile Properties

Picture an organisation that uses Active Directory for Identity Management, and their AD database contains a range of user properties. That shouldn't require too much imagination...

Now imagine that, after previously having no real trouble synchronising these properties into SharePoint on-premises User Profiles in the past, they now want to synchronise the properties into Office 365 SharePoint Online User Profiles. It would be natural to assume that once ADFS and DirSync were correctly configured, it would be pretty basic to set up property mappings as needed. Unfortunately, things aren't quite so simple...

The short story is that only the following properties get synchronised (here I'm ignoring internal reference properties that end users won't care about):

AD Attribute	User Profile Property
givenName	First Name
sn	Last Name
displayName	Name
telephoneNumber	Work Phone
department	Department
title	Title
title	Job Title
manager	Manager
wWWHomePage	Public Site Redirect
proxyAddresses	SIP Address
mail	Work e-mail
physicalDeliveryOfficeName	Office

This is a good start but many organisations will want more (Mobile Number is the first one that comes to mind). Unfortunately, it just isn't possible in the current version of SharePoint Online. You can't add any more properties to be synchronised or even change the mappings for the properties that are already being synchronised. If you try to modify a User Profile Property through the Office 365 Admin UI, the relevant options are presented but not usable:

Here's a property which is already being synchronised through ADFS that I'd like to modify but there are no Source Data Connections or Attributes to select.

Here's the not so helpful dropdown when I click on Source Data Connection. It looks like this UI has been left in SharePoint Online but has no function at this point in time.

What is really frustrating is all that text on the left relating to BCS which would be a viable option in SharePoint on-premises (and was suggested to me by multiple people). Unfortunately, this process requires configuring the BCS service application which is not possible in SharePoint Online. This Community forum post reiterates that the approach is not available for Office 365.

I've discussed this with a few people within Microsoft and the limitation has been acknowledged although it's not directly documented anywhere.

One key fact to remember in understanding the cause of this limitation is that the synchronisation is a two step process. First, the properties are synchronised from your on-premises domain up into the Office 365 domain using ADFS and DirSync (I'm no expert on this process and I've probably oversimplified it). This process allows the data to become available to the Exchange and Lync components of Office 365. The properties that are synchronised in this process are listed by Microsoft here, as far as I am aware this cannot be customised (but I'd welcome it if anyone can correct me here).

Secondly, the properties are synchronised from the cloud domain into SharePoint Online User Profiles using a synchronisation process which is completely internal and hidden from Office 365 tenancy administrators. The documentation here shows the default property mappings for this process for SharePoint 2010 which as far as I can tell applies to both on-premises and Office 365 instances of SharePoint. The key difference is that on-premises SharePoint provides a friendly interface for customising these mappings whereas SharePoint Online offers no UI or administrative capacity at all.

By combining the above documentation it's fairly simple to work out which properties make it all the way from start to finish. If you want more properties then you'll have to put together a completely custom process, but that's a topic for another post...

Note that all of this discussion is based on the 2010 (Wave 14) version of SharePoint Online, I haven't seen ADFS hooked up to a Wave 15 version yet so I'm not sure whether the situation will improve in the upcoming release of Office 365. I'd love to hear if anyone can shed some light on this!

SharePoint 2013 Explorer

UPDATE: Newer version of the tool now available here.

Here is one of the first developer utilities specifically built for SharePoint 2013. It will let you peek and poke at the internals of your SharePoint 2013 site collections, either on-premises or in the cloud.

The tool is based on SharePoint Explorer 365, read this post for the full details of its origin, capabilities and usage instructions. The changes in this new version are as follows:

• Updated passive authentication code for Office 365 Preview as detailed in this post.
• Compiled against SharePoint 2013 versions of Microsoft.SharePoint.Client.dll and Microsoft.SharePoint.Client.Runtime.dll, which provide a range of additional objects, properties and methods compared to the 2010 versions (see below screenshots).
  Because of this, this version of the tool will not work against SharePoint 2010 site collections.
• Compiled against the .NET Framework 4.0, so this needs to be installed if you want to run the tool.

Here is the download, read my previous post on SharePoint Explorer 365 regarding disclaimers and such.

You can see the expanded capacity of the 2013 version of the Client Object Model by comparing screenshots of the tool compiled with 2013 assemblies with screenshots of the same tool compiled with the 2010 assemblies.

Site Collection objects and properties in 2010 Client OM assemblies

Site Collection objects and properties in 2013 Client OM assemblies

Methods available against the Site Collection in 2010 assemblies

Methods available against the Site Collection in 2013 assemblies

This expansion of Client OM capabilities in SharePoint 2013 is great news, especially for Office 365 developers. The screenshots above relate to site collections, but there are new capabilities for webs, lists, and other SharePoint objects as well. This tool, or a .NET reflection tool, provides great insights into these new capabilities.

Note that SharePoint 2013 is in Preview and is liable to change as it matures (this is especially true of the Online version), so the tool may stop working at any time. In the meantime, have fun with it!

Thanks again to Stefan Stanev for building such an easily extendible SharePoint utility and providing the source code to the SharePoint community!