SharePoint 2013: Personal Site instantiation queues and bad throughput

In SharePoint 2013 the way Personal Sites (aka My Sites) are created have been totally rebuilt to support the new way of utilizing the Personal Sites. In this post I will go through how Personal Sites are provisioned, asynchronously, and bust a couple of myths about how interactive Personal Site instantiations should be “prioritized” and increase throughput.

Background

Personal Sites or My Sites were previously created “on-demand”. When a user went to his or hers non-existing My Site the provisioning started while the user waited for the site to be created, painfully watching the spinning animated gif. This worked fine in SharePoint 2010, and earlier, but now with SharePoint 2013 so much more are depending on the user having a Personal Site – everything from the social stuff in SharePoint 2013 (yes, not all SharePoint customers have yet wandered down the Yammer road) to the really great Office 2013 interaction (SkyDrive whatever). Requiring users to “manually” creating their Personal Sites is no longer a good option…

Note: this post is valid for SharePoint 2013 RTM – and hopefully some of the myths/bugs/features/stuff that is explained in this post will be fixed in upcoming releases…

Asynchronous Personal Site instantiation

In SharePoint 2013 the Personal Site is no longer created synchronously while the users wait for the site to be created. Instead all Personal Site provisioning are handled asynchronously by the Timer Job service (OWSTIMER.exe). So how does all this work? Whenever a user visits the My Site Host for the first time they are forced to create a Personal Site (depending on the My Site creation settings in the User Profile Service Application). The Personal Site instantiation are then added to one of possibly three queues (more about these in a bit). A set of timer jobs picks up the instantiation requests from these queues, creates the Personal Site and finally sends an e-mail to the user. The benefits of this new approach are many; the user can continue to do other stuff in SharePoint (real productive work instead of facetweeting for instance), the provisioning are not hogging the w3wp.exe process, we avoid performance bottlenecks, we should(!) see improved Personal Site creation throughput etc. There are of course drawbacks of this approach, such as it could potentially take some time before the personal site is created, since we have to wait for the timer job etc. But all in all this is something that is necessary (especially for the cloud) and something is very good.

The  Instantiation queues

I mentioned that there are three queues for the Personal Site instantiation.

• The Interactive Request Queue
• The Second Interactive Request Queue
• The Non-interactive Request Queue

First of all – the difference between the Interactive and Non-interactive requests are basically that all “requests” to create a Personal Site through the web user interface are called interactive, while those requests coming from the Office client are non-interactive. The reason of having a second interactive request queue is that requests coming from the user interface – users actually wanting to facetweet – should have a higher priority and a higher personal site provisioning throughput.

Each queue are (by default) polled every minute, by three timer jobs – one per queue, Each Timer Job will retrieve five (5) items from the queue and process them one at a time. According to some (ahem, SPC12) presentations this timer job are executed on every WFE, but that’s not what happens in real life. The Timer Jobs are based on the SPWorkItemJobDefinition Job Definition Type. This is a really nice timer job implementation that has a queue per content database. In this case the queue exists in the Content Database where the My Site Host is residing. This also means since the lock type of the job is Content Database per Web Application – only one server will run this timer job. So, to sum it up – for each queue and every minute a total of 5 Personal Sites can be created, which means about 300 Personal Sites per hour per queue – if you have the hardware that can handle that! One might think now that since there are two interactive queues, we can get a throughput of 600 Personal Sites an hour, well…no…

The second (and useless) interactive queue

The idea with the second interactive queue is that interactive personal site requests should have a higher throughput, and that is a good idea. But unfortunately something went wrong when this was implemented. Stop reading now if you don’t like digging deep into SharePoint internals and just accept the fact that this implementation is flawed…

The three queues, or rather the three Timer Jobs are created per Web Application through a hidden feature called MySitesInstantiationQueues. This feature creates the timer jobs and also configures an object in the config database (SPPersistedObject) which contains a value of how many queues that should be used. The funny thing is that this is an internal definition (the SPPersistedObject) and it sets the number of instantiation queues to 1, not 2. This means that we’re hardcoded to only have one instantiation queue, and the second interactive timer job just don’t do anything since it’s queue is always empty – even though we could use reflection to modify it, but then we’re in unsupported territory. So there goes the idea of bumping up the number of Personal Sites created asynchronously.

You can very easily see this in the ULS logs:

Also, if we take a look directly at the configuration object we’ll see that it is configured only for one queue:

So the idea of having servers provisioning up to 600 Personal Sites an hour we instead have 300 Personal Sites per farm per hour. Imagine a new corporate portal launch spiced with social features for 20.000 users and about 5% of the users visiting the site at launch. This will give us 1.000 users in the queue and a potential queue time of 3:20 hours, at least – not that great in my opinion. Just imagine if you did some marketing of your new launch and only a couple of more users tried to use the social features…

Well, the good thing is that we don’t have the web servers hogged with creating thousands of Personal Sites, prohibiting real work.

Increasing the throughput anyways…

This section should really be classified under the unsupported things you shouldn’t do, ever, unless you need to… But for a large Enterprise Social deployment using SharePoint 2013 it might be a necessity, to not piss the end users off by having them waiting four hours to get started using SharePoint 2013.

If you don’t want to provision all the Personal Sites in advance, which in most cases are a dumb idea, you could instead of just relying on the queues to provision the Personal Sites you could read the queue yourself and create the personal sites “manually”. You need to find the database where the My Site is hosted and then query the ScheduledWorkItems table for items. Query for where the Type is E94A6CAA-B0F5-4897-B489-585CA50C7803 (which is the id of the first interactive instantiation queue, the second queue has it’s own Type Id – but you will never see that in here :-)). Find those with the InternalState equal to 8 – those are the ones waiting to be processed. Using this information you can use PowerShell or similar to spread out the load on other servers to create the personal sites.

To create the actual personal sites you could use the CreatePersonalSite() method of the UserProfile object, this method will bypass the queue and immediately create the site. 

Summary

By now you should be familiar with the process of how SharePoint provisions Personal Sites in SharePoint 2013. It’s a better approach than in previous versions. The idea of having multiple queues and allowing several servers to help out with provisioning the personal sites is good, but currently flawed. Hopefully this will get fixed in upcoming cumulative updates. As it is now it’s not working for a large enterprise rolling out SharePoint Enterprise Social, so you have to be very careful when planning a roll out of Enterprise Social in SharePoint 2013…or use Yammer…