Twas the nightmare before content deployment

A lot of things have been written on +,-,*,/ and ^ of content deployment. Recently I had to go thru the seemingly-simple-stuff-called-content-deployment, but came out a bit wise. There are definitely some excellent resources, for eg this Spence’s blog. Though I do not agree with the view that Content Deployment is simple. It may be simple for someone who has been working with SharePoint for 12 years. In my view, content deployment either is so simple that everyone has trouble understanding its simplicity, or every one is just stupid. Some errors are cryptic and some rectifications are just not that obvious. This blog describes the woes of both migration to 2010 and a subsequent content deployment.

Abbreviations

• CD (Content Deployment)
• CDB (Content DataBase)
• FCD (Full CD)
• ICD (Incremental CD)
• SPS 2010 (SharePoint Server 2010)
• WTF (you know what)

Premises

• There exists a Server-A, with a 2007 SP site pointing to CDB2007-A;
• The CDB2007-A needs to be migrated to a new Server-B (Dev server) and upgraded to SPS 2010 via db-attach method, the new CDB will be called CDB2010-B
• There is another Server-C (QA server). This will also have a new CDB, lets call it CDB2010-C

Problem

Do a FCD from CDB2010-B to CDB2010-C. This is where the fun begins.

Solution (Kinda)

Wisdom #1: You cannot do a FCD or ICD from a MOSS 2007 CDB to SPS 2010 CDB.

Can I… ?

No, you cannot.

So after you do your upgrades to all these servers, and if your CDB2007-A has changed, you cannot move your content from CDB2007-A to CDB2010-B or CDB2010-C. You have to do db-attach again. Because of the schema changes between MOSS 2007 and SPS 2010.

Wisdom #2: Make sure your Language Packs are installed in both Server-B and Server-C (as it was in Server-A)

We missed this due to Microsoft’s ingenious policy of naming all SPS language pack downloads as “ServerLanguagePack.exe”. Yes, whether you download an English or Spanish or Arabic language pack, they will all be called “ServerLanguagePack.exe”. So if you download multiple language packs, it is your responsibility to rename the downloaded file name in order to avoid overwriting it. Only when you run it and you find the actual information is in that language, you know that you are installing the correct language pack. The default download url will download the English language pack. Use Change language dropdown to download the language pack of your choice. Surely ServerLanguagePack-en-US.exe, ServerLanguagePack-es-es.exe would have been helpful. But Microsoft’s, and in general .Net’s naming conventions are nothing to rave about. Site vs Web vs WebSite vs WebApp vs Page vs SitePage vs Application Page anyone?

Wisdom #3: Once the language pack is installed, goto 14 HIVE and check if a directory with language code exists (Eng – 1033, Spanish – 3082 etc). You can also verify that looking in the Control Panel > Programs and Features which would say that the language pack is installed. Another way is goto Registry Editor, under Hive, check for InstalledLanguages key which would contain the language pack code.

If you do not install the Language packs, you will get errors. This error when translated to plain English means Spanish Lang Pack is missing:

[2/28/2011 10:45:05 AM] [File] [picture.png] Progress: Exporting File /_catalogs/masterpage/es-es/Preview Images/picture.png.
[2/28/2011 10:45:05 AM] [File] [picture.png] Error: Cannot open file “_catalogs/masterpage/es-es/Preview Images/picture.png”.
[2/28/2011 10:45:05 AM] [File] [picture.png] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.GetFileAsStream(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob, String bstrEtagNotMatch, String& pEtagNew, String& pContentTagNew)

— Prepare the Server-B (Dev)

• Back up CDB2007-A database using SQL Server-Backup
• Restore CDB2007-A as a new CDB, and call it CDB2010-B
• Provide security permissions (dbowner) to the new farm admin account/users/remove if any old users from CDB2010-B
• Create a new WebApp-B on Server-B (only one site collection, but several subsites)
• Mount CDB2010-B to WebApp-B
• Mount-SPContentDatabase “CDB2010-B” -DatabaseServer “dbserver” -WebApplication http://WebApp-B

Wisdom #4: In Server-B and Server-C, activate the required Features, same as those that existing Server-A. Having required features not activated will result in CD failure. Typically this error is very clear expressing that a required feature is missing or not activated.

• Deploy any WSP-s and update the web.config as required.
• Verify WebApp-B is working fine.

— Prepare the Server-C (QA); FCD to this server is our target

• Create a new WebApp-C on Server-C

Wisdom #5: The site collection on WebApp-C must have the SAME root site template as that of the WebApp-B.

This is the formula:

WebApp-B.SiteCollection.RootSite.SiteTemplate must be equal to WebApp-C.SiteCollection.RootSite.SiteTemplate
If this is not the case, on a FCD, you will get a Site template mismatch error, like this:

[2/28/2011 10:47:05 AM] [Web] [] Progress: Importing
[2/28/2011 10:47:05 AM] [Web] [] Error: Cannot import site. The exported site is based on the template SPS#0 but the destination site is based on the template STS#0. You can import sites only into sites that are based on same template as the exported site.
[2/28/2011 10:47:05 AM] [Web] [] Debug: at Microsoft.SharePoint.Deployment.WebSerializer.IsWebTemplateCompatible(String sourceWebTemplateName, String destinationWebTemplateName)
at Microsoft.SharePoint.Deployment.WebSerializer.SetObjectData(Object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector)
at Microsoft.SharePoint.Deployment.XmlFormatter.CallSetObjectData(Object obj, SerializationInfo objectData, ISerializationSurrogate surrogate, ISurrogateSelector selector)
[2/28/2011 10:47:05 AM] FatalError: Cannot import site. The exported site is based on the template SPS#0 but the destination site is based on the template STS#0. You can import sites only into sites that are based on same template as the exported site.
[2/28/2011 10:47:05 AM] Debug: at Microsoft.SharePoint.Deployment.WebSerializer.IsWebTemplateCompatible(String sourceWebTemplateName, String destinationWebTemplateName)

To find that, open powershell and execute the following command on Server-B:

$site = Get-SPSite(“http://WebApp-B”)
$site.allwebs[0].webtemplate

The result will be like SPS, STS#0, MPS, BLANK INTRANET etc. Complete list can be found in Microsoft’s site.

In our case it was SPS, which is actually, sigh, a SharePoint Portal Server 2003’s team site template. But we assumed it was STS#0. Some consultant had the wisdom of creating a SPS root site in MOSS 2007.

• Anyway, created a new SPS team site on WebApp-C.
• You can do it by GUI, Powershell or stsadm. Apparently, after searching several blogs and attempting to do it with GUI and Powershell, the conclusion is to use stsadm:

stsadm -o createsite “http://WebApp-C” -siteowner “yourname” -owneremail “yourname@yourcompany.com” -sitetemplate “SPS”

• Mount the CDB2010-C to WebApp-C
• Mount-SPContentDatabase “CDB2010-C” -DatabaseServer “sps2010sqlserver” -WebApplicatin http://WebApp-C
• Now set up a FCD on Server-B from WebApp-B targetting WebApp-C and keep watching the report.

Wisdom #6: Make sure you Mount the CDB2010-C when you are logged in as Farm Admin Account. Also make sure none of your files are checked out in CDB2010-B using SPD 2010. Checked-out but Published files cause CD failures.

Now got the error:

System.Data.SqlClient.SqlException: The DELETE statement conflicted with the REFERENCE constraint “FK_Dependencies1_Objects”. The conflict occurred in database “xxxxx” table “dbo.Dependencies”, column ‘ObjectId’. The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

Wisdom #7: Exit PowerShell; Open new PowerShell; Unmount and Mount the CDB2010-C again; Start the FCD.

Yes, opening a new PowerShell seems to clear up some cache and do it correctly!

Now the we are progressing… to the next error!

[2/28/2011 5:45:16 PM] [List] [fpdatasources] Progress: Exporting List fpdatasources.
[2/28/2011 5:45:16 PM] [List] [fpdatasources] Error: Object reference not set to an instance of an object.
[2/28/2011 5:45:16 PM] [List] [fpdatasources] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.ExpandListSchemaForExport(String bstrUrl, String bstrTitle, Int32 lTemplateID, String bstrFeatureId, String bstrFields, String& pbstrFullSchema, String& pbstrIndexSchema)

Wisdom #8: Goto http://centraladmin/_admin/ServiceJobDefinitions.aspx. Find and Run the “Timer Service Recycle”.

Here is what happened: When trying to do FCD, we were getting the “Object reference not set” in relation to _catalogs/fpdatasources. We detached the CDB2010-C, reattached, ensure its mounted as Farm Admin, still this error didnt go away. Took a break, came in the next day morning and tried the FCD – it went past the above “Object reference not set” error, nothing was changed in between! (even though it did not fully succeed). I scouted the logs. We had set the FCD to Hourly. So this error was there in the 6 AM logs, but in the 7 AM logs this error was not there anymore. So there is some service that ran in between that cleared up stuff. I took a diff of all services between 5-6 AM and 6-7 AM and “Timer Service Recycle” was the only difference which was set to run at like 6:40 AM. Now I ran this manually, it took about 10 mts on our servers, gave it a few more minutes to settle down. Ran the FCD again, voila, the _catalogs/fpdatasources error is gone.

Ok Now the FCD export was smooth!

• But the imports are failing. Lets look at the Server-C logs (target server).

[2/28/2011 6:29:21 PM] [Folder] [_catalogs] Progress: Importing
[2/28/2011 6:29:21 PM] [Folder] [_catalogs] Verbose: Source URL: _catalogs
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Progress: Importing
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Verbose: Source URL: _catalogs/fpdatasources
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Error: Unable to import the folder _catalogs/fpdatasources. There is already an object with the Id 95ff96a8-e0c8-4f42-8d2f-6cf057b78e98 in the database from another site collection.
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Error: Unable to import the folder _catalogs/fpdatasources. There is already an object with the Id 95ff96a8-e0c8-4f42-8d2f-6cf057b78e98 in the database from another site collection.
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Debug: at Microsoft.SharePoint.Deployment.FolderSerializer.GetFolderFromDatabase(SPWeb parentWeb, Guid folderId, String sourceFolderUrl, Boolean restoreRecycleBinItems)
at Microsoft.SharePoint.Deployment.FolderSerializer.GetFolder(SPWeb parentWeb)

Wisdom #9: If you had multiple errors before, it is best to restart by recreating the WebApp-C in Server-C. That seems to take the above error away.

Finally we are left with these errors:

[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Progress: Importing
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: Source URL: _catalogs/masterpage/Forms/mod-view.aspx
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: Destination URL: /_catalogs/masterpage/Forms/mod-view.aspx
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: pages\viewpage.aspx 4
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Error: An unexpected error has occurred.
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.SetGhostedFile(String setupPath, Byte setupPathVersion, Int32 iVersion, Guid& pguidListId, Guid& pguidDocId, Int32 lDoclibRowId, Int32 iAuthorId, String authorLogin, String serverRelativeUrl, String targetUrlWebRelative, Boolean isInDocLib, Boolean isMigration, Boolean isPublishing, Object varProperties)

[3/1/2011 6:03:10 AM] [List] [wfpub] Progress: Importing
[3/1/2011 6:03:10 AM] [List] [wfpub] ExtendedVerbose: Creating
[3/1/2011 6:03:10 AM] [List] [wfpub] Error: A list, survey, discussion board, or document library with the specified title already exists in this Web site. Please choose another title.
[3/1/2011 6:03:10 AM] [List] [wfpub] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.CreateListOnImport(String bstrUrl, Guid& pguidListId, String bstrTitle, String bstrDescription, Int32 lTemplateID, String bstrFeatureId, Guid guidRootFolderId, Int64 llFlags, Int32 iVersion, Int32 iAuthor, String bstrFields, String bstrContentTypes, String bstrImageUrl, String bstrEventSinkAssembly, String bstrEventSinkClass, String bstrEventSinkData, Guid guidDocTemplateId, String bstrViews, String bstrForms, Boolean bCompressedSchema)

There are lot of the unexpected errors. As of writing this, we are not past this issue, not sure what’s the cause. The errors happen to be coming from the SPS’ internal files (AllItems.aspx etc). Some blog said to try Chris O’ Brien’s CD utility, we tried, didnt help. Some post said to match GUIDs of the List Default Pages (All Items, Edit Form etc) in the manifest.xml generated by CD utility. I haven’t figured that out yet.

Wisdom #10: We finally ended up with Backing up and Restoring the CDB2010-B on CDB2010-C, and doing an ICD instead of a FCD.

I know I didn’t use the WTF abbreviation at all, but it will be polite of you to add it after every error.