Recover site collection from a backup

I had accidentally restored a site collection (using Restore-SPSite) from another web application over our ongoing development site collection. Which meant the front-end (SharePoint Designer) work done on the development was lost.

Luckily I had a SQL database backup of the content database. I got the database restored into another similar existing web application (instead of creating a new web application again).

When I hit the site, I got the error

<pre>
400 BAD Request
</pre>

Now in the Central Admin, Manage Content Databases, I see that the content database indeed had one site collection. Ok, site collection is there, but is there a web?

$s = Get-SPSite http://mysite
$s.openWeb()

Url
----

So, the rootweb is missing or not recognized. When I googled various forums, surprisingly there is not a solution to be found for this. I tried recycling the apppool, iisreset, mounting the content database with a new assignment id. Nothing worked.

But the solution itself is simple. Just detach and re-attach the content database to the web application.

Dismount-ContentDatabase $contentDb
Mount-ContentDatabase -ContentDatabase $contentD -DatabaseServer $dbServer -WebApplication http://mysite

Inconspicuous SharePoint Quirks #2

Mystery of the vanishing Publishing Hyperlink

Note: Publishing Hyperlink can be created as a site column “Hyperlink with formatting constraints”.

There seems to be a bug in Publishing Hyperlink field in SharePoint 2010 and it does not happen consistently. Lets say there is a List with Publishing Hyperlink site column. When you edit this list item, and focus on the url, a new “Link Tools” (green tab) appears in the editor. This brings additional properties to edit.

If you change the link value here, change and save, sometimes the link value disappears. Or if you click on Remove Link button, the “click here to add new link” disappears. I was able to verify this consistently in a totally clean site (SharePoint Server Build  14.0.5128.5000). This means you cannot add a link at all anymore. Your only option is to delete the item and add it again. Goodluck if the item is huge and/or has some relation to other lists!

Fortunately there is powershell, you can update the link directly. The trick is, while the regular Hyperlink is a string, the Publishing Hyperlink is an object. So you have to update the object properties first and set the object itself to the listitem and then update the list item. The following powershell code shows that:

Param([string]$site0="http://mysharepoint", [string]$list0="My List", [string]$itemKey0="Item Key", [string]$linkUrl0="http://newlink", [string]$desc0="New description", [string]$toolTip0="New Tooltip")

$site = Get-SPSite($site0)
$rootWeb = $site.rootWeb
$lists = $rootWeb.Lists
$list = $lists[$list0]
$items = $list.Items
$listItem = $items | Where-Object {$_.Name -eq $itemKey0}
$linkUrl = $listItem["Link Url"]
$linkUrl.NavigateUrl = $linkUrl0
$linkUrl.Description = $desc0
$linkUrl.ToolTip = $toolTip0
# this is the trick - set the object back to listItem
$listItem["Link Url"] = $linkUrl
$listItem.Update()

Inconspicuous SharePoint Quirks #1

Mystery of the Unimportable WebPart

So Voltaire said “God protect me from my friends, I will take care of my enemies”. If there was a SharePoint God, I would probably pray to him “Protect me from the inconspicuous quirks, I will blogoogle the rest”.

So I have a wsp with a webparts feature. After deploying a wsp to the dev server, I added a webpart to a page which showed me a charming message “Cannot import webpart”. Call me schizophrenic but a voice inside me told that dont waste time at the logs, they are not going to help. This was working fine in my local vm, but trouble-some in dev server. So what’s wrong? After a miss-adventrous chasing of dead-ends, cul-de-sacs, and no-outlets finally figured a method that works consistently.

  1. First of all, ensure that the elements.xml, SharePointProjectItem.spdata, the .webpart file, feature.xml, package.xml are all correct. Make sure the Group is correct too, or else your users will not find the expected webpart.
  2. If your webpart is extending from Microsoft.SharePoint.WebPartPages.WebPart, you must use the wss/v2 schema, ie the webpart should have a .dwp extension. The newer .webpart extension/v3 schema works only for ASP.Net webparts
  3. After deploying the wsp, check if the web.config has the correct SafeControl entry with correct version of assemblies. Also check the GAC for correct version.
  4. Goto http://server/_catalogs/wp/Forms/AllItems.aspx. This lists all the webparts in Web Part Gallery. See the Modified Time, is it the latest? If not, delete all the webparts. (You can use a simple powershell script for that).
  5. Now deactivate the webpart feature (Disable-SPFeature featureName)
  6. Again activate the webpart feature (Enable-SPFeature featureName)
  7. Check the Web Part Gallery, now all the webparts should be added.
  8. Try adding the webpart again and if it still says “Cannot import webpart”, pray to SPGod.

So why was it working on my local vm? Oh yeah. Turns that there is a bug in Visual Studio 2010 Deploy command which would not Activate the features automatically. So some time ago I had set it to “No-Activation” deployment configuration and I was activating the features via powershell. That powershell activation was taking for adding the webparts correctly to the web part gallery. On the dev server, I was assuming that the install-spsolution automatically refreshes the web part gallery too, but it does not.

It is well blogumented that webparts are not removed from the gallery upon uninstall-spsolution, but now I know it does not update the webparts  either using install-spsolution, yet the feature is blissfully activated. Cunningly symmetrically logical eh?

Download files from Content Database

Problem

Compare master page content from two content databases, modified independently.

Pre-conditions

In a SharePoint migration project, was in a situation where the content of the application in two diffrent servers (say dev vs test) had changed independently. In other words, dev cdb was modified, so was the test cdb. And both changes were almost valid. I had to individually pick and choose changes from dev and test and merge into a single source. And there were several such files.

SharePoint Designer 2010 exibhits a Vedantic property of being the best, not-so-best and neither both all at the sametime. In other words, the goodness in you can never the triumph the evil of SPD and will eternally continue to fight with it.

Modifying each file and merging would be cumbersome, especially when SPD usually likes to mind its own business when you want it to respond. So I decided to download the files from both content databases locally, then use a sophisticated compare tool like Beyond Compare or WinMerge, merge them and upload it back to the content database.

Solution

Microsoft must have finally shed some sense of shame and hired a few Unix guys who probably are behind PowerShell. So here comes PowerShell again to the rescue to do that effortlessly.

#
$url = "http://yoursite"
$masterPageGallery = "/_catalogs/masterpage"
$destPath = "c:/site1"
#
$site = Get-SPSite($url)
$webs = $site.AllWebs
$rootWeb = $site.RootWeb
#Get the MasterPageGallery folder
#Another way to get MPG is $rootWeb.GetCatalog(116)
$folder = $rootWeb.GetFolder($masterPageGallery)
$files = $folder.Files
foreach ($f in $files) {
 [Byte[]] $fBytes = $f.OpenBinary()
 #Store the name explicitly for creating DestPath;
# by default $f := $f.Url, not $f.Name
 $fileName = $f.Name
 $destFile = "$destPath/$fileName"
 [System.IO.File]::WriteAllBytes($destFile, $fBytes)
}

Upgrading to SharePoint 2010 look and feel

Problem

After a recent SharePoint 2007 to 2010 upgrade (via database attach method) to one of our medium-sized web application, we were stuck with the old MOSS 2007 look and feel for SharePoint functionality. Moreover pretty much all our sites use several custom master pages. Doing a Visual Upgrade would reset all the sites to v4.master and would completely remove the custom master pages.

The task was to find an easy way to switch the look and feel for SharePoint functions to v4, but still retain our custom master pages for our sites. Yes we could have done it individually and manually, but why fret if the mighty PowerShell handy-man is around.

Solution

Iterate over the sites, remember the old master pages, reset the UI version and reset the old master pages.

Before running the scripts, it is useful to take a look at the web page for changing master page layouts http://sharepoint-web-app/<site>/_layouts/ChangeSiteMasterPage.aspx. There are two fields Site Master Page and System Master Page. The former is for publishing pages and the later is for forms and views (think AllForms.aspx and Views.aspx).


$rootSite = get-spsite("http://sharepoint-web-app")
$webs = $rootSite.AllWebs
foreach($w in $webs) {
 Write-Host
 Write-Host $w
 Write-Host "       " -Separator ": " $w.CustomMasterUrl $w.MasterUrl

 # CustomMasterUrl is the Site Master Page
 $oldCustomMasterUrl = $w.CustomMasterUrl
 # MasterUrl is the System Master Page
 $oldMasterUrl = $w.MasterUrl
 $w.UIVersion = 4
 $w.UIConfigurationVersionEnabled = $false
 # At this point, the CustomMasterUrl is set to v4.master
 $w.Update()
 # Reset to old master pages
 $w.CustomMasterUrl = $oldCustomMasterUrl
 $w.MasterUrl = $oldMasterUrl
 $w.Update()
}
$rootSite.Dispose()

The script iterates over all the sites and sub-sites, remembers their old master pages, switches to v4.master and resets the sites to old master pages. In effect, the Site Settings and other pages will have the SP 2010 look and feel.

Twas the nightmare before content deployment

A lot of things have been written on +,-,*,/ and ^ of content deployment. Recently I had to go thru the seemingly-simple-stuff-called-content-deployment, but came out a bit wise. There are definitely some excellent resources, for eg this Spence’s blog. Though I do not agree with the view that Content Deployment is simple. It may be simple for someone who has been working with SharePoint for 12 years. In my view, content deployment either is so simple that everyone has trouble understanding its simplicity, or every one is just stupid. Some errors are cryptic and some rectifications are just not that obvious. This blog describes the woes of both migration to 2010 and a subsequent content deployment.

Abbreviations

  • CD (Content Deployment)
  • CDB (Content DataBase)
  • FCD (Full CD)
  • ICD (Incremental CD)
  • SPS 2010 (SharePoint Server 2010)
  • WTF (you know what)

Premises

  • There exists a Server-A, with a 2007 SP site pointing to CDB2007-A;
  • The CDB2007-A needs to be migrated to a new Server-B (Dev server) and upgraded to SPS 2010 via db-attach method, the new CDB will be called CDB2010-B
  • There is another Server-C (QA server). This will also have a new CDB, lets call it CDB2010-C

Problem

Do a FCD from CDB2010-B to CDB2010-C. This is where the fun begins.

Solution (Kinda)

Wisdom #1: You cannot do a FCD or ICD from a MOSS 2007 CDB to SPS 2010 CDB.

Can I… ?

No, you cannot.

So after you do your upgrades to all these servers, and if your CDB2007-A has changed, you cannot move your content from CDB2007-A to CDB2010-B or CDB2010-C. You have to do db-attach again. Because of the schema changes between MOSS 2007 and SPS 2010.

Wisdom #2: Make sure your Language Packs are installed in both Server-B and Server-C (as it was in Server-A)

We missed this due to Microsoft’s ingenious policy of naming all SPS language pack downloads as “ServerLanguagePack.exe”. Yes, whether you download an English or Spanish or Arabic language pack, they will all be called “ServerLanguagePack.exe”. So if you download multiple language packs, it is your responsibility to rename the downloaded file name in order to avoid overwriting it. Only when you run it and you find the actual information is in that language, you know that you are installing the correct language pack. The default download url will download the English language pack. Use Change language dropdown to download the language pack of your choice. Surely ServerLanguagePack-en-US.exe, ServerLanguagePack-es-es.exe would have been helpful. But Microsoft’s, and in general .Net’s naming conventions are nothing to rave about. Site vs Web vs WebSite vs WebApp vs Page vs SitePage vs Application Page anyone?

Wisdom #3: Once the language pack is installed, goto 14 HIVE and check if a directory with language code exists (Eng – 1033, Spanish – 3082 etc). You can also verify that looking in the Control Panel > Programs and Features which would say that the language pack is installed. Another way is goto Registry Editor, under Hive, check for InstalledLanguages key which would contain the language pack code.

If you do not install the Language packs, you will get errors. This error when translated to plain English means Spanish Lang Pack is missing:

[2/28/2011 10:45:05 AM] [File] [picture.png] Progress: Exporting File /_catalogs/masterpage/es-es/Preview Images/picture.png.
[2/28/2011 10:45:05 AM] [File] [picture.png] Error: Cannot open file “_catalogs/masterpage/es-es/Preview Images/picture.png”.
[2/28/2011 10:45:05 AM] [File] [picture.png] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.GetFileAsStream(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob, String bstrEtagNotMatch, String& pEtagNew, String& pContentTagNew)

— Prepare the Server-B (Dev)

  • Back up CDB2007-A database using SQL Server-Backup
  • Restore CDB2007-A as a new CDB, and call it CDB2010-B
  • Provide security permissions (dbowner) to the new farm admin account/users/remove if any old users from CDB2010-B
  • Create a new WebApp-B on Server-B (only one site collection, but several subsites)
  • Mount CDB2010-B to WebApp-B
  • Mount-SPContentDatabase “CDB2010-B” -DatabaseServer “dbserver” -WebApplication http://WebApp-B

Wisdom #4: In Server-B and Server-C, activate the required Features, same as those that existing Server-A. Having required features not activated will result in CD failure. Typically this error is very clear expressing that a required feature is missing or not activated.

  • Deploy any WSP-s and update the web.config as required.
  • Verify WebApp-B is working fine.

— Prepare the Server-C (QA); FCD to this server is our target

  • Create a new WebApp-C on Server-C

Wisdom #5: The site collection on WebApp-C must have the SAME root site template as that of the WebApp-B.

This is the formula:

WebApp-B.SiteCollection.RootSite.SiteTemplate must be equal to WebApp-C.SiteCollection.RootSite.SiteTemplate
If this is not the case, on a FCD, you will get a Site template mismatch error, like this:

[2/28/2011 10:47:05 AM] [Web] [] Progress: Importing
[2/28/2011 10:47:05 AM] [Web] [] Error: Cannot import site. The exported site is based on the template SPS#0 but the destination site is based on the template STS#0. You can import sites only into sites that are based on same template as the exported site.
[2/28/2011 10:47:05 AM] [Web] [] Debug: at Microsoft.SharePoint.Deployment.WebSerializer.IsWebTemplateCompatible(String sourceWebTemplateName, String destinationWebTemplateName)
at Microsoft.SharePoint.Deployment.WebSerializer.SetObjectData(Object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector)
at Microsoft.SharePoint.Deployment.XmlFormatter.CallSetObjectData(Object obj, SerializationInfo objectData, ISerializationSurrogate surrogate, ISurrogateSelector selector)
[2/28/2011 10:47:05 AM] FatalError: Cannot import site. The exported site is based on the template SPS#0 but the destination site is based on the template STS#0. You can import sites only into sites that are based on same template as the exported site.
[2/28/2011 10:47:05 AM] Debug: at Microsoft.SharePoint.Deployment.WebSerializer.IsWebTemplateCompatible(String sourceWebTemplateName, String destinationWebTemplateName)

To find that, open powershell and execute the following command on Server-B:

$site = Get-SPSite(“http://WebApp-B&#8221;)
$site.allwebs[0].webtemplate

The result will be like SPS, STS#0, MPS, BLANK INTRANET etc. Complete list can be found in Microsoft’s site.

In our case it was SPS, which is actually, sigh, a SharePoint Portal Server 2003’s team site template. But we assumed it was STS#0. Some consultant had the wisdom of creating a SPS root site in MOSS 2007.

  • Anyway, created a new SPS team site on WebApp-C.
  • You can do it by GUI, Powershell or stsadm. Apparently, after searching several blogs and attempting to do it with GUI and Powershell, the conclusion is to use stsadm:

stsadm -o createsite “http://WebApp-C&#8221; -siteowner “yourname” -owneremail “yourname@yourcompany.com” -sitetemplate “SPS”

  • Mount the CDB2010-C to WebApp-C
  • Mount-SPContentDatabase “CDB2010-C” -DatabaseServer “sps2010sqlserver” -WebApplicatin http://WebApp-C
  • Now set up a FCD on Server-B from WebApp-B targetting WebApp-C and keep watching the report.

Wisdom #6: Make sure you Mount the CDB2010-C when you are logged in as Farm Admin Account. Also make sure none of your files are checked out in CDB2010-B using SPD 2010. Checked-out but Published files cause CD failures.

Now got the error:

System.Data.SqlClient.SqlException: The DELETE statement conflicted with the REFERENCE constraint “FK_Dependencies1_Objects”. The conflict occurred in database “xxxxx” table “dbo.Dependencies”, column ‘ObjectId’. The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

Wisdom #7: Exit PowerShell; Open new PowerShell; Unmount and Mount the CDB2010-C again; Start the FCD.

Yes, opening a new PowerShell seems to clear up some cache and do it correctly!

Now the we are progressing… to the next error!

[2/28/2011 5:45:16 PM] [List] [fpdatasources] Progress: Exporting List fpdatasources.
[2/28/2011 5:45:16 PM] [List] [fpdatasources] Error: Object reference not set to an instance of an object.
[2/28/2011 5:45:16 PM] [List] [fpdatasources] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.ExpandListSchemaForExport(String bstrUrl, String bstrTitle, Int32 lTemplateID, String bstrFeatureId, String bstrFields, String& pbstrFullSchema, String& pbstrIndexSchema)

Wisdom #8: Goto http://centraladmin/_admin/ServiceJobDefinitions.aspx. Find and Run the “Timer Service Recycle”.

Here is what happened: When trying to do FCD, we were getting the “Object reference not set” in relation to _catalogs/fpdatasources. We detached the CDB2010-C, reattached, ensure its mounted as Farm Admin, still this error didnt go away. Took a break, came in the next day morning and tried the FCD – it went past the above “Object reference not set” error, nothing was changed in between! (even though it did not fully succeed). I scouted the logs. We had set the FCD to Hourly. So this error was there in the 6 AM logs, but in the 7 AM logs this error was not there anymore. So there is some service that ran in between that cleared up stuff. I took a diff of all services between 5-6 AM and 6-7 AM and “Timer Service Recycle” was the only difference which was set to run at like 6:40 AM. Now I ran this manually, it took about 10 mts on our servers, gave it a few more minutes to settle down. Ran the FCD again, voila, the _catalogs/fpdatasources error is gone.

Ok Now the FCD export was smooth!

  • But the imports are failing. Lets look at the Server-C logs (target server).

[2/28/2011 6:29:21 PM] [Folder] [_catalogs] Progress: Importing
[2/28/2011 6:29:21 PM] [Folder] [_catalogs] Verbose: Source URL: _catalogs
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Progress: Importing
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Verbose: Source URL: _catalogs/fpdatasources
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Error: Unable to import the folder _catalogs/fpdatasources. There is already an object with the Id 95ff96a8-e0c8-4f42-8d2f-6cf057b78e98 in the database from another site collection.
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Error: Unable to import the folder _catalogs/fpdatasources. There is already an object with the Id 95ff96a8-e0c8-4f42-8d2f-6cf057b78e98 in the database from another site collection.
[2/28/2011 6:29:21 PM] [Folder] [fpdatasources] Debug: at Microsoft.SharePoint.Deployment.FolderSerializer.GetFolderFromDatabase(SPWeb parentWeb, Guid folderId, String sourceFolderUrl, Boolean restoreRecycleBinItems)
at Microsoft.SharePoint.Deployment.FolderSerializer.GetFolder(SPWeb parentWeb)

Wisdom #9: If you had multiple errors before, it is best to restart by recreating the WebApp-C in Server-C. That seems to take the above error away.

Finally we are left with these errors:

[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Progress: Importing
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: Source URL: _catalogs/masterpage/Forms/mod-view.aspx
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: Destination URL: /_catalogs/masterpage/Forms/mod-view.aspx
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Verbose: pages\viewpage.aspx 4
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Error: An unexpected error has occurred.
[3/1/2011 6:02:35 AM] [File] [mod-view.aspx] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.SetGhostedFile(String setupPath, Byte setupPathVersion, Int32 iVersion, Guid& pguidListId, Guid& pguidDocId, Int32 lDoclibRowId, Int32 iAuthorId, String authorLogin, String serverRelativeUrl, String targetUrlWebRelative, Boolean isInDocLib, Boolean isMigration, Boolean isPublishing, Object varProperties)

[3/1/2011 6:03:10 AM] [List] [wfpub] Progress: Importing
[3/1/2011 6:03:10 AM] [List] [wfpub] ExtendedVerbose: Creating
[3/1/2011 6:03:10 AM] [List] [wfpub] Error: A list, survey, discussion board, or document library with the specified title already exists in this Web site. Please choose another title.
[3/1/2011 6:03:10 AM] [List] [wfpub] Debug: at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.CreateListOnImport(String bstrUrl, Guid& pguidListId, String bstrTitle, String bstrDescription, Int32 lTemplateID, String bstrFeatureId, Guid guidRootFolderId, Int64 llFlags, Int32 iVersion, Int32 iAuthor, String bstrFields, String bstrContentTypes, String bstrImageUrl, String bstrEventSinkAssembly, String bstrEventSinkClass, String bstrEventSinkData, Guid guidDocTemplateId, String bstrViews, String bstrForms, Boolean bCompressedSchema)

There are lot of the unexpected errors. As of writing this, we are not past this issue, not sure what’s the cause. The errors happen to be coming from the SPS’ internal files (AllItems.aspx etc). Some blog said to try Chris O’ Brien’s CD utility, we tried, didnt help. Some post said to match GUIDs of the List Default Pages (All Items, Edit Form etc) in the manifest.xml generated by CD utility. I haven’t figured that out yet.

Wisdom #10: We finally ended up with Backing up and Restoring the CDB2010-B on CDB2010-C, and doing an ICD instead of a FCD.

I know I didn’t use the WTF abbreviation at all, but it will be polite of you to add it after every error.

SharePoint PowerShell cmdlets

Lately I have been doing a lot of stuff using PowerShell and avoiding Central Admin UI. It is the next best thing to MS DOS. Oops, sorry… I meant Unix shell. Okay, that sarcasm wasn’t strong enough, I admit…

Anywayz, I decided to list the PowerShell cmdlets I discover or google around so it is easy for me to get to. I will keep adding as I find more. Thanks to those who blogged or suggested neat ways of using PowerShell.

;Adding a Solution
Add-SPSolution -literalpath c:\VisualWebPartProject1.wsp
Install-SPSolution -identity VisualWebPartProject1.wsp -allwebapplications

;Verifying a solution
Get-SPSolution

;Enable a Feature
Enable-SPFeature -identity VisualWebPartProject1_Feature1 -url http://yoursiteurl/

;Verify feature
Get-SPFeature | Sort-Object DisplayName

;Update a solution
Update-SPSolution -literalpath c:\VisualWebPartProject1.wsp -identity VisualWebPartProject1 -GACDeployment

;Get all webs
get-spsite

;Get all sites within a web
get-spsite("http://yoursite").allwebs

;Get all webtemplates within a web
$s = get-spsite("http://yoursite")
foreach($w in $s.allwebs) {$w, $w.webtemplate)