ISQ #7: The case of uncontrollable InitializeControl

I occasionally see tweets that Visual Studio 2012 is the best IDE. They have not seen IntelliJ. I was a fan of Eclipse for a long time and after working on IntelliJ, its hard to go back. Yeah.. you’ve heard that so many times. Visual Studio 2012 doesnt come close to that, because IntelliJ works with you, never against you. There are many things plain frustrating with VS2012 (an yelling all caps menu, lousy gray theme, even lousier icons and text and so on), the rants are all over the net, so I will skip them.

Working on a SharePoint 2010 project with Visual Studio 2012, you might have seen the error “The name InitializeControl does not exist in the current context”.

The problem happens when using sandboxed VisualWebPart, lets say DisplayWebPart.ascx, which should generate a ascx.cs file and a ascx.g.cs file. But a major bug is that the g.cs file is not often generated properly and even if you force regenerate (by manually deleting files from File Explorer), it creates DisplayWebPart1.ascx.g.cs (a mysterious 1 being appended the file name). Well try consolidating that with the TFS, if you still happen to use that archaic software as your version control system. There are several solutions on msdn, stackoverflow etc, but none really worked for me. After a lot of scratching around, I found a way that consistently generates the g.cs file, correctly.

Here are the steps:

1. Ensure the Project’s, not the Solution’s, “Site URL” is set to a valid SharePoint URL. (Impressive dependency, to regenerate your designer code locally, you need to have a valid working reachable SharePoint url)
2. Right click on .ascx file, select Properties, and update the property “Custom Tool” to “SharePointWebPartCodeGenerator”.
3. Right click on the .ascx file and click Run Custom Tool. At this point, if the “DisplayWebPart1.ascx.g.cs” is regenerated, your frustration is acceptable.
4. Open the project.csproj file in Notepad++ or other editor, find the line DisplayWebPart1.ascx.g.cs
   You should see something like this:
   <Content Include=”DisplayWebPart\DisplayWebPart.ascx”>
   <Generator>SharePointWebPartCodeGenerator</Generator><LastGenOutput>DisplayWebPart1.ascx.g.cs</LastGenOutput>
   </Content>
5. Change the LastGenOutput value to DisplayWebPart.ascx.g.cs
6. Go back to Visual Studio 2012 and it asks to Reload the project (or Discard). 
7. Reload the project
8. Right click on .ascx file and Run Custom Tool again
9. Voila, the 1 is gone and the file should be now called DisplayWebPart.ascx.g.cs.

So whats the moral of the story? Inconspicuosness is directly proportional to frustration. Like having a valid working reachable Site Url to regenerate a local designer code.

Inconspicuous SharePoint Quirks #6 – Automorphing internal column names

After a brief flirting with Groovy and Grails, its time for a SharePoint leather hunt. I had to add an existing site column “Order” (in the sense of sequencing) to an existing content type “CoolLinks”. Column added fine, and I added some data into the list using that content type.

I looked up the StaticName for that field, because thats what the code uses to query the data.

$s = Get-SPSite http://mysite
$w = $s.openWeb()
$f = $w.Fields["Order"]
$f.Title, $f.StaticName, $f.InternalName

Order, Order1, Order1

The webpart that queried that list, has the following code, using StaticName to query the data:

DataTable dt = web.List["CoolLinks"].Items.GetDataTable();
List<CoolLink> items = new List<CoolLink>();
items = (from item in dt.AsEnumerable()
select new CoolLink() {
LinkUrl = new PublishingLink(item.Lookup<string>("URL")),
Order = Convert.ToInt32(item.Lookup<double>("Order1")),
}
).ToList();

Yet my webpart is not retrieving the list data. I checked the ULS Logs and it has an error –

"Cannot find column 'Order1'"

I also had another List called Alerts, with another content type, which also used the same site column “Order”. The query to retrieve that data was exactly same as that of CoolLinks. Yet Alerts worked fine, but not CoolLinks.

So I startup Powershell, hoping to get a peek into the internal structure of the list, since the content types looked okay.

$s = Get-SPSite http://mysite
$w = $s.openWeb()
$list = $w.Lists["CoolLinks"]
$list.Items.GetDataTable()

This displays the data as it is queried. And what do I find —

URL              : http://somesite
Order0           : 1
ID               : 43
ContentType      : CoolLinks
....

Order0? Where did the 0 come from? The SharePoint API quietly changed the name of the column! I added the Order site column to another content type, it is still Order0. So now I have some Lists in my site collection which use Order1 and some which use Order0, though they are from the same site column.

I ran the .Net Reflector on Microsoft.SharePoint assembly, SPFieldCollection class. The code does check an internal name dictionary in the GetFieldByInternalName() method, which probably keeps a list of “keyword-type” column names and apparently appends arbitrary numbers to it.

Well, that’s the story. So if your Visual Studio goes on the fritz, or your code blows up, or your site collection conks out; before you call the Microsoft Support, turn on all the lights, check all the closets and cupboards, look under all the beds, ’cause you never can tell there just might be a different internal field name in your Content Type.

(Okay, you know where that classic line is from).

Inconspicuous SharePoint Quirks #5 – The Case of the Client Context

Cannot contact site at specified URL

The biggest advantage of SharePoint is probably the lack of useful official documentation. Thanks to it, numerous blogs and forums propose solutions and answers to more numerous problems. Some of them work, several don’t, as problem domain is always vastly huge when compared to the answer domain. For example, a simple combination of misbehaving USB driver + a command window open in non-Admin mode + an IIS session stored in-proc + neighbor’s dog’s wheezing bark + Saturn in conjunction with Mars in Aries constellation – all these could combinedly error an API out in SharePoint. And obviously no forum, let alone technical support, would be able to provide an answer to that. The programmer has to use the final weapon – perseverance.

So I was trying to use the ClientContext to get some data out of my sharepoint site, which is in a VM. Created a simple console project and the following code to get data from a site:

public Web GetRootWeb() {
Web rootWeb;
Uri uri = new Uri("http://sp:2010");
using (ClientContext clientContext = new ClientContext(uri) {
clientContext.Credentials = CredentialCache.DefaultNetworkCredentials;
clientContext.Load(clientContext.Web);
clientContext.ExecuteQuery();
rootWeb = clientContext.Web;
}
return rootWeb;
}

No matter what variations I tried – credentials, permissions, I always got the following error: “Cannot contact site at specified URL http://sp:2010&#8221;. Luckily I had another web application in the same server. And that worked. So something must be wrong with this sharepoint site. This site was a migration from 2007, so first I thought it could be a possibility.

Step 1: Inspect the closed-source code
The ClientRequestException was being thrown from the ClientContext.EnsureFormDigest() method. Having worked with open source for several years gives that irritable itch of peeping into others code. So I fired up .Net Reflector and inspected the method:

The first thing the method does is get an asmx file – its actually the _vti_bin/sites.asmx as the following method shows:

So does this service work against a soap query? Also notice the 2nd and 3rd highlights. Both conditions (actually 3) throw the same exception and error message. So which one failed here? Was a non 200 http code returned? Was the responseContentType not text/xml? Or was m_formDigestInfo null? Pretty much written by a programmer who does not know the fundamentals of debugging. And thats probably why Microsoft’s code is closed, so strangers like me cant code review it. As a best practice, error conditions and error messages should have a monogamous relationship or else a dna test is required to find out who is whose child.

Step 2: Rinse with a different agent – Use “Soap” UI
I fire up soap-UI 4.0 and add the wsdl. Since the site is behind NTLM Windows authentication, a challenge request pops up, I enter the credentials for the site (in this case the farmadmin account) and nope – it does not work. I get a 302 Found error and thats followed by a 401 Unauthorized in the same trace. Well 401 is not what I got before. I try the same with the second web app but that works fine again.

Step 3: Burp it out
After turning on both the http log and error log in the soapUI, I conclude that the credentials may not be passed correctly. After reading this thread, I fire up Burp Suite, goto Options tab, check the “do www authenticate” option, add the servers and the user name/password/domain combination, turn the interceptor off. Well, the 401 is not there anymore, because the BurpSuite makes sure that the header contains the username/password. But the 302 Found still happens.

Step 4: Always exchange dirty notes for good notes in the bank
Obviously IIS is doing some thing here. I replaced the sp:2010’s web.config with the second webapp’s web.config in-toto, recycle app pool and try to hit with soapUI. Voila, Im getting a successful connection now. So some line in the web.config is the culprit. Next I reverted the web.config and started aping section-by-section from the working web.config and I finally come to this line:

<!-- <sessionState mode="SQLServer" timeout="60" allowCustomSqlDatabase="true" sqlConnectionString="Data Source=db;Initial Catalog=SessionStateService_f84b37c3424f46afa09cdacd278a95fa;Integrated Security=True;Enlist=False;Connect Timeout=15" /> --?
<sessionState mode="InProc" cookieless="AutoDetect" timeout="20" />

This section is from the trouble-some site’s web.config. I uncomment the first line, comment the InProc mode line, recycle app pool and whoa! I get a successful connection. So the sessionState set to InProc mode is somehow influencing the soap query which is what the EnsureFormDigest uses and throws an ambiguous ClientRequestException.

I don’t know why a seemingly unrelated sessionState causes an issue with soap query when trying to connect via console application. Thats probably a reflection for another Christmas day. Atleast it works now from Console. I had to make a slight modification while trying to do the same from a web application. From console, the DefaultNetworkCredentials works fine, but from a web app, you have to use the actual site credentials to connect to the site.

The Title is a tribute to Perry Mason novels.

Inconspicuous SharePoint Quirks #4

Mystery of the Identity Impersonator

This one is not exactly a SharePoint quirk, but a SharePoint app led me to it. Which means its an IIS quirk. Or a .Net quirk. Or a Service Pack quirk. Or a Windows Server 2008 quirk. Or a Microsoft quirk. Or whatever…

My co-worker had written a form-based-authentication login page for a SharePoint application. The users are created in Active Directory and the usual frills of user account – login, security questions, change password, reset password et al.

The app worked fine in dev environment, was working fine in Stage but one fine Monday, broke in Stage. The user was created in the AD, but was set in a disabled state and returned an error back to the application. It was working in Dev fine though. The web.config, wsp solution, content databases are all exactly same between dev and stage. The only difference was the stage environment had Windows Server 2008 R2 SP1, while the Dev was not. We decided to backout SP1 from stage and lo, it started working again. Could it be the sheer gravitons of SP1 cause to distort the user creation in the Active Directory?

Since SP1 was causing the problem, we got the dev environment to the same SP1 patch level as stage. And yes, the user creation broke. Repeatable bugs are like flights crashing without casualties. Yeah, it crashed, but no one died. I started looking at the ULS and noticed there were no log statements. Coming from the Java world, I rely more on debug statements rather than run-time debugging. A friend of mine reminds me that in the .Net world programmers first put the solution in Debug mode, then start writing the code. But here the log statements were there in the code, yet were not written.

You can spot the issue here:

public static class ExceptionExtension {
public static string Unroll(this Exception ex) {
if (ex == null) return "";
string exStr = ex.Message + "\n" + ex.StackTrace;

if (ex.InnerException != null) {
exStr += "\n" + ex.InnerException.Unroll();
}

return exStr;
}

Test Usage:
PortalLog.LogString("ERROR: {0} || [{1}]", errorMessage, ExceptionExtension.Unroll(ex));

The above code wont work, because the correct usage of the extension is ex.Unroll(), not by a static method call. Once I fixed this, I could see the actual errors in ULS.

Here is how the account was being created usig LDAP:

entry = new DirectoryEntry(ldapPath, ldapUser, ldapPassword, AuthenticationTypes.Secure);
//create new user object and write into AD
user = entry.Children.Add("CN=" + samAccountName, "user");
user.Properties["userprincipalname"].Add(emailAddress);
user.Properties["samaccountname"].Add(samAccountName);
//...
user.CommitChanges();
user.Invoke("SetPassword", password);
user.Invoke("Put", new object[] { "userAccountControl", "512" });

System.DirectoryServices threw exception with an error code 0x80070005 (ACCESS_DENIED) while invoking SetPassword. I double checked that the ldapUser has identical permissions in dev and stage. How can the user be created but still access denied? More importantly, WHO is being denied? Certainly not the ldapUser.

I added the debug for WindowsIdentity.GetCurrent().GetName(). And it returned NT AUTHORITY\IUSR. In IIS, the anonymous access is enabled and in the web.config the identity tag is

<identity impersonator="true"/>

Obviously IUSR does not have permission to set the password. But why was it working in dev, before SP1? When I tried in another environment without SP1, the current Windows Identity was the same as the Application Pool user for that SharePoint Web Application and not the IUSR. Google-o! Er.. I mean, Bing-o!

Back to dev with SP1, I set the impersonator to false, and now I saw the current identity as the Application Pool user! W2k8 R2 without SP1 was not respecting the identity impersonator flag. Obviously without SP1, the IUSR anonymous user was having access to create an user in active directory, while SP1 prevented it. The impersonation is not required when you are creating user using the ldap service user and password, who already have access.

To me it looks like a security hole (pre-SP1). I could not find any online documentation or release notes mentioning about this. If you know of such documentation, please drop a comment!

Inconspicuous SharePoint Quirks #3

Mystery of the missing Content Type column

Is there something called “Over-architecting a solution”? Or do some people get carried away too much by Object Oriented Concepts that a simple if-else condition is enshrined into a dazzling hierarchy of classes, interfaces and er.. content types? SharePoint’s answer to creating a class/type or pseudo-interface is the Content Type. While a powerful feature, there are times even the Microsoft Team gets confused what to do with it, unconsciously bury a feature and consciously let it go undocumented.

There was a requirement to show different banners with different styles based on items in the “Banner List”. The different banner types were represented as “Content Types” and so the end user will create Banner Type 1, Banner Type 2 etc; set it to active and it would appear on the web page. The solution was designed in such a way that a calculated column in that list was populated based on the content type of the banner. That is, the calculated column had a formula in the BannerText sitecolumn:

=IF [Content Type] == "Banner Type 1" Then BannerText = "Red Banner" ELSE IF [Content Type] == "Banner Type 2" Then BannerText = "Blue Banner" ELSE IF...

The site column was relying on the Content Type of the list, that will be known at runtime. And that was working perfectly fine. In SharePoint 2007, that is.

Over to 2010. When I was trying to add a new item in that list, the calculated column Text would populate with a value “#NAME?” indicating that there is something wrong with the formula. Going back the site column, initially said the formula is fine, but when I tried to update it, errored out saying invalid column. Upon inspection, I see that Content Type is missing in the “Insert Columns” list.

First I thought my migration was wrong. Looked back at the SharePoint 2007 implementation, the “Content Type” column is there (see above image from MOSS 2007)! So Microsoft Team couldn’t handle the power the Content Types and removed it, I guess. But why was it removed? (see image below, from SharePoint 2010)

The _layouts/fldnew.aspx page inherits from Microsoft.SharePoint.ApplicationPages.BasicFieldEditPage. The method that creates the Insert Columns is WriteFieldNamesForFormulaColumnPicker(), so I loaded the .Net Reflector and inspected the code.

protected internal static void WriteFieldNamesForFormulaColumnPicker(TextWriter output, SPFieldCollection fields, string currentDisplayName)
 {
 SortedList list = new SortedList(new Comparer(fields.Web.Locale), fields.Count);
 foreach (SPField field in fields)
 {
 if ((((!(field is SPFieldText) && !(field is SPFieldCalculated)) && (!(field is SPFieldNumber) && !(field is SPFieldDateTime))) && (!(field is SPFieldChoice) || (field is SPFieldWorkflowStatus))) && (!(field is SPFieldBoolean) || (field is SPFieldAllDayEvent)))
 {
 continue;
 }
 if (((field.InternalName != "_UIVersionString") && !field.Hidden) && (field.Title != currentDisplayName))
 {
 list.Add(field.Title + "_" + field.InternalName, field);
 }
 }
 bool flag = true;
 foreach (SPField field2 in list.Values)
 {
 string title = field2.Title;
 SPHttpUtility.NoEncode("<option value=\"[", output);
 SPHttpUtility.HtmlEncode(title, output);
 SPHttpUtility.NoEncode("]\"", output);
 if (flag)
 {
 SPHttpUtility.NoEncode(" selected=\"selected\"", output);
 flag = false;
 }
 SPHttpUtility.NoEncode(">", output);
 SPHttpUtility.HtmlEncode((title.Length > 20) ? (title.Substring(0, 20) + "...") : title, output);
 SPHttpUtility.NoEncode("</option>", output);
 }
 }

So if the field is only one of text, calculated, number, datetime, choice, workflowstatus, boolean it is available for formulas. So what type is the Content Type column anyway? Using Powershell, we can determine that —

PS C:\Users\spfarmadmin> $w.Fields["Content Type"]

EnableLookup                : False
FieldRenderingControl       : Microsoft.SharePoint.WebControls.ComputedField
FieldRenderingMobileControl : Microsoft.SharePoint.MobileControls.SPMobileComputedField
FieldTypeDefinition         : Microsoft.SharePoint.SPFieldTypeDefinition
TypeDisplayName             : Computed

So since the field is Computed type, it does not appear in the insert column list. The same code in SharePoint 2007 didn’t have this restriction. So there is no way to find content type from within a site column. So I had to redesign the list now. Instead of multiple content types, I just added a new dropdown column which contain a list of values which was exact same as the content type. That way the functionality still worked without changing the underlying custom code. So a list with several content types and calculated columns to determine a text could be replaced with a simple dropdown list! Instead of multiple content types and formulas.

But a fundamental feature was removed. May be the next release of SharePoint 2014 will not have content databases?

Inconspicuous SharePoint Quirks #2

Mystery of the vanishing Publishing Hyperlink

Note: Publishing Hyperlink can be created as a site column “Hyperlink with formatting constraints”.

There seems to be a bug in Publishing Hyperlink field in SharePoint 2010 and it does not happen consistently. Lets say there is a List with Publishing Hyperlink site column. When you edit this list item, and focus on the url, a new “Link Tools” (green tab) appears in the editor. This brings additional properties to edit.

If you change the link value here, change and save, sometimes the link value disappears. Or if you click on Remove Link button, the “click here to add new link” disappears. I was able to verify this consistently in a totally clean site (SharePoint Server Build  14.0.5128.5000). This means you cannot add a link at all anymore. Your only option is to delete the item and add it again. Goodluck if the item is huge and/or has some relation to other lists!

Fortunately there is powershell, you can update the link directly. The trick is, while the regular Hyperlink is a string, the Publishing Hyperlink is an object. So you have to update the object properties first and set the object itself to the listitem and then update the list item. The following powershell code shows that:

Param([string]$site0="http://mysharepoint", [string]$list0="My List", [string]$itemKey0="Item Key", [string]$linkUrl0="http://newlink", [string]$desc0="New description", [string]$toolTip0="New Tooltip")

$site = Get-SPSite($site0)
$rootWeb = $site.rootWeb
$lists = $rootWeb.Lists
$list = $lists[$list0]
$items = $list.Items
$listItem = $items | Where-Object {$_.Name -eq $itemKey0}
$linkUrl = $listItem["Link Url"]
$linkUrl.NavigateUrl = $linkUrl0
$linkUrl.Description = $desc0
$linkUrl.ToolTip = $toolTip0
# this is the trick - set the object back to listItem
$listItem["Link Url"] = $linkUrl
$listItem.Update()

Inconspicuous SharePoint Quirks #1

Mystery of the Unimportable WebPart

So Voltaire said “God protect me from my friends, I will take care of my enemies”. If there was a SharePoint God, I would probably pray to him “Protect me from the inconspicuous quirks, I will blogoogle the rest”.

So I have a wsp with a webparts feature. After deploying a wsp to the dev server, I added a webpart to a page which showed me a charming message “Cannot import webpart”. Call me schizophrenic but a voice inside me told that dont waste time at the logs, they are not going to help. This was working fine in my local vm, but trouble-some in dev server. So what’s wrong? After a miss-adventrous chasing of dead-ends, cul-de-sacs, and no-outlets finally figured a method that works consistently.

1. First of all, ensure that the elements.xml, SharePointProjectItem.spdata, the .webpart file, feature.xml, package.xml are all correct. Make sure the Group is correct too, or else your users will not find the expected webpart.
2. If your webpart is extending from Microsoft.SharePoint.WebPartPages.WebPart, you must use the wss/v2 schema, ie the webpart should have a .dwp extension. The newer .webpart extension/v3 schema works only for ASP.Net webparts
3. After deploying the wsp, check if the web.config has the correct SafeControl entry with correct version of assemblies. Also check the GAC for correct version.
4. Goto http://server/_catalogs/wp/Forms/AllItems.aspx. This lists all the webparts in Web Part Gallery. See the Modified Time, is it the latest? If not, delete all the webparts. (You can use a simple powershell script for that).
5. Now deactivate the webpart feature (Disable-SPFeature featureName)
6. Again activate the webpart feature (Enable-SPFeature featureName)
7. Check the Web Part Gallery, now all the webparts should be added.
8. Try adding the webpart again and if it still says “Cannot import webpart”, pray to SPGod.

So why was it working on my local vm? Oh yeah. Turns that there is a bug in Visual Studio 2010 Deploy command which would not Activate the features automatically. So some time ago I had set it to “No-Activation” deployment configuration and I was activating the features via powershell. That powershell activation was taking for adding the webparts correctly to the web part gallery. On the dev server, I was assuming that the install-spsolution automatically refreshes the web part gallery too, but it does not.

It is well blogumented that webparts are not removed from the gallery upon uninstall-spsolution, but now I know it does not update the webparts  either using install-spsolution, yet the feature is blissfully activated. Cunningly symmetrically logical eh?