Debugging Challenge #6

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

@Echo off

set direct=1
echo %cmdcmdline% | find /i "%~n0" >nul
if not %errorlevel% == 1 set direct=0

REM powershell.exe -file .\AdHocScripts\FlushBlobCache.ps1
Set e=%errorlevel%
if not %e% == 0 goto error

powershell.exe -Version 2 -file .\AdHocScripts\FarmIISReset.ps1
Set e=%errorlevel%
if not %e% == 0 goto error

goto success

:error
    Echo ERROR, errorlevel %e%
    if %direct% == 0 pause else exit %e%
    goto end
:success
    Echo done
    if %direct% == 0 pause
:end

Debugging Challenge #5

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

// @param date  string in the format YYYY-MM-DD
function isBeforeToday(date: string) {
  return new Date(date).getTime() 
      < Math.floor(Date.now() / 86400000) * 86400000;
}

Debugging Challenge #4

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

let csv_split seperator str =
  let csvSplit = new Regex(
    "((?:\")[^\"]*(?:\"(?=,|$)+)|(?<=,|^)[^,\"]*(?=,|$))", 
    RegexOptions.Compiled) in
  csvSplit.Matches(str)
    .OfType<Match>()
    .Select(fun m -> m.Value.TrimStart(','))
    .ToArray()

Debugging Challenge #3

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

int index = addressLine.Length;
for (int i = 0; i < 10; i++)
{
    int position = addressLine.IndexOf(i.ToString());
    if (position != -1 && position < index)
        index = position;
}
string street = addressLine.Substring(0, index).Trim();
string nr = addressLine
            .Substring(index, addressLine.Length - index)
            .Trim();

Debugging Challenge #2

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

export class ActivityService {
    activities : Activity[] = [];
    constructor(){
        this.activities = [
            {
                title: "Arrange meeting",
                type: "task"
            }
        ];
        setTimeout(function () {
            this.activities.push({ 
                title: "Meeting",
                type: "appointment"
            });
        }, 3000); 
    };
}

Debugging Challenge #1

Previously on Dr. Lambda's blog:

I have deviced a series of debugging challenges, some are easy, some are really hard, all come from real live systems. Good luck!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

private static long IncrementRowVersion(long rowversion)
{
    return rowversion++;
}

Bug Hunt: Debugging Challenge #0

So, I really like puzzles, and challenges. Just like the Javascript Drinking Game. I also used to post programming challenges online, most were extremely challenging, and very time consuming. Therefore I have long been thinking how to make exercises that are both challenging, relevant, and light.

My idea was some sort of debugging challenge. As a developer I spend most of my time doing this anyway, so I figured my colleagues must too, and it would be great practice. Because we are living in a culture were attention span is getting shorter and shorter I knew that the entire thing has to fit on a page, the description should be bulleted, and the code cannot be more than a handful of lines. If any of these fail most people wont give it a second glance.

I decided on a simple point system, starting with a pretty trivial task – to hook the reader. Just identify what looks wrong. Then draw them in by asking why they said that, and finally the real challenge: How to fix it. The Bug Hunt has a start date and an end date, and everybody collects points all season. At the end the victor can have his pick of which project to work on.

After spending a year keeping an eye out, I finally feel like I have collected enough material for the first "season". All the code comes from actual live systems that I have come in contact with as a consultant. They are obviously annonymized, but the structure of the code, and more importantly the bugs were real. As such, the challenges have very different flavor, some are included because they are very difficult, others are deceptively easy. They span different programming languages and technologies, however, in most cases this context must be inferred.

I highly recommend that you share challenges that you like (I print and hang them on the walls in the office), you might even use them to set up a Bug Hunt at your own office. If so, I would be happy to provide the answers.

Let the 0th Bug Hunt begin, and a good hunt to all participants!

The Challenge

• 1 point if you can spot where the error is.
• +1 point if you can explain why.
• +2 points if you can explain how to fix it.

SELECT a.* FROM Notification a
LEFT OUTER JOIN (
    SELECT CONVERT(uniqueidentifier, 
                MIN(CONVERT(char(36), b.[ID]))) AS ID
            , CAST(b.[Message] AS varchar(255)) AS Message_varchar
            , b.[To]
    FROM Notification b
    GROUP BY Message_varchar, b.[To]
) AS KeepRows ON
a.[ID] = KeepRows.[ID]
WHERE a.[Sent] = 0
AND KeepRows.[ID] IS NULL

Import-ant Tool

Motivation

Importing data into Microsoft Dynamics CRM is convenient; however, the data we have to import is usually not perfect, therefore, we often end up importing and deleting the same data many times before we are satisfied. Both of these processes are quite slow, and does require a bit of manual work. Another caveat of this method is that we cannot start using the data until it is imported correctly, because relations to it will be broken.

In this post, we implement a small F# script to import data into Microsoft Dynamics CRM. However, if the data is already in the CRM system the script corrects the existing record with the new data, thereby preserving relations already in place. This means, we can start using our data immediately, without fear of loss.

Remember: this is a technical post; a usage/tutorial post will follow next week.

Prerequicits

For once we are not working with XML, thus we only need:

• F#, TypeProviders (in particular the XrmDataProvider)
• Microsoft Dynamics CRM
• And our own FetchXML library.

Utilities

Apart from the FetchXml library we made last week, we only need this simple function:

let rec assoc key map = 
  match map with
    | [] -> None
    | (k, v) :: map' ->
      if k = key
      then Some v
      else assoc key map'

The Library

Our goal is to load data from a CSV-file (comma-separated values) into CRM.

It is straightforward to load in strings. However, it is a different story for option sets or lookups. Option sets are easier if we have them as the values, as opposed to the labels. Although, this makes it much less readable, and harder to work with.

Another goal is to keep the data as human readable as possible. Readable data is easier to work with, and less error prone. It also allows less technical users to work with the data, e.g. in Excel.

Because we also want the library to be able to update records, which are already in CRM, therefore we need some way to specify which records to update.

My favorite solution to these problems is to use sophisticated headers. Our headers should specify both what type of data it is, and supply additional information necessary to load the data. The headers should also specify which attributes uniquely define a record; this could be an account number, or an email address. We call these fields primary.

Initialization

There are many types of data in CRM. In this prototype, we only address a few very common types; strings, booleans, local option sets, global option sets, and lookups.

For simplicity, I have decided that you can only set lookups based on one attribute of the target entity. We also represent both local and global option sets simply as a mapping from labels to values. We also allow columns to be ignored.

type mapping =
  | Simple
  | Ignore
  | OptionSet of (string * int) list
  | Lookup of string * string
type field =
  { name : string;
    typ : mapping;
    primary : bool; }

In order to initialize the mapping we need to define some syntax:

• If a field starts with # the column will be ignored
• Otherwise the header should be the logical name of the attribute in CRM
• If the name is followed by a * this column is primary
• If the name (and possibly *) is followed by only a :, then the field is the label of a local option set. (Example (Account): paymenttermscode:)
• If the name (and possibly *) is followed by only a : and the logical name of a global option set, then the field is the label of that global option set. (Example (Opportunity): budgetstatus:budgetstatus)
• If the name (and possibly *) is followed by only a : and the logical name of an entity, and a . and the logical name of an attribute of the entity, then the field should look up a record of that entity based that attribute. (Example (Account): primarycontactid:contact.emailaddress1)

let init_mapping ent (s : string) =
  s.Split [|';'|]
  |> Array.map (fun f ->
    let f = f.Trim () in
    let f = f.Split(':') in
    let primary = f.[0].EndsWith("*") in
    let name = 
      if primary
      then f.[0].Substring(0, f.[0].Length - 1)
      else f.[0] in
    if name.StartsWith("#")
    then { name = name; typ = Ignore; primary = primary }
    else if f.Length = 1
    then { name = name; typ = Simple; primary = primary }
    else 
      let f = f.[1].Split('.') in
      if f.Length = 1
      then 
        let options =
          if f.[0].Length = 0
          then 
            let req = RetrieveAttributeRequest () in
            req.EntityLogicalName <- ent;
            req.LogicalName <- name;
            let resp = xrm.OrganizationService.Execute req :?> RetrieveAttributeResponse in
            let pList = resp.AttributeMetadata :?> PicklistAttributeMetadata in
            pList.OptionSet.Options
            |> Seq.map (fun o -> (o.Label.UserLocalizedLabel.Label, o.Value.Value))
            |> Seq.toList 
          else
            let req = RetrieveOptionSetRequest () in
            req.Name <- f.[0];
            let resp = xrm.OrganizationService.Execute req :?> RetrieveOptionSetResponse in
            let oSetMeta = resp.OptionSetMetadata :?> OptionSetMetadata in
            oSetMeta.Options
            |> Seq.map (fun o -> (o.Label.UserLocalizedLabel.Label, o.Value.Value))
            |> Seq.toList in
        { name = name; typ = OptionSet options; primary = primary }
      else
        let entity = f.[0] in
        let field = f.[1] in
        { name = name; typ = Lookup (entity, field); primary = primary }
    )

Setting Data

Now setting a value based on the mapping and a string is straightforward.

Notice: lookup is from last week’s post.

let set_value (e : Entity) m (v : string) =
  if v.Length = 0
  then ()
  else
    match m.typ with
      | Ignore -> ()
      | Simple -> 
        if v = "true" then
          e.Attributes.Add(m.name, true)
        else if v = "false" then
          e.Attributes.Add(m.name, false)
        else
          e.Attributes.Add(m.name, v)
      | OptionSet map -> 
        match assoc v map with
          | None -> printfn "Warning: %s not found in optionSet" v; ()
          | Some i -> e.Attributes.Add(m.name, OptionSetValue i)
      | Lookup (ent, f) ->
        e.Attributes.Add(m.name, EntityReference (ent, lookup ent f v))
  

Getting Data

As mentioned above we sometimes want to update existing records instead of creating new ones. The way we determine this is if a record exist which is equal on all the primary fields to the one we would create, then we update this one instead. To test this we again use our FetchXml library.

Notice: we require that the record be unique. If we did not do this, we could not guarantee that we consistently update the same record.

let get_entity ent ms vals =
  let fxml = 
    (ms, vals)
    ||> Array.fold2 (fun st m v -> 
      if not m.primary
      then st
      else 
        match m.typ with
          | Ignore -> st
          | Simple -> FetchXml.add_condition m.name (FetchXml.Equals v) st
          | OptionSet os -> FetchXml.add_condition m.name (FetchXml.Equals (string (assoc v os).Value)) st
          | Lookup (ent, f) -> FetchXml.add_condition m.name (FetchXml.Equals ((lookup ent f v).ToString("B"))) st
      ) (FetchXml.init (Crm.Entities.unsafe ent) |> FetchXml.set_count 2)
    |> FetchXml.generate in
  let ec = xrm.OrganizationService.RetrieveMultiple (FetchExpression fxml) in
  let e = Entity () in
  e.LogicalName <- ent;
  if ec.Entities.Count = 0 then
    e
  else if ec.Entities.Count = 1 then
    e.Id <- (ec.Item 0).Id;
    e
  else
    printfn "Fxml: %s" fxml;
    failwithf "Lookup failed: multiple %s records found." ent

Importing/Repairing Data

With these helpers, the main function simply loads the file, line-by-line, and calls the helpers. Currently the library assumes the default export format for CSV files from Excel, which means un-quoted strings separated by semicolons.

let action act (e : Entity) =
  if e.Attributes.Contains("name")
  then printfn "%s: %s" act (string (e.Attributes.["name"]))
  else printfn "%s entity" act

let import_data entity =
  let lines = File.ReadAllLines (cfg.rootFolder + "/" + entity + ".csv") in
  let mapping = init_mapping entity lines.[0] in
  lines
  |> Array.skip 1
  |> Array.iter (fun valS ->
    try
      let vals = valS.Split [|';'|] |> Array.map (fun v -> v.Trim ()) in
      if vals.Length <> mapping.Length
      then failwithf "Not enough values"
      else
        let e = get_entity entity mapping vals in
        (mapping, vals)
        ||> Array.iter2 (set_value e);
        if e.Id = System.Guid.Empty
        then action "Created" e; xrm.OrganizationService.Create e |> ignore
        else action "Updated" e; xrm.OrganizationService.Update e
    with
      e -> printfn "Error: %s; %s" (e.Message) valS
    )

Quality Control

As I am devoted to high quality software, we should take a step back and examine how solid this library is.

First, this library is only as good as the data it gets. We can feed it destructive data, either because it overwrites good data, or contains invalid references, or something else. In practice though, bad data is most commonly rejected by CRM, in which case the tool skips the line, prints the error, and the faulty data. Similar to the built in import tool in CRM.

Another potential problem is if the data contains semicolons, in which case the input line is split incorrectly. However, there is no way to protect against this with the format we chose. If this happens, the line will be skipped, and printed.

Because of the primary fields, the tool never creates duplicates. This also means that we can run it multiple times without any additional risks. Further, because any error is printed, along with the triggering data, we get an easy overview over what has been skipped, and usually why. This means that we can easily find the flaw, correct it, and run the import tool again.

Playing Fetch with XML

Motivation

Importing data into Microsoft Dynamics CRM is pretty convinient, however the data we have to import is usually not perfect, therefore we often end up importing and deleting the same data many times before we are satisfied. Both of these processes are quite slow, and does require a bit of manual work. This method also means that we can't start using the the data until it is imported correctly because relations to it will be broken. Even in the situations where the data is perfect, like when we move data from a development or testing system to a production system, we first have to export it, manually change the format, and manually import it again.

Our next project is to make data import easier. We cannot cover everything within this area, so this is a running project, which we will develope and expand over time.

In this post, we make the foundation for a neat and minimal library for querying the CRM system. The most common way for users to query the system is using the Advanced Search functionality. Advanced Search provides an interface where we can choose which attributes to display and set up conditions on which records to retrive. Behind the scenes this is represented in as XML more specifically FetchXML. FetchXML is the way we are going to query the system. XML is not the nicest interface for humans, thus we are going to abstract away the syntax and start introducing types.

Remember: this is a technical post, a usage/tutorial post will follow next week.

Prerequicits

In order to build an F# library to abstract away XML we need to be familiar with the following concepts:

• XML (as a reference CRM's FetchXML)
• F#, TypeProviders (in particular the XrmDataProvider)
• and of course Microsoft Dynamics CRM.

Utilities

For this library, we don't need alot of new utilities. Actually, we only need one new function:

module StringUtils =
  let capitalize_first (str : string) = 
    string (System.Char.ToUpper str.[0]) + str.Substring(1)

Unsafe FetchXml

In the first version of the library we focus only on the basic functionality. Afterwards we consider how to improve usability by introducing types to help catch bugs. The library consists of five functions for: initialization, setting how many records to retrieve, setting up conditions on the lookup, choosing which attributes to retrieve, and finally generating the finished FetchXML.

As mentioned earlier on this blog: I like chain-calling. So like last time we have a value – the 'needle' – that is 'threaded' through all the calls. This needle is the last argument to every function, and every function returns it, or a variation of it.

Initialization

For now, we are only interested in basic functionality we only need to store:

• which entity we want to retrieve from
• how many records to retrieve
• which attributes to retrieve
• and the conditions on the records.

The final value is a string representing XML, therefore many of these can be represented as strings. However, for the conditions there are some advantages to using a custom datatype. First, it is difficult for users to remember what conditions are possible, and how to write them – their format and keyword. If the conditions are represented by a custom datatype these problems are solved by code completion and custom code, respectively. Second, if we ever want to extend the library, if the conditions were strings we might have to parse them, which we would rather not.

Note: I have only chosen a few central types of condition, but it should be easily extended with more types.

module FetchXml =
  type condition =
    | Equals of string
    | In of string list
    | ContainsData

  type fetchxml =
    { entity : string;
      count : int option;
      attributes : string list;
      conditions : (string * condition) list }

  let init ent =
    { entity = ent;
      count = None;
      attributes = [];
      conditions = []; }

By default, we retrieve all attributes and all records.

Setting how many Records to Retrive

Limiting how many records to retrieve is as easy as updating count:

  let set_count count fxml =
    { fxml with count = Some count }

Chosing which Attributes to Retrive

Similarly, if we want to limit which attributes to retrieve we simply add them, one at a time:

  let add_attribute attr fxml =
    { fxml with attributes = attr :: fxml.attributes }

Setting up Conditions on the Lookup

Finally, to limit which records to retrieve we simply add attribute-conditions pairs, one at a time:

  let add_condition attr cond fxml =
    { fxml with conditions = (attr, cond) :: fxml.conditions }

Generating the Finished FetchXML

Having a fetchxml-record it is straightforward to generate the XML. One thing to note is that we cannot use amporsant in values in FetchXML.

  let generate fxml =
    let e = fxml.entity in
    "<fetch mapping=\"logical\"" + 
    (match fxml.count with
      | None -> ""
      | Some c -> " count=\"" + string c + "\"") +
    " version=\"1.0\">" + 
    "<entity name=\"" + e + "\">" +
    List.foldBack (fun a acc -> "<attribute name=\"" + a + "\" />" + acc) fxml.attributes "" +
    "<filter type=\"and\">" +
    List.foldBack (fun (a, c) acc -> 
      match c with
        | Equals s ->
          "<condition attribute=\"" + a + "\" operator=\"eq\" value=\"" + s.Replace("&", "&") + "\" />" + acc
        | In vs ->
          "<condition attribute=\"" + a + "\" operator=\"in\">" +
          List.foldBack (fun v acc -> "<value>" + v + "</value>" + acc) vs "" +
          "</condition>" + acc
        | ContainsData ->
          "<condition attribute=\"" + a + "\" operator=\"not-null\" />" + acc) fxml.conditions "" +
    "</filter>" +
    "</entity>" +
    "</fetch>"

This concludes the core library. This is fine if we want to use it with other code, but people are not great with strings. People make spelling mistakes, forget which entities have which attributes, or even more subtly, forget to correct code when it changes.

Invent datatypes

One way to improve these problems is to use types. We need a way to connect entities and attributes with types, but where all entities still have 'similar' types. For this, we use polymorphism. Here is a toy example of how this would look:

module Attribute =
  module systemuser =
    type attribute = Name | Fullname
    let string_of_attribute = function
      | Name -> "name"
      | Fullname -> "fullname"

module Entity =
  type 'a entity = private { logical_name : string; string_of : 'a -> string }
  let string_of_entity e = e.logical_name
  let string_of_attribute e a = e.string_of a

  let systemuser = 
    { logical_name = "systemuser"; 
      string_of = Attribute.systemuser.string_of_attribute }

Notice that because the entity record is private we cannot accidentally make an invalid entity.

Generate datatypes

Writing the datatypes for all attributes and entities is unmanageable for people. However, even if we could, it would be static so we would need to update it every time something changed in the CRM system. We need a way to generate it:

let meta = 
  let er = RetrieveAllEntitiesRequest () in
  er.EntityFilters <- EntityFilters.Attributes;
  xrm.OrganizationService.Execute(er) :?> RetrieveAllEntitiesResponse in
meta.EntityMetadata
|> Array.fold (fun (attrs, ents) i -> 
  let attributes = i.Attributes |> Array.fold (fun acc a -> acc + " | " + StringUtils.capitalize_first (a.LogicalName)) "" in
  let string_of = i.Attributes |> Array.fold (fun acc a -> acc + "      | " + StringUtils.capitalize_first (a.LogicalName) + "-> \"" + a.LogicalName + "\"\n") "" in
  (attrs + 
   "  module " + i.LogicalName + " = \n" +
   "    type attribute = " + attributes + "\n" +
   "    let string_of_attribute = function \n" + string_of,
   ents +
   "  let " + i.LogicalName + "= \n" +
   "    { logical_name = \"" + i.LogicalName + "\"; \n" +
   "      string_of = Attributes." + i.LogicalName + ".string_of_attribute } \n")
   ) ("module Attributes = \n", 
      "module Entities = \n" +
      "  type 'a entity = private { logical_name : string; string_of : 'a -> string }\n" +
      "  let string_of_entity e = e.logical_name\n" +
      "  let string_of_attribute e a = e.string_of a\n")
|> fun (attrs, ents) -> 
  File.WriteAllText (cfg.rootFolder + "/Crm.fsx", attrs + ents);

Safe FetchXml

With all these datatypes, we can modify the fetchxml-record to use these instead of strings:

type 'a fetchxml =
  { entity : 'a Crm.Entities.entity;
    count : int option;
    attributes : 'a list;
    conditions : ('a * condition) list }

Of course we also need to change generate accordingly, but that is straightforward.

As mentioned earlier, string actually work better with other code, thus if we still want this option we need to add this "entity-generator" to the Entities-module.

  let unsafe lname = { logical_name = lname; string_of = id }

Quality Control

As this library does not modify any data, we don't need to be as critical of it.

A Note on Performance

My focus is usually more on correctness and aesthetics; for once, we will consider the performance. The problem is that we are generating an 80 kB file. This file needs to be read, parsed, and type checked, which turns out to be very slow.

One solution to this problem is to split up the entities into separate files, but then we have to forgo the advantages of the private record. Namely that we cannot obtain an illegal entity, which is aesthetically less pleasing, but in practice may be the right choice.

Scripting Form Scripts

Motivation

With Microsoft Dynamics CRM we can put client-side scripts on forms, which can be very useful.

Unfortunately no part of the process is pleasant. First there is the JavaScript, which I personally really don't enjoy. Then, on the CRM side, we have to upload the script, go to the form editor, properties, add the dependencies (one at a time mind you), register the call-back, save and publish the form. Finally we can test our script, at which point we discover that we used the wrong name for the call-back or forgot one of your dependencies, not to mention that the JavaScript itself could have bugs, and we have to go through the entire process again.

It may seem like a small issue, after all, once the script is up and running we can skip most of the steps above. But if you manage a lot of CRM systems you may find yourself often spending half an hour to an hour just getting a new script running for the first time, excluding coding it. This is too wasteful and error-prone for my taste, so I decided to automate a lot of it.

Remember: this is a technical post, a usage/tutorial post will follow next week.

Prerequicits

Client-side scripts are considered part of a forms layout. The form layout in CRM is represented as XML, and through the standard API the only way to affect it is by manipulating this XML. Thus, before we begin, the reader is expected to be familiar with:

• XML (as a reference CRM's FormXML)
• JavaScript (or something that compiles to JavaScript)
• F#, TypeProviders (in particular the XrmDataProvider)
• and of course Microsoft Dynamics CRM.

Utilities

Let us start by defining some convinient functions.

I find the functional string library to be missing a few useful functions, in this case only one:

module StringUtils =
  (** Partition a string into two at the last occurance of c from the left *)
  let partition_right (c : char) (s : string) =
    let i = s.LastIndexOf c in
    (s.Substring(0, i), s.Substring(i+1)) 

The XML library we will be using comes from C#, which means it uses lots of side-effects and is (mostly) untyped, neither of which are good in my oppinion, however I'm too pressed for time to write my own XML library. Having said that, I prefer to have these functions:

#r "System.Xml.dll"
open System.Xml

module XmlUtils =
  (** Selects a single node, or creates it if it doesn't exist *)
  let rec select_node (doc : XmlDocument) path =
    let lookup = doc.SelectSingleNode path in
    if lookup <> null
    then lookup
    else 
      let (path', node) = StringUtils.partition_right '/' path in
      let new_node = doc.CreateElement node in
      (select_node doc path').AppendChild(new_node)

  (** Selects a list of nodes, and creates the path to it if the path doesn't exist *)
  let select_nodes (doc : XmlDocument) path = 
    let (path', node) = StringUtils.partition_right '/' path in
    let parent = select_node doc path' in
    parent.SelectNodes(node) |> Seq.cast

  (** Appends a node to a path, created the path if it doesn't exist *)
  let append_node (doc : XmlDocument) path node =
    (select_node doc path).AppendChild(node) |> ignore

  let to_list (xs : XmlNodeList) =
    let rec loop i acc =
      if i < 0
      then acc
      else loop (i - 1) ((xs.Item(i)) :: acc) in
    loop (xs.Count) []

Finally, because I want to support multiple versions of CRM, I need to detect which one we are using, which also means we need to instantiate the CRM TypeProvider:

module VersionUtils =
  type version = private { v : int [] }
  let of_string (s : string) = { v = Array.map int (s.Split('.')) }
  let major v = v.v.[0]

#r "microsoft.xrm.sdk.dll"
open Microsoft.Xrm.Sdk
#r "FSharp.Data.DynamicsCRMProvider.dll"
open FSharp.Data.TypeProviders
#r "FSharp.Data.DynamicsCRMProvider.Runtime.dll"
open FSharp.Data.TypeProviders.XrmProvider.Runtime.Common

(* CRM TypeProvider *)
type XDP = XrmDataProvider<"https://some_domain/XRMServices/2011/Organization.svc"
                , Username="username"
                , Password="password">
let xrm = XDP.GetDataContext()
type XRM = XDP.XrmService

#r "Microsoft.Crm.Sdk.Proxy.dll"
open Microsoft.Crm.Sdk.Messages
let sdk_version = 
  let crmVersion = 
    let req = new RetrieveVersionRequest() in
    let resp = xrm.OrganizationService.Execute(req) :?> RetrieveVersionResponse in
    resp.Version
    |> VersionUtils.of_string
    |> VersionUtils.major in
  match crmVersion with
    (* https://support.microsoft.com/da-dk/lifecycle?p1=15707 *)
    | 5 -> "2011"
    | 6 -> "2013"
    | 7 -> "2015"
    | 8 -> "2016"
    | _ -> failwith "Unsupported CRM version"

And also we need to publish our changes eventually:

let publish () = 
  let req = Microsoft.Crm.Sdk.Messages.PublishAllXmlRequest () in
  xrm.OrganizationService.Execute(req);

The Library

The library itself consists of five functions for: initialization, registering dependencies, registering call-backs, validating the XML, commiting the XML to CRM.

When I started designing this library I had a goal. I like chain-calling, so we should have something, let's call it a 'needle', that can be 'threaded' through all the calls. This means that the needle has to be the last argument, and every function has to return it, or a variation of it.

Initialization

For ease of use we have a record containing all information that the library needs:

• the name of the form
• the name of the entity
• the xml for the form
• and a few references to the form.

The initialization is just filling those fields:

type form_info = 
  { name : string;
    entity : string;
    doc : XmlDocument;
    form_id : System.Guid;
    original_formxml : string }

let init (form : XRM.systemform) =
  let xml = form.formxml in
  let doc = new XmlDocument() in
  doc.LoadXml xml; 
  { name = form.name;
    entity = form.objecttypecode;
    doc = doc;
    form_id = form.Id;
    original_formxml = form.formxml }

Registering Dependencies

For registering something, the only real consideration is wether it is already registered or not. Therefore, the structure of the next two functions is the same: check if the argument is already registered, if it is then skip, otherwise register it.

let register_dependency file fi = 
  let already_registered = 
    XmlUtils.select_nodes fi.doc "/form/formLibraries/Library"
    |> Seq.exists (fun n -> 
      n.Attributes.GetNamedItem("name").Value = file) in
  if already_registered
  then 
    printfn "%s is already registered on %s (%s)" 
      file fi.name fi.entity; 
    fi
  else
    let libNode = fi.doc.CreateElement("Library") in
    libNode.SetAttribute("libraryUniqueId", 
      System.Guid.NewGuid().ToString("B"));
    libNode.SetAttribute("name", file);
    XmlUtils.append_node fi.doc "/form/formLibraries" libNode;
    fi

Registering Call-backs

The structure here is essentially the same, although the XML is slightly more involved.

type event_type = OnLoad | OnSave
let event_type_to_string = function OnLoad -> "onload" | OnSave -> "onsave"
let register_callback t file func fi = 
  register_dependency file fi |> ignore;
  let already_registered = 
    XmlUtils.select_nodes fi.doc "/form/events/event/Handlers/Handler"
    |> Seq.exists (fun n -> 
      n.Attributes.GetNamedItem("functionName").Value = func &&
      n.Attributes.GetNamedItem("libraryName").Value = file) in
  if already_registered
  then 
    printfn "%s (%s) is already registered on %s (%s)" 
      func file fi.name fi.entity; 
    fi
  else
    let handlerNode = fi.doc.CreateElement "Handler" in
    handlerNode.SetAttribute("handlerUniqueId", 
      System.Guid.NewGuid().ToString("B"));
    handlerNode.SetAttribute("functionName", func);
    handlerNode.SetAttribute("libraryName", file);
    handlerNode.SetAttribute("enabled", "true");
    handlerNode.SetAttribute("passExecutionContext", "false");
    handlerNode.SetAttribute("parameters", "");
    let handlersNode = fi.doc.CreateElement "Handlers" in
    let eventNode = fi.doc.CreateElement "event" in
    eventNode.SetAttribute("name", event_type_to_string t);
    eventNode.SetAttribute("application", "false");
    eventNode.SetAttribute("active", "false");
    handlersNode.AppendChild(handlerNode) |> ignore;
    eventNode.AppendChild(handlersNode) |> ignore;
    XmlUtils.append_node fi.doc "/form/events" eventNode;
    fi

Validating the XML

This is the most critical part of the library. We also have to spent the most time thinking about it. First consideration is that we don't want to commit something without having validated it. Second when we have validated something we don't want it to change without having to validate it again. We will deal with these in turn.

In order to ensure that something is validated before it is committed we take advantage of the types. We introduce a new type so that commit takes something that you can only get through validate. We still need all the information from form_info, and the way we prevent users from making a validated value themselves is by making it private.

type validated = private { fi : form_info }

Making sure that something doesn't change after validation is a bit more subtle. Because the XML library has side effects someone could accidentally dublicate a file_info, validate one of them, make changes in the other one, thereby changing the xml of the validated one -- because the doc fields point to the same object -- but because the first is already validated we can commit it. It would look something like this:

let fi = CrmUtils.FormXml.init someform in
let v = CrmUtils.FormXml.validate fi in
(* change fi.doc *)
CrmUtils.FormXml.commit v

The solution I went with is to clone all the mutable data, in this case only doc.

open System.Xml.Schema;
let validate fi = 
  let validationEventHandler sender (e : ValidationEventArgs) =
    match e.Severity with
      | XmlSeverityType.Error -> 
        printf "Validation error: %s\n" e.Message; exit (1)
      | XmlSeverityType.Warning -> 
        printf "Validation warning: %s\n" e.Message
      | _ -> failwith "Impossible" in
  (* "Adding a schema to the XmlSchemaSet with the same 
      target namespace and schema location URL as a schema 
      already contained within the XmlSchemaSet will return 
      the original schema object."
      - https://msdn.microsoft.com/en-us/library/1hh8b082(v=vs.110).aspx *)
  fi.doc.Schemas.Add(null, cfg.rootFolder + @"\SDK\" + sdk_version + @"\Schemas\FormXML.xsd") |> ignore;
  fi.doc.Validate(new ValidationEventHandler(validationEventHandler));
  { fi = { fi with doc = fi.doc.Clone() :?> XmlDocument } }

Note: You need the CRM SDKs in ".\SDK\201X"

Commiting the XML to CRM

This part is the dangoures one. We already validated our XML, but what if there is something we overlooked? A universal advice applies here too: "Always backup your data".

Another thing to note is that CRM is a little picky about the XML you submit to it, so we need to suppress the <?xml version="1.0" encoding="UTF-8"?>.

open System.IO
let commit v =
  (* "If the directory already exists, this method does not 
      create a new directory"
      - https://msdn.microsoft.com/en-us/library/54a0at6s(v=vs.110).aspx *)
  Directory.CreateDirectory(cfg.rootFolder + @"\backup") |> ignore;
  File.WriteAllText(cfg.rootFolder + @"\backup\" + 
    System.DateTime.Now.ToString("yyyyMMddHHmmss") + "-" + 
    v.fi.entity + "." + v.fi.name + ".xml", 
    v.fi.original_formxml);
  let ws = new XmlWriterSettings() in
  ws.OmitXmlDeclaration <- true;
  let sw = new StringWriter() in
  let writer = XmlWriter.Create(sw, ws) in
  v.fi.doc.Save(writer);
  let entity = new Entity() in
  entity.Id <- v.fi.form_id;
  entity.LogicalName <- "systemform";
  entity.Attributes.Add("formxml", sw.ToString());
  xrm.OrganizationService.Update(entity)

This concludes the core library. Next week we will look at how to use this library out of the box, and how to build some neat functions on top of it.

Quality Control

As I am devoded to high quality software we should take a step back and examine how solid this library is. Even though this works and could save us a lot of time we shouldn't ignore potential risks such as overwriting files, destroying forms, or in the worst case locking up the entire CRM system.

Is it supported to change the XML of a form?

We already know that FormXML is documented by Microsoft. Further from Microsofts website (Customize entity forms) we have:

Editing the form definitions from an exported managed solution and then re-importing the solution is a supported method to edit entity forms. When manually editing forms we strongly recommend you use an XML editor that allows for schema validation.

Regarding the validation part, as we have seen, our library actually requires the XML to be validated before it can be committed.

If something goes wrong, then what?

As just mentioned, changing the XML is supported and we require the XML to be validated, making it very unlikely for something to go wrong. Regarding the script files, we always check if files exists before writing to them. Finally each time we commit XML to CRM we make a backup of what the XML was before the commit, thus, even if we could destroy XML we can easily restore it.

Acknoledgements

Thanks to Ramón Soto Mathiesen, Jacob Blom Andersen, and Martin Kasban Tange for feedback and discussion during the development of this library.