Via Ian Moulster's blog, I just picked up on Tafiti which is a Silverlight interface to live search, that has some useful features like the ability to save searches.

If you are attending MIX UK 07 and planning on blogging or just want to pick up on all the announcements coming out of Mix UK 07, there is now a back network to help you keep in touch.

The Next Gen boys appear to have pulled off a little bit of a coup and set up a new Cambridge user group, hosted out at Microsoft's Cambridge Research Lab. That gives them access to some great speakers. They have a session on F# planned for Tuesday 18th September. Very cool. Friend of London .NET user group and MS developer evangelist Mike Ormond will also be there talking about things Silverlight.

 I'd go if I was in the country that week as it's only an hour on the train from London.

As usual we are taking a break for August, but we will get back to running our monthly presentations in August.

And its going to be a big one, so put September 27th in your calendar now, because noted ASP.NET guru Dino Esposito will be joining us (thanks to INETA) to share his insights. More details to follow, but for now, make sure that you block out the date in your diary.

Rick Strahl wants to know how to do dynamic queries in LINQ. I'd point him in the direction of either my earlier post on C# 3.0 Specifications or Mike Taulty's article.

Now I don't usually do memes, though I got tagged on this one by Daniel Moth, but I had something interesting to say on this.

Some of the best advice on becoming a better developer comes from the Pragmatic Programmer and a number of the philosophies towards improvement I have directly come from that book, particularly their comments on taking responsibility for your work and career by investing regularly in your knowledge portfolio. I try to commit to at least 6 hours of professional development a week, be that coding, reading, etc outside of the day job and I've spoked a couple of times about how Ruby is my 'new language' this year.

But in addition this year I have decided to improve what I would call my 'keyboard proficiency'. I've always been a bit of a 'drive by mouse' guy, but I think it is creating dead-ends for me that are stopping me moving further.  I've taken to learning the keyboard shortcuts in VS 2005/8 more thoroughly (by the expedient of moving the mouse out of reach) and brushing up on my CodeRush templates with the interactive window. As the Pragmatic Programmer says: Use a single editor well.

But more than this a lot of the developers I respect seem to have a real fluency with shortcuts, consoles, scripts and use them to automate repetitive tasks. It occurred to me that a  lot of the practices I champion, like TDD, Automated Builds, Continuous Integration probably came from individual developers whose mind sets were focused on using scripting tools for productivity. Indeed the Pragmatic Programmer says: Use the Power of Command Shells, Learn a Text Manipulation Language, Don't Use Manual Procedures...

So my goal is to stop being a WIMP and start using command shells, write ruby scripts to automate parts of my work, make sure that any procedure I'm engaging in is automated. Prompted by a conversation with Zi I have downloaded Powershell, and I am going to walk through the pain of using that instead of applets or explorer. I'll try to learn how to use P4 and Subversion from the command line instead of relying on P4win and Tortoise SVN. I'll manipulate IIS 7.0 from the command line, instead of from its shell.

I don't imagine it is an easy journey, old habits die hard especially under pressure. But I think a lot of Windows developers fall down by comparison to Unix developers, because of our over-reliance on the GUI, which makes it too easy not to have to understand exactly what we are doing, or look for efficiency opportunities through automation. Even playing with PowerShell and understanding the power of pipes and cmdlets is a powerful journey for some people.

Of course once again the pragmatic programmers were right. I must re-read their book again, to see how I can improve again.

I just noticed over at Ian Moulster's blog that early-bird registration for Mix ends this Friday. So if you have not registered, think about doing it soon. Mix will be one of the top UK events for MS developers and designers. We are not getting a worn re-tread of what happened in US, but a unique event with its own content.

This is the last of my posts on LINQ before switching over to Beta2; I'm playing catch-up but be aware that some things will have changed. I will post any changes for Beta 2 to both this and Being Ignorant in LINQ to SQL as time permits.

What are Specifications?

Specification is a design pattern authored by Martin Fowler and Eric Evans. If you are not familiar with specifications then you can read more about them in Eric Evans book Domain Driven Design, or just read this comprehensive paper. If you are not sure what Specifications are, you might want to read that before you read this. For those of you who just want a refresher, or who do not want to read the article first: a specification is a pattern for expressing a rule which you want to use to test an object. A specification is ultimately used to separate two orthogonal concerns: testing objects and the objects themselves. An example should make this clearer.

Instead of writing:

customer.IsInSalesRegion("London");

we want to write:

SalesRegionSpecification salesRegionSpecification = new SalesRegionSpecification("London");

salesRegionSpecification.IsSatisfiedBy(customer);

We separate the specification - the test for a customer being in London - from the customer itself. Separation of responsibility not only means that the code is easier to maintain, because changes to our specifications do not imply changes to Customer, but also allow us to extend the range of specifications without changing Customer. In addition as well as using specifications independently, we can combine them:

SalesRegionSpecification salesRegionSpecification = new SalesRegionSpecification("London");

AccountSinceSpecification accountSinceSpecification = new AccountSinceSpecification(new TimeSpan(1997,0,0));

Specification qualifyingLondonAccountsSpecification = salesRegionSpecification.And(accountSinceSpecification);

qualifyingLondonAccountsSpecification .IsSatisfiedBy(customer);

Note that a specification has characteristics shown by Fluent Interfaces in that it is readable API. The three primary uses of the Specialization pattern are to:

• Validate an object
• Select an object from a list
• Specify the requirements for creation of an object  

Hard Coded Specifications

The basic implementation of a specification is something like this:

public interface ISpecification
{
    bool IsSatisfiedBy(Customer customer);
}

public class CustomerInLondonSpecification : ISpecification
{
    public bool IsSatisfiedBy(Customer customer)
    {
        return customer.City == "London"
    }
}

and we can use it like this:

DataSource dataSource = new DataSource();
List<Customer> londonCustomers = dataSource.GetCustomerList().SelectSatisfying(new CustomerInLondonSpecification());

    ...

assuming something like: 

public static class Spec
{
   public static List<Customer> SelectSatisfying(this List<Customer> existingCustomers, ISpecification specification)
   {
       List<Customer> customers = new List<Customer>();
      foreach (Customer customer in existingCustomers)
      {
          if (specification.IsSatisfiedBy(customer))
              customers.Add(customer);
      }

      return customers;
}

Of course we can use delegates to do something similar by treating code as data, so we can call it at a time of our choosing. In C# 2.0 there is a even generic delegate for this purpose called Predicate: delegate Boolean Predicate<T>(T obj) and a number of methods, such as, on List<T>: List<T> FindAll(Predicate<T> match), which take a predicate to allow selection.

in C# 3.0 this enables us to use lambda expressions to write code as follows:

Predicate<Customer> isCustomerInLondon = c => c.City == "London"
List<Customer> londonCustomers = dataSource.GetCustomerList().FindAll(isCustomerInLondon);

or even

List<Customer> londonCustomers = dataSource.GetCustomerList().FindAll(c => c.City == "London");

which in essence, fulfills the implementation requirements of a hard coded specification. And of course LINQ really just gives us another way of writing this:

var query =
    from c in dataSource.GetCustomerList()
    where c.City == "London"
    select c;  

which shows the relationship between LINQ and specifications in that LINQ is one way of implementing selecting an object from a list.  

Parameterized Specifications

The trouble with hard-coded specifications is that we are going to end up with type explosion if we need a lot of them (or a lot of LINQ expressions), so we then arrive at the idea of parameterizing them. That way instead of a specification that checks if the customer is in London and one that checks if they are in Seattle, we can just have one to check if they are in a city and pass in the city that we are after as a paramter to the specification when we construct it.

So, extending the example above we could implement a parameterized method as follows:

public class CustomerForCitySpecification : ISpecification
{
    private string city;

    public CustomerForCitySpecification(string city)
    {
        this.city = city;
    }

    public bool IsSatisfiedBy(Customer customer)
    {
        return customer.City == city;
    }

}

 and use it like this:

ISpecification isInLondon = new CustomerForCitySpecification("London");
List<Customer> londonCustomers = dataSource.GetCustomerList().SelectSatisfying(isInLondon);

Of course we can also use delegates to achieve the same effect, by expanding the number of parameters. Of course this would be painful if we had to create a new delegate for each and every combination of parameters and return type. Fortunately C# 3.0 comes to the rescue by defining a range of generic delegates, provided we want less than three parameters, we just have to remember that the last type is the return type:

public delegate RT Func<RT>();
public delegate RT Func<T0, RT>(T0 a0);
public delegate RT Func<T0, T1, RT>(T0 a0, T1 a1);
public delegate RT Func<T0, T1, T2, RT>(T0 a0, T1 a1, T2 a2);
public delegate RT Func<T0, T1, T2, T3, RT>(T0 a0, T1 a1, T2 a2, T3 a3);

so we can use these to get lambdas to provide us with a specfication as before:

Func<Customer, string, bool> isCustomerIn = ( c, cty) => c.City == cty;
List<Customer> londonCustomers = dataSource.GetCustomerList().SelectSatisfying(isCustomerIn, "London");

again assuming something like:

public static List<Customer> SelectSatisfying(this List<Customer> existingCustomer s, Func<Customer, string, bool> specification, string city)
{
    List<Customer> customers = new List<Customer>();
    foreach (Customer customer in existingCustomers)
    {
       if (specification(customer, city))
          customers.Add(customer);
    }

    return customers;
}

although I find this becomes a little clumsier than the object version. Of course LINQ really provides the best way of doing list selection variants here:

public List<Customer> GetCustomersFor(string city)

{

    return
       from c in dataSource.GetCustomerList()
       where c.City == city
       select c;  

}

Composite Specifications

While encapsulating our specifications helps us to reduce duplication and improve maintainability, the real power of a specification comes from combining them. The common approach to combining specifications is to use the composite pattern. The composite pattern is a simple way of representing tree structures. Within a composite nodes are polymorphic, that is they share the same type (i.e. interface), but can have different implementations. Nodes may be leaf nodes or they may contain other nodes of the same type. The composite pattern lets us treat leaf nodes and compositions uniformly. What we need to know most, is that its great for lightwieght parsing of an expression.

Lets modify our specification interface to add some logical operations

public interface Specification<T>
{
    bool IsSatisfiedBy(T customer);
    Specification<T> And(Specification<T> other);
    //... more operators
}

public abstract class AbstractSpecification<T>: Specification<T>
{
    public Specification<T> And(Specification<T> other)
    {
        return new AndSpecification<T>(this, other);
    }

    abstract public bool IsSatisfiedBy(T customer);
}

public class AndSpecification<T> : AbstractSpecification<T>
{
    private Specification<T> rhs;
    private Specification<T> lhs;
    public AndSpecification(Specification<T> rhs, Specification<T> lhs)
    {
        this.rhs = rhs;
        this.lhs = lhs;
    }

    public override bool IsSatisfiedBy(T candidate)
    {
        return rhs.IsSatisfiedBy(candidate) &&
        lhs.IsSatisfiedBy(candidate);
    }
}

which lets us write code like this:

Specification<Customer> isInLondon = new CustomerCitySpecification("London").And(new CustomerCountrySpecification("UK"));
List<Customer> londonCustomers = dataSource.GetCustomerList().SelectSatisfying(isInLondon);
ObjectDumper.Write(londonCustomers, 1);

given something like this:

public static List<Customer> SelectSatisfying(this List<Customer> existingCustomers, Specification<Customer> specification)
{
    List<Customer> customers = new List<Customer>();
    foreach (Customer customer in existingCustomers)
    {
       if (specification.IsSatisfiedBy(customer))
          customers.Add(customer);
    }

    return customers;

}

 the line:

Specification<Customer> isInLondon = new CustomerCitySpecification("London").And(new CustomerCountrySpecification("UK"));

is expressive because it is readable and within the ubiquitous language of the domain.

Of course we could just write

Predicate<Customer> isCustomerInLondon = c => c.City == "London" && c.Country == "UK"

which is clearly as expressive, but is hard to re-use as it its parts are not composable, leading to duplication. So what we want is some way of making our predicates composable and re-usable without losing their expressiveness.

C# 3.0 lambda expressions can be represented either as delegates or as expressions, a parsed version of the lambda expression. So we can use expressions to store our lambda expressions and recombine them:

Expression<Predicate<Customer>> isCustomerInLondon = c => c.City == "London"
Expression<Predicate<Customer>> isInUK = c => c.Country == "UK"
Expression<Predicate<Customer>> isLondonUKCustomer = isCustomerInLondon.And(isInUK);

given something like:

public static Expression<Predicate<T>> And<T>(this Expression<Predicate<T>> rhs, Expression<Predicate<T>> lhs)
{
    //Parameters to both expressions need to be the same and of type T
    ParameterExpression parameter = Expression.Parameter(typeof(T), "p");
    var invokedExpression = Expression.Invoke (lhs, rhs.Parameters.Cast<Expression> ());
    return Expression.Lambda<Predicate<T>> (Expression.And (rhs.Body, invokedExpression), rhs.Parameters);
}

But once we start to write Expression<Predicate<Customer>> our abstractions are leaking into our fluent interface ruining the effect, so what we would like to do is wrap them up within a class when we create them to clean up the syntax. What we are looking for is something like:

ISpecification<Customer> isCustomerInLondon = new Specification<Customer>(c => c.City == "London");
ISpecification<Customer> isInUK = new Specification<Customer>(c => c.Country == "UK");

ISpecification<Customer> isLondonUKCustomer = isCustomerInLondon.And(isInUK);

List<Customer> londonCustomers = dataSource.GetCustomerList().SelectSatisfying(isLondonUKCustomer);

 

which we can do by writing the following

public interface ISpecification<T>
{
    Expression<Predicate<T>> Predicate {get;set;}
    ISpecification<T> And(ISpecification<T> other);
    bool IsSatisfiedBy(T customer);
}

abstract public class AbstractSpecification<T> : ISpecification<T>
{
    protected Expression<Predicate<T>> predicate;
    public abstract bool IsSatisfiedBy(T value);

    public Expression<Predicate<T>> Predicate
    {
        get
        {
            return predicate;
        }
        set
        {
           predicate = value;
        }
   }

   public ISpecification<T> And(ISpecification<T> other)
   {
        return new AndSpecification<T>(this, other);
   }

}

public class Specification<T> : AbstractSpecification<T>
{
    public Specification(Expression<Predicate<T>> predicate)
    {
        this.predicate = predicate;
    }

    public override bool IsSatisfiedBy(T value)
    {
       return predicate.Compile().Invoke(value);
    }
}

public class AndSpecification<T> : AbstractSpecification<T>
{
    private ISpecification<T> rhs;
    private ISpecification<T> lhs;
    public AndSpecification(ISpecification<T> rhs, ISpecification<T> lhs)
    {
       this.rhs = rhs;
       this.lhs = lhs;
    }

    public override bool IsSatisfiedBy(T candidate)
    {
        ParameterExpression parameter = Expression.Parameter(typeof(T), "p");
        var invokedExpression = Expression.Invoke (lhs.Predicate, rhs.Predicate.Parameters.Cast<Expression>());
        Expression<Predicate<T>> andPredicate = Expression.Lambda<Predicate<T>>(Expression.And (rhs.Predicate.Body, invokedExpression), rhs.Predicate.Parameters);
        return andPredicate.Compile().Invoke(candidate);
   }
}

The interesting part here is that use the expression tree that we have for both lambda expressions to merge the two expressions into one new one. You can find a post where Matt Warren talks about this on MSDN Forums. Metaprogramming can look a little scary, but we're just working at a slightly higher level than when writing the code directly. The key to this is that it makes specification composition 'simple'.

Of course IQueryable<T> is composable so that I can write something like this to achieve a similar effect for LINQ queries, by using a prior IQueryable. So given something like this:

IQueryable<Customer> customersInLondon = 

    from c in customers
         where c.City == city
         select c;

I can compose it by writing something like

IQueryable<Customer> customersInLondonUK =

     from c in customersInLondon
         where c.Country == "UK"
         select c;

Which means that given an IQueryable<Customer>, which might be an collection on a DB, or an IEnumerable<Customer> collection I called ToQueryable() on (see my previous post on Being Ignorant in LINQ to SQL as to why this is important) I can keep adding new specifications to the original query until I have my required test.

And of course you might simplify this as above by writing something like:

public static IQueryable<T> IsInCity(this IQueryable<T>  source, string country)
{
    return

      from c in source
         where c.City == city
         select c;

}

which enables something like:

IQueryable<Customer> customers = ... //db or collection source

IQueryable<Customer> londonCustomers = customers.IsInCity("London");

Specifications and Repositories

There is a link between specifications and repositories in that repositories often act as the collections against which specifications are applied to produce a set of matching results. Look over at Being Ignorant with LINQ to SQL if you need more on this. This tends mean we want code that looks something like this

 

DataContext context = new DataContext(ConfigurationManager.ConnectionStrings["NorthWind"].ConnectionString, Mapping.GetMapping());

UnitOfWork unitOfWork = new UnitOfWork(context);
CustomerRepository customerRepository = new CustomerRepository(unitOfWork);

ISpecification<Customer> customersInLondon = new Specification<Customer>(c => c.City == "London");
ISpecification<Customer> customersInUK = new Specification<Customer>(c => c.City == "UK");

ISpecification<Customer> customersInLondonUK = customersInLondon.And(customersInUK);

List<Customer> matchingCustomers = customerRepository.FindBySpecification(customersInLondonUK);

but I can't extend the repository by writing something like:

public List<Customer> FindBySpecification(ISpecification<Customer> specification)
{
    return
        (from c in Customers
        where specification.IsSatisfiedBy(c)
        select c).ToList();
}

because LINQ can't convert specification.IsSatisfiedBy(c) to valid SQL. But I can write:

public IEnumerable<Customer> FindBySpecification(ISpecification<Customer> specification)
{
    IQueryable<Customer> customerQuery = from c in Customers select c;
    IQueryable<Customer> restrictedCustomerQuery = customerQuery.Where<Customer>(specification.Predicate);
    return restrictedCustomerQuery.ToList();
}

The only slight wrinkle to get this to work is that I need to switch from using a Predicate<T> to a Func<T, bool> because IQueryable<T> does not expect the former, as follows:

public interface ISpecification<T>
{
    Expression<Func<T, bool>> Predicate {get;set;}
    ISpecification<T> And(ISpecification<T> other);
    bool IsSatisfiedBy(T customer);
    ISpecification<T> Not();
    ISpecification<T> Or(ISpecification<T> other);
}

abstract public class AbstractSpecification<T> : ISpecification<T>
{
    protected Expression<Func<T, bool>> predicate;
    public abstract bool IsSatisfiedBy(T value);

    public Expression<Func<T, bool>> Predicate
    {
        get
        {
            return predicate;
        }
        set
        {
            predicate = value;
        }
    }

    public ISpecification<T> And(ISpecification<T> other)
    {
        return new AndSpecification<T>(this, other);
    }

}
public class Specification<T> : AbstractSpecification<T>
{
    public Specification(Expression<Func<T, bool>> predicate)
    {
         this.predicate = predicate;
    }

    public override bool IsSatisfiedBy(T value)
    {
       return predicate.Compile().Invoke(value);
    }
}

public class AndSpecification<T> : AbstractSpecification<T>
{
    private ISpecification<T> rhs;
    private ISpecification<T> lhs;
    public AndSpecification(ISpecification<T> rhs, ISpecification<T> lhs)
    {
        this.rhs = rhs;
        this.lhs = lhs;

        ParameterExpression parameter = Expression.Parameter(typeof(T), "p");
        var invokedExpression = Expression.Invoke (lhs.Predicate, rhs.Predicate.Parameters.Cast<Expression>());
        Predicate = Expression.Lambda<Func<T, bool>>(Expression.And (rhs.Predicate.Body, invokedExpression), rhs.Predicate.Parameters);
    }

    public override bool IsSatisfiedBy(T candidate)
    {
        return Predicate.Compile().Invoke(candidate);
    }
}

and just for expressiveness we could use more C# 3.0 language features to improve the conciseness of our calling code:

var customersInLondon = new Specification<Customer>(c => c.City == "London");
var customersInUK = new Specification<Customer>(c => c.City == "UK");

var customersInLondonUK = customersInLondon.And(customersInUK);

or even

var customersInLondonUK = new Specification<Customer>(c => c.City == "London").And(new Specification<Customer>(c => c.City == "UK"));

Of course the win here is that our specification does not care if the datasource is in-memory or persisted elsewhere, it relies on the IQueryable<T> implementation to understand that, and just modifies that IQueryable.

Further Reading

Those of you with an interest in Dynamic Queries might want to check out Mike Taulty's post, and its follow ups here, and here. The major difference is that Mike uses string based expressions instead of lamdas to build his queries, modelling a different interaction style.

Matt Warren has a great series of posts over on his blog on buildling an IQueryable provider for those of you with an interest programming with expression trees, which pretty much explains the how of LINQ to SQL and as such is a great source on how to 'roll your own'.