Generics, Reflection, Expression Trees in real life.

Update: Please see my GitHub gists for more: Dynamic Rule Engine Builder

What is reflection and how is it useful?

What are expression trees and how are they useful?

What are generics and again, how are they useful?

The code below, somewhat edited, goes towards explaining these three .net concepts.

Two databases were migrated into one and needed all 77 tables to be checked. This was an in house solution and a class per table pattern was used throughout. Here, I quickly realised that repeated my algorithm was a fool’s errand so spent a few hours understanding how to create my own generic version. Along the way, it became clear that the primary key in most tables cannot be used to check that the fact any particular row represents is the same fact on the original database. So, whilst I couldn’t eliminate all manual customising, I could reduce it down to column selection.

In other words, per table there are specific columns that MUST be the same so the member representation in any class could be reflected and utilised. This is the heart of the source code further down:

Public Function getMatchingMerge(item As T, matchProperties As List(Of PropertyInfo)) As T
    Dim getPropList = item.GetType().GetProperties.Where(Function(s) s.Equals(matchProperties.Where(Function(x) x.Name = s.Name).Select(Function(a) a).FirstOrDefault)).ToList
    If propDict.Count < matchProperties.Count Then propDict = getPropList.ToDictionary(Function(p) matchProperties.Where(Function(s) s.Name = p.Name).First)

    If _rules.Count <> matchProperties.Count Then
        _rules.Clear()
        For Each Lambda In From pair In propDict
                           Let leftProperty = pair.Key
                           Let rightProperty = pair.Value
                           Let leftParameter = Expression.Parameter(item.GetType, item.GetType.Name)
                           Let rightParameter = Expression.Parameter(rightProperty.DeclaringType, rightProperty.DeclaringType.Name)
                           Let left = Expression.Property(leftParameter, leftProperty)
                           Let right = Expression.Property(rightParameter, rightProperty)
                           Let leftEqualsRight = Expression.Equal(left, right)
                           Select Expression.Lambda(Of Func(Of T, T, Boolean))(DirectCast(leftEqualsRight, Expression), New ParameterExpression() {leftParameter, rightParameter}).Compile
            _rules.Add(Lambda)
        Next
    End If

    Dim _match = Mergelist.Where(Function(r) _rules.All(Function(x) x.Invoke(item, r) = True)).FirstOrDefault  
    Return _match
End Function

The rule engine builder can be used to generate predicate Lambda functions – in response to the arbitrary number of properties a.k.a columns that are passed in. These properties are intended to allow identification of a unique, specific fact in both the source and merge database tables. In other words, is Person 21 (source) the same as Person 47 (merge)? Generics are analogous to templates in C++ and allow you to have a strongly type data structure – be it a list, queue, dictionary – of any type you can create. So here I can have a List(Of TypeFromSomeTable) without casting from Object to TypeFromSomeTable. However, I also needed to define common operations that applied to any type I may create from a database query. The operations are:

Create source List(of YourType).

Create a merge List(of YourType).

Generate a rule engine-here we reflect the property types to iterate over.

Create a List(of MatchedPair(of YourType) GetMatchingMerge creates a rules list on first use then uses it to determine what the matching merge item is for a given source item.

Reflection: A simple way to use it is accessing a DLL in your running application. You can load the assembly in the DLL itself then query it for types and members using methods in System.reflection. You can list types, methods, properties and execute them. I’ve used it to create custom reporting DLL’s in one project. Here I needed it to examine a type I sent into the instance of my BuildGenericMergeList class.

Expression trees: Using code as data – create functions at runtime, based on changing conditions. Here I do not know the number, name or type of properties that will be sent until the calling classes send them. The Reflected properties matching our requirements are projected into a list of PropertyInfo, this list is iterated over and expressions built up from the reflected types and names. We step through the properties, generating a left and right parameterexpression, then a left/right memberexpression, finally bundling them into an expression.Lambda and compiling. The result is added to the rules() list. The weakness is each class you must build separately, defining what the match properties actually are for each table (or query). In addition, the table rows need to be represented by a class. So this pattern will require adaption for any ORM that does anything more complex than class per table representation. This took about 5 hours on a Friday evening to figure out and I’m rather pleased with the result :).

Imports System.Linq.Expressions
Imports System.Reflection
''' <summary>
'''This generic class encapsulates the main algorithms for building a rule engine and a list of matched source/merge 
'''"rows" To prove whether a row in source and target are equivalent. 
''' A property in the classes we may be using represent columns in a table or query.
'''Say we have an instance thus: 
'''SourcePerson = New Person with {.Name = "ommitted db query", 
'''                                .DOB =" ommitted db query"}
'''MergedPerson = New Person with {.Name = "ommitted db query", 
'''                                .DOB =" ommitted db query"}
'''We add these instances to a List(of Person) so we get SourceList(of Person), MergeList(Of Person).
'''Then we would like to build some list where each item is a match pair of Persons. The engine below can be told to match on .Name and .DOB; 
'''it will generate the rules as needed and use them to decide if instances from each list are in fact, the same "fact". 
'''
'''So we get a List(of MatchedPair(Of Person)) where MatchPair has members .SourcePerson and .MergePerson. In addition, 
'''using the IName, the MatchedPair instance will guarantee an Overrides ToString that returns a member from MergePerson (in this case).
''' </summary>
'''<typeparam name="T"></typeparam>
'''<remarks></remarks>

Class BuildGenericMergeList(Of T As INamed)
Private Property _SourceList As New List(Of T)
Public ReadOnly Property SourceList As List(Of T)
    Get
        Return _SourceList
    End Get
End Property

Private Property _MergeList As New List(Of T)
Public ReadOnly Property Mergelist As List(Of T)
    Get
        Return _MergeList
    End Get
End Property

Private Property _unMatchedSource As New List(Of T)
Public ReadOnly Property unMatchedSource As List(Of T)
    Get
        Dim temp = mMatchedPairs.Select(Function(x) x.sourceItem).ToList
        _unMatchedSource = temp.Except(_SourceList).ToList
        Return _unMatchedSource
    End Get
End Property

Private ReadOnly Property mDictBuild As Boolean
    Get
        Return _SourceList.Count = 0 And _MergeList.Count = 0
    End Get
End Property
Public Sub New(query As String, action As Action(Of String, dbConnect, List(Of T)))  
    _action = action
    If query.Length > 0 Then BuildList(query)
End Sub

Private Sub BuildMatchingPairs(properties As List(Of PropertyInfo))
    For Each t In SourceList
        Dim mergeItem As T = getMatchingMerge(t, properties)
        If IsNothing(mergeItem) = False Then mMatchedPairs.Add(New MatchPair(Of T) With {.sourceItem = t, .mergeItem = mergeItem})
    Next
End Sub

Private Property mMatchedPairs As New List(Of MatchPair(Of T))
Public ReadOnly Property MatchedPairs(properties As List(Of PropertyInfo)) As List(Of MatchPair(Of T))
    Get
        If mMatchedPairs.Count = 0 Then BuildMatchingPairs(properties)
        Return mMatchedPairs
    End Get
End Property

Private Sub BuildList(query As String)
    Dim dbset = New testingDBSet
    If mDictBuild Then
            For Each dbConnect In {dbset.getDbConnect(dbset.SourceDB), dbset.getDbConnect(dbset.TargetDB)}
            BuildList(query, dbConnect, SourceList)
        Next
        BuildList(query, dbset.getDbConnect(dbset.MergedDB), Mergelist)
    End If
End Sub

Private Property _action As Action(Of String, dbConnect, List(Of T))

Private Sub BuildList(ByVal query As String, ByVal dbConnect As dbConnect, ByVal list As List(Of T))
    _action.Invoke(query, dbConnect, list)
End Sub

Private Property propDict As New Dictionary(Of PropertyInfo, PropertyInfo)

Private ReadOnly _rules As New List(Of Func(Of T, T, Boolean))

Public Function getMatchingMerge(item As T, matchProperties As List(Of PropertyInfo)) As T
    Dim getPropList = item.GetType().GetProperties.Where(Function(s) s.Equals(matchProperties.Where(Function(x) x.Name = s.Name).Select(Function(a) a).FirstOrDefault)).ToList
    If propDict.Count < matchProperties.Count Then propDict = getPropList.ToDictionary(Function(p) matchProperties.Where(Function(s) s.Name = p.Name).First)

    If _rules.Count <> matchProperties.Count Then
        _rules.Clear()
        For Each Lambda In From pair In propDict
                           Let leftProperty = pair.Key
                           Let rightProperty = pair.Value
                           Let leftParameter = Expression.Parameter(item.GetType, item.GetType.Name)
                           Let rightParameter = Expression.Parameter(rightProperty.DeclaringType, rightProperty.DeclaringType.Name)
                           Let left = Expression.Property(leftParameter, leftProperty)
                           Let right = Expression.Property(rightParameter, rightProperty)
                           Let leftEqualsRight = Expression.Equal(left, right)
                           Select Expression.Lambda(Of Func(Of T, T, Boolean))(DirectCast(leftEqualsRight, Expression), New ParameterExpression() {leftParameter, rightParameter}).Compile
            _rules.Add(Lambda)
        Next
    End If

    Dim _match = Mergelist.Where(Function(r) _rules.All(Function(x) x.Invoke(item, r) = True)).FirstOrDefault  
    Return _match
End Function

End Class

Public Class MatchPair(Of T As INamed)
Implements INamed
Property sourceItem As T
Property mergeItem As T

Overrides Function ToString() As String Implements INamed.Name
    Return mergeItem.Name
End Function

End Class

Public Interface INamed
Function Name() As String
End Interface

'''Used by calling classes to build a property list.'''

Module ReflectionUtility
Function GetMemberInfo(Of T)(expression As Expression(Of Func(Of T))) As MemberInfo
    Dim member = TryCast(expression.Body, MemberExpression)
    If member IsNot Nothing Then Return member.Member
    Throw New ArgumentException("Expression is not a member", "expression")
End Function
End Module