Async in C#6

Async in C#6 – Simple example

When you have a large amount of data to be loaded, you won’t be able interact with the window – even to move it around – until the loading is complete. Unless the loading task is done asynchronously. Doing this used to require hundreds of lines describing state machines and so on – Jon Skeet gives a good account of this in his Pluralsight course on Async.

I like his example of the pizza delivery as the means of explaining asynchrony – you order your meal over the phone but don’t hang about at the front door waiting, you go get the dishes ready. Here, we order our data and setup our display whilst it arrives.

Below, is the results of a couple of hours scrapping with syntax to achieve the same result in a toy WPF app. Here, I’ve got a 120 superheroes, belonging to classes who are assigned random grades for several subjects. The notion came from an early morning drill where I was exploring how quickly could I throw a small object hierarchy together from an external source.

The original code is here though likely to be added to as I use other drills on it. Note the use of $ string interpolation rather than string.format – a nice new feature.

A non lambda version, requiring the use of the Async/Await in the main method body itself. And an alternative that hides this requirement completely inside a lambda of it’s very own. Basically you can declare a lambda to be async using the following syntax: async()=> await SomeTask()

 /// <summary>
    /// Non lambda version, the methods have to be async
    /// </summary>
    async void getHereosTask2()
        {
        Wilma.Text = await getHeroes2();
        }


    async Task<string> getHeroes2()
        {
        return await Task.Run(() =>
        {
            StringBuilder sb = new StringBuilder();
            foreach (StudentClass c in School.SuperHeroSchool)
                {
                sb.Append($"Class Name: {c.Name} \nDate:{DateTime.Today.ToShortDateString()}\n");
                foreach (Student student in c.Students)
                    {
                     sb.Append($"\n*** {student.Name}***\n");
                    foreach (var grade in student.Grades)
                        {
                        sb.Append($"{grade.Key} : {grade.Value} \n");
                        }
                    }
                }
            return sb.ToString();
        });
        }

Lambda style:

 /// <summary>
    /// Async style one: here we hide the need for the async inside lambdas :)
    /// Notice how the second method, Task<string> is NOT async.  Instead we just make a nice async lambda lambada!
    /// </summary>
      void getHereosTask()
        {
          Task<string> sb = Task.Run(async() => await getHeroes());
          Wilma.Text = sb.Result;
        }

       Task<string> getHeroes()
        {
        return Task.Run(() =>
        {
            StringBuilder sb = new StringBuilder();
            foreach (StudentClass c in School.SuperHeroSchool)
                {
                sb.Append($"Class Name: {c.Name} \nDate:{DateTime.Today.ToShortDateString()}\n");
                foreach (Student student in c.Students)
                    {
                    sb.Append($"\n*** {student.Name}***\n");
                    foreach (var grade in student.Grades)
                        {
                        sb.Append($"{grade.Key} : {grade.Value} \n");
                        }
                    }
                }
            return sb.ToString();
        });
        }

Final note: at a later stage, an F# version will be done just to practice crossing the idioms over! 🙂

Dynamic Rules engine – update

As an exercise in working with C#, I decided it was time I translated my dynamic rules engine over from VB.

Here is the translated Github gist: C# Dynamic Rules Engine

A rules engine itself is a useful concept for many scenarios, business rules, validation, filtering and more.

At the time I wrote this, I was working with legacy code, no ORM for the database – and classes generated from underlying database tables. It was a class per table, nothing more complex than this.

An operation was performed – by a third party – to merge 2 databases into 1. However, we couldn’t rely on the primary key that was in use any more. A proportion of records had their keys replaced. Not an ideal scenario but it was what we had.

A method of fact checking our table rows was required as various transformations were applied, in code, to try and line up the relationships to child tables in the database. Please note, there were no foreign key constraints either.

As a novice programmer (when does one come out of that title?), I figured that I could help – a little – if there was a way to show a row from a table in database 1 was the same as a row in the matching table in database 3 (the merged database). The only solution I saw at the time was fact checking columns. This required human intervention and knowledge of the data, which I had. About 12 critical tables were involved.

The aim of the game was to create a definition of the members (representing columns) that must match both sides for any row to be a match for it’s twin in the other database’s table. For example: a name, an address and an NI was enough to say this record for a person is the same on both sides. But a name and address was less certain.

As an exercise, I elected to work on this at home and learn what I could to find the neatest way to do this dodgy sounding task.

In the process I covered generics, interfaces, reflection and expression trees. That was a Friday evening – I was done by 2am.

The code worked, we were able to test it out on a number of cases and found that 99% of the data was fine, the rest was easily bought into line.

There’s a long list of reasons why such fact checking, in the way I describe here, is not a good idea. However, as an exercise in learning the programming concepts covered, I still am pleased with the original code. An F# version is likely to be a touch shorter – especially if I attempt it functionally!

Last caveat: The human aspect of having to know the columns means this is unviable for most scenarios and I regard the approach as a piece of throwaway code. Also, it could be argued as too complex a solution for the task at hand. Especially with the performance cost of the amount of reflection used.

The snippet below is the heart of the code:

/// <summary>
    /// The Dynamic Rules Engine itself
    /// Here we have to pass in a class instance and the properties to be matched.
    /// using reflection to get the members, put them in a list and generate a list of lambda predicates expressions
    /// We compile and run these - they all have to be true to get a match. We return a match item, if one exists.
    /// </summary>

    private T getMatchingMerge(T item, List<PropertyInfo> matchProperties)
        {
        List<PropertyInfo> getPropList = item.GetType().GetProperties().Where(s =>
                       s.Equals(matchProperties.Where(x => x.Name == s.Name).FirstOrDefault())).ToList();
        if (propDict.Count < matchProperties.Count)
            {
              propDict = getPropList.ToDictionary(p => matchProperties.Where(s => s.Name == p.Name).First());
            }

        if (_rules.Count != matchProperties.Count) {
            _rules.Clear();  
            foreach (var lambda in from Pair in propDict
                                   let leftProperty = Pair.Key
                                   let rightProperty = Pair.Value
                                   let leftParameter = Expression.Parameter(rightProperty.DeclaringType, rightProperty.DeclaringType.Name)
                                   let rightParameter = Expression.Parameter(rightProperty.DeclaringType, rightProperty.DeclaringType.Name)
                                   let left = Expression.Property(leftParameter, leftProperty)
                                   let right = Expression.Property(rightParameter, rightProperty)
                                   let leftEqualsRight = Expression.Equal(left, right)
                                   // select Expression.Lambda<Func<T, T, bool>>(leftEqualsRight, Expression), new Parameter.Expression() { leftParameter, rightParameter }).Compile()
                                   select Expression.Lambda<Func<T, T, bool>>((Expression)leftEqualsRight, new ParameterExpression[] { leftParameter, rightParameter }).Compile())
                _rules.Add(lambda);
        }
        T _match = MergeList.Where(r => _rules.All(x => x.Invoke(item, r) == true)).FirstOrDefault();
        return _match; //I've not covered the path where there is no match!
     }

C# Dynamic Rules Engine

Random Art Generation with F#

An evening of random art generation with Phil Trelford and the F# Cambridge meetup group:

Blog to be updated with code and explanation!

The gist for the code is here: Random Art Source Code

Thanks to Mark Gray, Phil Trelford and the F# Cambridge meetup group for an interesting evening and the food & drink!

Also, thank you @ Charles Roddie for an interesting discussion on maths and math tuition :).

F# digits recognisor

How can a computer recognise handwritten digits done by many people?

Get a picture of each digit written, turn the image into a long list of numbers representation colours and find a way to compare these with ones it already knows the answer to.

That’s what the digits recogniser I discuss below does.

This is the first moderately complex F# code I’ve given time to and is about 6 hours of proper coding time (conveniently ignoring twitter and google searching time!). However, I’ve highlighted each of the steps and hopefully made the code reasonably accessible to follow.

Any comments and suggestions, please find me on twitter or github. The code can be found here- LiveScript.fsx is the code file in question.

A simple project that springboards directly from a digits recogniser coding dojo found via the F# community – thank you to both!.

The code dojo format offers a scripted step by step approach to learning f# whilst attempting a moderately complex task. As my aim with F% involves machine learning and AI, this is an ideal first step.

The digits recogniser works on vectorised images of handwritten digits – the training set consists of 10,000 samples, the validation set, 50,000. Vectorised – imagine of pulling the thread from a square coloured piece of cloth until you have just a thread and no cloth: you get bits of colour on a long thread. When we vectorise an image, our thread is simply a line of numbers – an array with each number between 0 and 255 (representing 256 colours).

It is a simple example of supervised learning, As we have known data to work with. What will happen is each example in the training set is mathematically compared to an example in the validation set to find the nearest match. The calculation is a simple matrix calculation that finds the Euclidean distance between one vector and another good explanation here!. For our purposes it’s a calculation that simply tries to match on of the vectors in the training set with an unknown sample that we present to the program; in other words, one thread of numbers that looks most like another!

The steps are: for each point in vectors A and B take the difference then square it, then sum them together and take the square root, the result is the Euclidean distance.

However, this is not the only possible calculation we could apply to two vectors and we may need to try others.

The organisation of my code reflects this need by allowing for an arbitrary list of functions to be applied to a given set of data, timing and measuring each for accuracy.

The reason is it may be reasonable to implement a basic genetic algorithm, using code quotations, to generate new functions ; the only rule being they have to fit the data type expected. Using such a procedure we could use a second level of machine learning to attempt to evolve a better formula as measured by time and accuracy.

For each of the stages, I’ll provide the code and explanation next:

First, acquire the data, strip the headers with an array slice then using mapi (this allows us to get a tuple of an index label and an array of pixels to work with) pipe the data straight into an array of records.

open System
open System.IO

let trainingFile = @"C:\Dropbox\Dojo-Digits-Recognizer\Dojo\trainingsample.csv"
let validationFile = @"C:\Dropbox\Dojo-Digits-Recognizer\Dojo\validationsample.csv"

type Example = { Label:int; Pixels:int[] }

let getData file  =
     File.ReadAllLines(file).[1 ..]
         |> Array.map(fun x -> x.Split(',') |> Array.map(fun s -> (int) s)) 
         |> Array.mapi(fun i x -> {Label = i; Pixels = x})

let trainingExamples,validationExamples = getData trainingFile, getData validationFile

We then define the basic distance function that will work through the numbers in both vectors:

let distance (points1:int[],points2:int[], calcFn : int->int->float) = 
                let f (n:int) = float n
                Array.map2(calcFn) points1 points2 
                |> Array.sum |> sqrt

Next we have a simple comparison function that will use the distance function to compare a single unknown sample against all the vectors in the training set:

let findMinimumPixels s sample calcFn = 
    sample  
    |> Array.map(fun x -> (x.Label,  x.Pixels), distance (x.Pixels , s.Pixels, calcFn ))
    |> Array.minBy(fun x -> snd x) |> fun x -> (fst x |> fun a -> snd(a).[0]) , s.Pixels.[0] 

Then we need the basic classifier function that will map all the unknown vectors in a required set using the findMinimumPixels function. It takes an array of integers to use as an index thus allowing us to choose how many or even which samples in the validation set to work with. Here I simply want the first 50 as a quick trial.

let classify (unknown:int[]) calcFn =
    unknown |> Array.map(fun i-> validationExamples.[i] |> fun s -> findMinimumPixels s trainingExamples calcFn) |> Array.map(fun x ->     fst x, snd x)
let dataSet = [|0 .. 50|]

Collate the results, calculate the accuracy:

let results data calcFn = classify data calcFn|> Array.map(fun x -> match x with | n, m when n = m -> 1 | _,_ -> 0) |> Array.sumBy(fun x -> float (x) / float (data.Length) * 100. ) 

Use a simplistic timer to gauge performance – optional:

let timedResult cmd =
let timer = System.Diagnostics.Stopwatch.StartNew()
cmd 
timer.Stop() 
printfn "%f" timer.Elapsed.TotalMilliseconds 
timer.Reset

Define the functions we’d like to trial against – this work is not complete but I’ve setup the code to allow for more functions than Euclidean distance to be explored. The list is a simple example of using functions as data as well. It would be fairly straightforwards to utilise quotations to compile together other functions for this list, provided they matched the data type required.

let functions  = [fun p1 p2 -> (float p1  -  float p2)**2.; //find minimum euclidean distance
              fun p1 p2 -> ((float p1  -  float p2)/2.)**2.; //variation of the first one, no difference in accuracy.
              fun p1 p2 -> sqrt(float p1 * float p2)]    //A nonsense one added to test the notion of a list of functions

Finally, for each function defined try the algorithm plugging each function in and return the accuracy/timing:

for i in functions do
  timedResult (printfn "%A  accuracy achieved" (results dataSet i)) |> ignore

The code so far produces 94.06% accuracy on 100 samples, however at time of writing I confess to an index error with 500 samples. There are several issues to address to complete this code: index error @ > 500 samples, speed and the fact that sqrt is fixed rather than a parameter.

Future versions could include exploring parallel execution – trying out the functions listed in their own separate tasks for example, in addition, the function list could be populated using a genetic algorithm that tries to find a better function that fits int->int->float. The accuracy and timing could be used in combination as a form of cost function – where we hunt for the best tradeoff.

Notes on F#: I like this language, it connects well to other languages with type providers, doesn’t require excessive formality and if you can make the compiler happy then code generally just runs.Problems tend to come from thinking more than forgetting to check for null and so on. Though the trade off is making the compiler happy can be a challenge due to it’s strong type inference.

There are a lot more people and sites to mention, so I’ll limit this to a few that will lead easily to the rest:

A response to Ari Massoodi on Twitter

I’ve always had an aversion to the fixed quantified of talent or innate gifts. Never felt there was deep enough study or evidence that proves the existence of such things beyond circumstances. No one has explained Mozart or any other well known genius; we don’t have objective facts that prove its inbuilt not trained. Instead we have a lot of well reasoned (and not so well reasoned) debate on the subject. 

How someone is trained and how they understand things they learn is crucial. If they don’t connect principles and patterns underlying what they are taught then learning remains hard work. However if a child is lucky enough to learn how these things connect together early enough, they will, with hard work, advance rapidly. Caveat – with a supportive environment!

For me, until proven conclusively otherwise, I take what I feel is a slightly positive view of humanity: innate talent is too fixed an idea and that the correct training, at the right time, with sufficient work can produce miracles. 

In other words, if I was to be a virtuoso at violin, I needed to have started very young with the right training. Please note, it’s not even merely a matter of 10,000 hours of hard work – it is quality of training as well. So it’s not a view that means everyone can become anything-timing and luck play as large a part for one. 

I think I take a view that the idea of inborn gifts is not a useful one: it offers us no forwards path for individuals and is an idea that conceptually supports a fixed mindset.

My note here is purely a personal view of innate gifts-a debate that will probably continue for decades unresolved. 

A long time, no post

Warning, this post doesn’t have a point, it’s a miscellaneous entry, a catch up job.

The past months have been busy, got a job, found a new house, left job to move to new house, spent a few days in hospital and had yet another break from coding.

Wrote that all in one breath.

Since I haven’t written on here for a while, I thought I’d cover 9 months in one, if slightly mangled sentence.

So what happened to the blog and the code then?

Well, I spent six months intensively coding in SQL/VB in a corporate environment and found I didn’t want to code much at home. Funny that, given the Hero Programmer image that was popular.

Right after leaving this job, I managed to block my bile duct with something and ended up with a few days in hospital and a couple of month of listlessness whilst recovering. Yep, it really does take away most of one’s mental energy hurting that bit of the body.

However, I’ve begun to rekindle my enthusiasm for F# and coding. In addition, I’d like to see my other website end up as a book – www.starship-whisperer.com.

The starship fiction site is an unusual idea that has also lead to my learning enough Blender to create models that generally crash my GPU to render. That probably means I’ve still not quite learnt enough. Blender is an interesting, comprehensive piece of open source rendering software that I find fantastic – certainly for designing and rendering space ships! As for the fiction, I’ve chosen to embed live google documents into the pages so that as I edit, they are public. It was an idea that I had to help me continue the story.

However, I learnt a lesson with this: in software careful design and testing is a major part of the job. In writing, at least for some writers, such planning kills the story. When I started, I just wrote and maybe rewrote a little. I found tale after tale.

Then I discovered technique – like plotting and schemes and character bibles. The story dried up, the interest started to pall, I was writing to a plot and I felt I’d blown it. Yet whilst this was happening, I found myself diverting onto Blender and creating the imagery instead. I’ve never done 3D design so this was a big challenge (still is). Some days I lost hours and hours just placing the edges and vertices for one wing – my ships have 18 wings!

In the process I realised that 3D work is simply programming, but in a visual 3 dimensional plane. If you need to, you can talk to Blender via python in the console.

As for the coding? I’ve a gene expression algorithm project that was interrupted by getting a job for one and it’s about time I started tackling a few other technologies that keep cropping up as a “requirement”. Though I do scratch my head about companies listing some must have requirements that probably take a developer a week or less to learn?

Details, Details, Details

I was asked what I had spent the last few weeks doing at an interview recently. As I’ve had a keyboard break, here are a few things I’ve been doing instead. As I missed out on this stuff at school, I decided to teach myself a few useful manual skills: carpentry, some steel work and a bit of restoration.

Damaged porthole: 1mm stainless steel frame, split in five places and being put back together.

I cut the fold making it easier to straighten using hammer and blocks of sapele.

Then, welds are done for me, leaving me with some therapeutic work – hours of polishing. I create a rubber bobbin to allow me to use a wide variety of emery and micro mesh abrasives on my Dremel.

The only disappointment is the remaining dent as removing it would split the weld.

A year off and expectations.

I’ve had nearly a year from full time work, a year that is supposed to disadvantage me in our competitive marketplace. So why do it?

At first it wasn’t planned, I intended a few months to myself and then start hunting for work. I had a few interviews handed to me by recruiters but didn’t end up in a role and found I was happy to do other things. Initially I spent time learning more F# and data science,  passed a Coursera module on R programming and learnt enough python to follow various courses in including statistical methods and mechanics again on Coursera.

However I also discovered I had an interested in making things, fixing things and working on boats with the result that I spent many happy weeks working on a couple of boats, far removed from desks and keyboards. I contemplated leaving IT completely and working on boats or related work full time instead.

I learnt to code in fits and starts during my time at my last role and became increasingly aware of my ignorance. Scott Hanselman describes this rather well in his essay on imposter syndrome: the more you learn the more you realise you need to learn and the more insecure in your knowledge you become.   My work role wasn’t coding, we didn’t use many libraries and our codebase largely procedural. This meant learning current technology was going to be homework only by and large. By the time I came to leave my role, I’d learnt enough OOP and functional programming to get by and apply for senior programmer roles. These would be stretch roles of course for me but I never learnt anything by sticking well within my comfort zone. Unfortunately, the few interviews I took convinced me that my official experience coding was outside of their comfort zone and would always bar my way into the career.

It was February when this knocked me for six, at least regarding programming and to be blunt, I quit trying. I was getting angry and frustrated at a system designed, it seemed, to keep anyone who didn’t meet the ticklists from ever doing so. Not enough experience means we won’t even consider helping you out of your catch 22, the risk is too high.

Maybe they’re right, I can’t read the future in better than they. On the other hand, repeatedly I’m told risk nothing, gain nothing therefore I feel that hiring managers could do with getting out of their own comfort zones and question whether their ticklists and need for so many years of experience gives them good enough results or whether they should gamble on people who perhaps would not ever pass the standard criteria.

I’m coming to the last weeks of my year off and am contemplating what to do for the best. For many employers you are meant to be definite, sorted, have a business plan for your life and be able to elevator pitch your ambitions to fit with their needs. 

I can’t offer that.

My ambitions vary, my plans for the future hazy, I enjoy life, learning, figuring stuff out and being creative. 

This is going to be a tough sell, as smart, conventional, just edgy enough to be different but only just and having followed a mostly conventional route is what people want to hear. Along with formally well educated.

This should be fun.

As for the year off-you live but once, time is irreplaceable and my headstone will not read “I wished I’d spent more time at the office”. 

As for my expectations?

Now, I’m interested in interesting junior dev positions or suitable short term contracts. In junior roles, I’ll get time to refresh and update my skills, yet be able to bring more to the table sooner than perhaps would be expected. As for ambition, as long as I can continue to train, study and am increasingly enabled to make bigger contributions the moment it’s merited, I’ll be happy.

I gave myself a hard time trying to push for stretch roles with appropriate pay. A decent challenge, a great team with an interesting company is what I need, so any takers?

Marketing Personality

The personality one expects to be marketable I’m sure doesn’t sound like this:

I’m a Deep extrapolator and divergent thinker who branches and distracts for a pastime with many hobbies and skills that are frequently dropped and picked up again. I get short intense periods of obsession often leading to decent basic competency or understanding of often esoteric areas. Mostly I’m not particularly accepting of petty restrictions and vastly prefer the lightest touch of authority structures. And I prefer work that allows a decent, regular amount of contact with clients as I can’t solve the right problems without fully understanding the context and domain.

So that’s what I’ve to market not:

The latest buzzword filled character description that one is meant to have: Results oriented problem solver with an outstanding academic record and a long history of high performance.

Everyone is a problem solver of some type, most folks will work hard leaving the redundant results orientated and the annoying latter clauses as filtering filler for the shallowest of people farming analysts out there.

Graphipedia, Context and knowledge

Motivation – The Grand Plan?

Rather large caveat: The approach below simply extracts links and builds a graph database based on page title and links within it, you get the neighbours of any one page this way.

However, when I started I was after what links and WHY. The second aspect is a much more challenging aspect – discussed further down.

Quite simply I was studying something completely different, gene expression programming, and realised I knew nothing about neural networks. So I backtracked and started on these, to find I didn’t have a deep grasp of stats and probability. I’d been doing Coursera courses and kept seeing similar key concepts bandied about: Markov, Monte Carlo, percolation, model thinking and many more. Things I’ve half a grasp of.

I’d seen Graphipedia on GitHub before and I like* concept mapping*. Neo4j stores the relationships AND the item – a perfect concept mapper!

I want a way to automate the tedious aspects of building contextual understanding – finding out what do I don’t know and need to know in order to understand this new thing? You could say I wanted an automatic ignorance mapper!

Can I get a tool that will tell me what concepts I need to understand other concepts and what other concepts knowing the first concept helps me to understand. <<|Yep, I really did this with that sentence.

I burnt a lot of bandwidth and decided to get ALL of Wikipedia, from here:

Wikipedia Data Dump

Given that I’m allergic to wheel invention, I performed Google fu and found this handy Java program:

Graphipedia – Original Author

Graphipedia – Source

Building Graphipedia requires Maven – go here for this and the command mvn install in the folder you extracted Graphipedia to. I had to set my JAVA_HOME correctly to allow Maven to run.

This blog helped

There are two stages to the import task:

Extracting links

java -classpath graphipedia-dataimport.jar org.graphipedia.dataimport.ExtractLinks wiki.xml towiki-links.xml

Importing them to the graph database

java -Xmx3G -classpath graphipedia-dataimport.jar org.graphipedia.dataimport.neo4j.ImportGraph towiki-links.xml graph.db

These stages took about twenty minutes on an i7 9xxx with 32gb and ssd.

Now the fun part: you end up with about 7gb of a neo4j graph db that needs a little persuasion to load up. I had numerous issues with garbage collection blocking and memory performance.

One noteable point, the type of relationship is Link. That’s all, along with the type of node being a page with one property, it’s title.

So to take this further, more steps would be needed. For example, predicting the type of link using machine learning would potentially yield a graph with meaningful relations. However, this is not a trivial problem if we are only relying on the content of wikipedia.

Match (p0:Page{title: 'Neural Networks'}) RETURN p0

Screenshot 2014-03-23 00.46.07

Finally, this graph database has significant performance issues – partly to do with my own level of knowledge of course. Being 7.8 gb with only one type of relationship probably doesn’t help either.

However, here are the settings I’ve used to get it going from the Neo4j properties:

neostore.nodestore.db.mapped_memory=150M
neostore.relationshipstore.db.mapped_memory=1000M
neostore.propertystore.db.mapped_memory=500M
neostore.propertystore.db.strings.mapped_memory=500M
neostore.propertystore.db.arrays.mapped_memory=10M

# Enable this to be able to upgrade a store from an older version
allow_store_upgrade=true

# Enable this to specify a parser other than the default one.
#cypher_parser_version=2.0

# Keep logical logs, helps debugging but uses more disk space, enabled for
# legacy reasons To limit space needed to store historical logs use values such
# as: "7 days" or "100M size" instead of "true"
keep_logical_logs=7 days

Neo4j Performance

The JVM settings I’ve altered to maximum heap before garbage collection occurs:

 -Xmx1G

Before anyone says it, I’m not a Java developer so please look here if you need proper help with this aspect:

Java Flags

Final Thoughts

I’m intent on building a simple knowledge differential engine and then use this to develop an intelligent personal learning agent. Both aspects will require lengthy investigation into the world of AI and machine learning – it’s a recursive project as I started it BECAUSE I knew that there’s lots that I don’t know that I don’t know <|* Yep, I did it again!*

The concept is touch the concepts you want to know about, your presented with various concepts that are related, you indicate how much you think you know about each. Other assessment modes may be adapted over time as there is a lot of existing work to examine on this.

Ultimately, the software learns enough about what you know to provide efficient, relevant concept maps to assist you in building your own mental cognitive map of any complex area.

The act of learning isn’t something that should be automated but the act of finding a map for the territory is perhaps one that can be made far more efficient, leaving the learner with just the act of getting to grips with the relevant stuff.

Solving this, originally meant a convenience tool for me in learning about AI and other complex topics.

Now, I see that to do it, I’ll have to know a fair bit of practical AI, stats/probability, parsing and all that, graph databases and all sorts of intermediate items.

It’s a little bit recursive!

Concept mapping I find to be superior to the later, popularised mind mapping because, like Neo4j, you are storing the relationship type between concepts. In mind mapping, everything springs from a central organising concept whereas in concept mapping, such a mind map would be merely a node.

I’m going back to the drawing board with this because ideally this needs to be an on demand agent that pulls what it needs, when it needs it, intelligently. I’m expecting to have the opportunity to use evolutionary programming like gene expression programming, neural networks, F# and a few other tools to realise this little concept. However, I also expect that I’ve underestimated the complexity of the task by about 2 orders of magnitude!