Author: Julien Grossmann, Political and Violent Risk Data Analyst, HIS Economic and Country Risk

I’ve been using KNIME Analytics Platform for a year and a half, and in this time, KNIME has become a vital part of my work. As a political and violent risks data scientist, I am often confronted with incomplete or badly structured data. But with KNIME, I can always find ways to efficiently clean up, organize and analyze my data, and this, despite a total lack of programming or coding knowledge.

One of the most frustrating aspects of working with unstructured or semi structured data, is that they are rarely ready made for your needs. Therefore, they always require extensive manual clean up.

Worse, if your needs are constantly shifting, the data restructuration and clean up are too.

For instance, the typical data I work with involve datasets of events (such as civil unrest or terrorism attacks). Most data sets have basic meta data like date, location, and a short description of the incident, but there is only so much you can conclude from such basic dataset. Often I need to drill down, and look for specific groups, or specific actions or targets.

In the old days, we would use basic search functions in excel, or for the less geeky of us, we would do it manually.

So I decided to create a little workflow using the KNIME Textprocessing extension. The idea was to be able to mine large datasets rapidly, using a customizable list of keywords.

The KNIME Streaming Executor is an extension that currently "hides" inside KNIME Labs. Not many of our users are aware it exists, so let's find out what it can (and can't) do and why you should be interested.

Streaming?

If you are used to KNIME's standard execution model you will know that connected nodes in a workflow are executed one after the other. The obvious advantage is that you can easily understand what is currently executing and what the (intermediate) results are, produced by each of the nodes in your workflow. That significantly simplifies debugging, because you can see immediately if some intermediate step isn't producing the expected result. You can also reset/continue with the execution at any point in the workflow without re-running the whole workflow -- saving on computation time. Another benefit of the standard execution model is that you can use KNIME's Hiliting to inspect the results and also explore your data using some of the view nodes.

However, the standard execution model also has some drawbacks. Each node needs to cache its results, which requires additional temporary space. (Note that KNIME doesn't duplicate the data from node to node but saves only what has changed between subsequent nodes ... so this isn't as bad as it might sound.) Additionally, as a node doesn't start processing the data until its upstream node has completed, it will need to read the dataset from start to end. Depending on whether those data live in main memory or on hard disc (which is KNIME's choice and depends on data size + available memory) this may require additional I/O operations, which can slow things down. You will notice that if you have a long chain of simple preprocessing nodes, most of the time is spent on just reading the data and caching the intermediate results.

And what Jenkins has to do with sliced bread

If you are a KNIME user you are probably familiar with the mechanism that lets you install additional extensions and update an existing installation with later versions. If you are a KNIME developer you have probably wondered what kind of magic is involved to make this possible: getting from hundreds of line of Java code to an online update that allows users to install and update extensions. In this week's blog post we will reveal some (if not all) of this magic.

The wizard we are using in KNIME is called Buckminster (not be confused with Richard Buckminster Fuller). Buckminster's product sheet describes it as "a set of frameworks and tools for automating build, assemble & deploy (BA&D) development processes in complex or distributed component-based development". The nice thing about Buckminster is that you can use it in your SDK but there is also a headless application which is suitable for fully automating the process.

Although Buckminster is quite well documented (the BuckyBook), initially it can be overwhelming. Therefore we would like to take a closer look at the essentials that are required to build an update site from scratch.

Thanks to Bob Muenchen (muenchen.bob@gmail.com) for permission to share his post from r4stats.com blog post of February 22, 2016
http://r4stats.com/2016/02/22/vcf/

That post refers to two graphs here labelled "Figure1: Gartner Magic Quadrant for 2016. What’s missing?" and "Figure2: Figure 2. Gartner Magic Quadrant for 2015". Since we do not have permission to show those graphs on our site, you might want to quickly click on the link to view them before returning here to read more.

The IT research firm, Gartner, Inc. has released its February 2016 report, Magic Quadrant for Advanced Analytics Platforms. The report’s main graph shows the completeness of each company’s vision plotted against its ability to achieve that vision (Figure 1.) I include this plot each year in my continuously updated article, The Popularity of Data Analysis Software, along with a brief summary of its major points.