New InMemory Adapter for fast data processing/enrichment

InMemory Adapter - for fast data processing/enrichmentThe new InMemory Adapter is considered an absolute highlight. The new InMemory Adapter was developed to take full advantage of the trend of increasingly powerful computers with more and more main memory, making the processing of intermediate data possible, completely in memory.

With the InMemory Adapter as many “internal” tables can be created as required, which can then be populated, enriched and transformed while running a Definition and of course subsequently used as input.

Tasks that previously required a temporary database table can be easily converted to using InMemory tables. As a demonstration the example “Merging CSV and Excel data using an SQLite database” is adapted to use InMemory tables, incl. video example.

Advantage: As processing is completely performed in main memory it is significantly faster than was previously possible.

The use of InMemory tables is in principle very simple. Insert the Adapter and if necessary define a name for the table. Fields can either be included in advance or further fields simply drawn from the field list in the Adapter by drag & drop. Optionally the data types can be modified for specific needs, although normally the default FlowHeater data type STRING should suffice.

On the READ side the data source of InMemory tables presented can also be filtered by a WHERE clause in a similar way to SQL tables. Naturally, the data can also be ordered in an alternative sort sequence.

Release notes for version 4.1.4 on February 2, 2019

Further changes

SQL, Lookup and String Replace Heaters support InMemory tables

The String Replace, SQL and Lookup Heaters can directly access the new InMemory tables. When in these Heaters the respective options for processing InMemory tables is checked, it is also possible to access tables that were output by other processing steps than the present one. This significantly speeds up the process, especially in the replacement of values (CSV Lookup) using the String Replace Heater with long CSV lists.

Batch Module – outputs the processing step

As of this version the Batch Module reports the current processing step incl. the number of records processed on the command line as well as in the log file.

If-Then-Else Heater can compare strings ignoring leading/trailing space characters

In the processing of text files it is often the case that database fields contain extra leading and/or trailing space characters. When it was necessary to test such fields they previously required an additional String Trim Heater to enable compares to succeed consistently. As of this version, it is possible to specify in the IF-THEN-ELSE Heater whether leading/trailing spaces should be taken into account for the compare, or ignored.

DoWork - Test and Run popup with display of processing step

The output of warnings and debug reports in DoWork (Test and Run popup) has been enhanced to include the number of the current processing step. In the case of a complex Definition this simplifies things somewhat.

TextFile Adapter – automatically replaces line breaks in field content

In database fields, the content is often saved with embedded line breaks. This causes problems when output to a text file, such as CSV, which can inevitably lead to data being placed in the wrong field or column. As of this version, the TextFile Adapter supports automatic removal of possible line breaks in/from database fields.

Designer – display of recently opened Definitions

The length of the list of recently opened Definitions can now be configured in the Designer. Previously only the last 10 were listed.

Generate XML files with or without BOM

For unicode (UTF) character sets an XML Adapter option supports the creation of XML files with or without BOM (byte order mark)

Corrections

• Memory Mode – correction to Memory Mode in conjunction with Parameter setting
  
  
• FlowHeater Server – for file monitoring the “hand over current filename” option is now checked by default.
  
  
• Opening older Definitions – with the introduction in version 4.3.1 of several processing steps the backward compatibility to Definitions constructed by older versions was unfortunately interrupted. This version restores compatibility and Definitions going back to version 1.x can now be opened and modified.
  
  
• SQL and Lookup Heaters – cured a fault that occurred when a default value was specified and unless the default value applied while processing the first record, any FlowHeater Parameter it contained remained unresolved for the processing of subsequent records.
  
  
• AutoID Heater – corrected counting of values in combination with the GroupBy Heater.
  
  
• Excel Adapter – fixed error when attempting to read a write-protected Excel workbook.
  
  
• Excel Adapter – corrected problems when the “Assign columns dynamically by field name ” was checked and insufficient columns were available in the worksheet.