News
New InMemory Adapter for fast data processing/enrichment
InMemory Adapter - for fast data processing/enrichmentThe new InMemory Adapter is considered an absolute highlight. The new InMemory Adapter was developed to take full advantage of the trend of increasingly powerful computers with more and more main memory, making the processing of intermediate data possible, completely in memory.
With the InMemory Adapter as many “internal” tables can be created as required, which can then be populated, enriched and transformed while running a Definition and of course subsequently used as input.
Tasks that previously required a temporary database table can be easily converted to using InMemory tables. As a demonstration the example “Merging CSV and Excel data using an SQLite database” is adapted to use InMemory tables, incl. video example.
Advantage: As processing is completely performed in main memory it is significantly faster than was previously possible.
The use of InMemory tables is in principle very simple. Insert the Adapter and if necessary define a name for the table. Fields can either be included in advance or further fields simply drawn from the field list in the Adapter by drag & drop. Optionally the data types can be modified for specific needs, although normally the default FlowHeater data type STRING should suffice.
On the READ side the data source of InMemory tables presented can also be filtered by a WHERE clause in a similar way to SQL tables. Naturally, the data can also be ordered in an alternative sort sequence.
Support for several processing steps in a Definition
Several processing steps in a DefinitionAs of this version FlowHeater now supports several processing steps within a single Definition (.fhd file). Any number of processing steps can be added to a Definition, which are then run consecutively in the sequence given.
Tip: FlowHeater Parameters are “globally” available within a Definition. This makes a number of interesting new options available for processing import/export data. For example, it is possible to record specific data from an upstream processing step in FlowHeater Parameters and use these values in a downstream processing step, e.g. as filter criteria.
Note: Heaters/Functions that are configured to fire “at the start” or “at the end” of the run now apply to each processing step! For example, if a script or SQL command must only be run at the very end of a Definition then this script (.NET Heater) or SQL command (SQL Heater) should only be defined in the last processing step.
Restrictions: At present several processing steps is only supported when using “Mass Data Mode”. The use of several processing steps is not presently supported with “Memory Mode”, and any attempt to use it this way will terminate with an error report.
Preview of later development: We offer here a brief outlook on future functionality that will profit from this new feature. The development of an “internal” memory-like Adapter is planned, using which will simplify the assembly, merging and processing of data from several sources that can then finally be exported.
FlowHeater Server optimizations
FlowHeater Server - Exit CodesFlowHeater Server is enhanced with some powerful new functions by this version. Not only is it possible to monitor folders for the appearance of new files, but also now for modified files. Processing starts as soon as the file is no longer open in any other process.In addition, it is now possible to check “Exit Codes” to determine if tasks executed were “Successful” or terminated with “Warnings” or with “Errors”. Formerly, only a fixed value of “0” could be checked, indicating success. Particular exit codes can be specified, as well as ranges like “0-10” or combinations such as “0, 5-10, 200”. Also new is that exit codes are given for warnings. The most recent run of a task is now highlighted in color in the task list. Yellow indicating the task ended with warnings and red when the task last ended with an error.
Notification options have also been extended. Email notifications can now be sent for warnings as well. Previously emails were only possible for successful runs or fatal errors.
New option in AutoID Heater to increment "only every X records"
Increment only every X recordsAs of this version of FlowHeater you are able to establish a counter using the AutoID Heater that only increments when a specified number of records/rows have been processed. This makes it relatively straightforward to handle very large exports, for example, where the output must be split into multiple CSV files (every X records start a new export file). Naturally this option supports the use of FlowHeater Parameters. This way the maximum number of rows in a CSV export file can be dynamically set by a runtime parameter in Batch Mode.
e.g. FHBatch.exe /SetParameter MAX-REC=10000 export-csv.fhd
In order for this to function properly the counter generated must of course be saved in another FlowHeater Parameter and this Parameter is then used as a placeholder in the output filename of the TextFile Adapter.
e.g. export$FILEID$.csv
This will trigger the Textfile Adapter to output a new file automatically each time the AutoID counter value is incremented.
There is a short video example of this included in the AutoID Heater documentation.
New CMD Heater – executing an external script or program
This version adds another powerful function (Heater) to FlowHeater. With the new CMD Heater it is possible to not only dynamically execute external programs, but also run scripts (Perl, PHP, PowerShell, etc.) while an import/export is being run. The actual FlowHeater process will pause until the external processing is complete. Possible output from the external process can be redirected via FlowHeater Parameters for subsequent use by the FlowHeater Definition. If execution of the external program/script aborts with an error then the exit code of the task can be checked by an IF-THEN-ELSE Heater and if necessary the FlowHeater run also aborted using the Stop Heater.
