Dynamic data import / export

datenbank

simply more productive
Up to 10,000 records can be processed with the freeware version!
Welcome, Guest
Username: Password: Remember me
Welcome to the FlowHeater support forum.

Questions about using FlowHeater are answered here.

TOPIC: Append Only New Records

Append Only New Records 5 years 2 months ago #2158

  • Niel Bullock
  • Niel Bullock's Avatar
  • Offline
  • New Member
  • Posts: 6
I have a table with over 550K records and would like to bring over only the new ones. The table has an auto number field so I only need records where Read.NextRecord > Write.NextRecord.

Sounds easy but I am just not sure how work this in FlowHeater
The administrator has disabled public write access.

Re:Append Only New Records 5 years 2 months ago #2159

  • FlowHeater-Team
  • FlowHeater-Team's Avatar
  • Offline
  • Administrator
  • Posts: 240
Hi Niel,

Usually you just switch on then "Insert data" and "Ignore existing records" options on the WRITE side. With this FlowHeater decide by the primary key on the WRITE side if the record exits.

Note: It´s necessary to connect the primary key fields from the READ side to the WRITE side.

In your case the "auto number field" can be a problem! Please post your table schema (READ and WRITE) and I could make a little example for you.

The problem with this method is that FlowHeater must check all 550k records on the WRITE side for existing! This can be take a while.

Forecast for the next Version 2.1.1 coming soon in October) Here you can use dynamical filters like this.
  1. Determinate the last ID in a previous SQL statement
  2. Use this id in the SQL select statement on the READ side like this
    select * from your_table where id > %ID from WRITE$ side%
Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.
The administrator has disabled public write access.

Re:Append Only New Records 5 years 1 month ago #2164

  • Niel Bullock
  • Niel Bullock's Avatar
  • Offline
  • New Member
  • Posts: 6
Has this enhancement been setup in the new version?
The administrator has disabled public write access.

Re:Append Only New Records 5 years 1 month ago #2165

  • FlowHeater-Team
  • FlowHeater-Team's Avatar
  • Offline
  • Administrator
  • Posts: 240
Hi Niel,

Yes this feature is released with the current version 2.2.1. There’s only a short time gap between new versions and the online help translation! Please excuse the delay.

First you have to define a parameter $MAX_ID$. You can find the parameter configuration dialog in the menu Edit->Parameters.

In this parameter the SQL Heater (new in version 2.2.1) stores the result from the SQL statement

Select max(id) from t_import

Note: The SQL Heater runs the SQL statement before the definition starts!

On the READ side you can now use this parameter to build a select SQL statement like this

Select * from t_export where id > $MAX_ID$

I’ve made a brief example; please have a look to the attachment.

Attachment dynamic_sql_filter.zip not found

Attachments:
Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.
The administrator has disabled public write access.

Re:Append Only New Records 5 years 1 month ago #2166

  • Niel Bullock
  • Niel Bullock's Avatar
  • Offline
  • New Member
  • Posts: 6
This works perfectly, Thanks!
The administrator has disabled public write access.
Moderators: FlowHeater-Team
Time to create page: 0.078 seconds