Dynamic data import / export

datenbank

simply more productive
Up to 10,000 records can be processed with the freeware version!
Welcome, Guest
Username: Password: Remember me
Welcome to the FlowHeater support forum.

Questions about using FlowHeater are answered here.

TOPIC: Fbatch extremely slow

Fbatch extremely slow 2 years 2 weeks ago #2338

  • rajeev
  • rajeev's Avatar
  • Offline
  • New Member
  • Posts: 4
:(
I used Fbatch to batch upload CSV files to a table of size about 50 MB and used go smoothly.
Recently, I used Fbatch in the same way as I used and the file of 50 MB to be uploaded to PSQL, but this time it took about 7 -8 hours to upload for one file that used to get uploaded in 15 to 20 mins maximum. What could be wrong.
I need to upload about 30 file for each month x 4 to the psql and I am trying to figure out the issue but not sure.

thanks in advance
The administrator has disabled public write access.

Fbatch extremely slow 2 years 1 week ago #2339

  • FlowHeater-Team
  • FlowHeater-Team's Avatar
  • Offline
  • Administrator
  • Posts: 240
Hi Rajeev,

I guess the issue is related to the PostgreSQL transaction behavior!

FlowHeater usually use one big database transaction to import/update records in a databases. Just for PostgreSQL this isn’t sometime too much and PostgreSQL couldn’t handle big data import or updates as fast as expected. I’ve never seen such database transaction behavior in other databases engines. Currently I’m searching for perhaps a PostgreSQL option to improve the performance during large data imports or updates.

To prevent such slow PostgreSQL imports/updates you have to configure the FlowHeater PostgreSQL Adapter. Open the PostgreSQL Adapter configurator on the WRITE side and switch to the tab “Advanced”. Here are two possible options possible.
  • You disable database transactions
  • You configure the PostgreSQL Adapter to perform an database auto commit for example every 5,000 records.


Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.
The administrator has disabled public write access.

Fbatch extremely slow 2 years 1 week ago #2340

  • rajeev
  • rajeev's Avatar
  • Offline
  • New Member
  • Posts: 4
I did not get the first step correct. the picture shows tick on the use database transaction, but you suggested ' You disable database transaction'.
The administrator has disabled public write access.

Fbatch extremely slow 2 years 1 week ago #2341

  • FlowHeater-Team
  • FlowHeater-Team's Avatar
  • Offline
  • Administrator
  • Posts: 240
The two options are separately.

It´s possible to disable the use of database transactions for this Definition, this I do not recommend.

The better way to do this is the second option. Do not disable database transactions and configure to perform an AutoCommit after 5.000 processed records. I think this will improve your PostgreSQL performance in this case much more. The posted picture shows this option.
Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.
The administrator has disabled public write access.
Moderators: FlowHeater-Team
Time to create page: 0.077 seconds