FlowHeater - The Data Import / Export Specialist
See also the existing examples of using the MySQL Adapter Adapter, which may answer your question: CSV text file import , data migrating , exporting only new records

MySQL Transaction / Table Locks

More
7 years 5 months ago #2417 by Heinz
MySQL Transaction / Table Locks was created by Heinz
Hi there,

I want to request some more details about how the MySQL writeAdapter handles insert records.

Does it create a transaction around the entire batch when executed?
Does it lock the table for updates?

I am asking because I am having some problems when running large batch imports (50 000 to 100 000 records) against a table, other processes reading/updating this table seem to hit a transaction time out on a lock?

would you suggest to rather go the route of generating SQL script and running that script afterwards?

PS: Still loving the product!!
PPS: I hear XML is on the way - is JSON support in the pipeline?

Please Log in or Create an account to join the conversation.

More
7 years 5 months ago - 3 years 1 month ago #2419 by FlowHeater-Team
Replied by FlowHeater-Team on topic MySQL Transaction / Table Locks
Hello Heinz,

Usually FlowHeater creates one big transaction during the entire import/update process. This means the affected records are blocked for updates for other application/connections. Note: The certain behavior depends on the used database and the used transaction level as well.

Of course you could change the FlowHeater standard transaction behavior on the “Advanced” tab in the MySQL Adapter configurator.
Here you can disable the “Use Database Transactions” option in general for the import/update process, not recommended. This will degrees the database performance a lot.

A more better and recommended way for large batch import processes is to enable the “Perform Auto Commit after x written records”. In your case I would suggest 1000 (or 5000) records. This option should block other applications/connection just a few seconds (1 or 2) and the database import performance should be quite the same like before.



Yes, an XML Adapter is coming soon :) Below you find two links were you may download an early Alpha Version of V4. This Version can be used without any restriction as long V4 isn’t official released. Feel free to test it!

JSON support is planned for next year.

FlowHeater V4 Alpha 32 bit
FlowHeater V4 Alpha 64 Bit

Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.
Attachments:
Last edit: 3 years 1 month ago by FlowHeater-Team. Reason: change download links to alpha version

Please Log in or Create an account to join the conversation.

  • Alexander Deliyannis
  • Offline
  • New Member
  • New Member
More
3 years 1 month ago - 3 years 1 month ago #3544 by Alexander Deliyannis
Replied by Alexander Deliyannis on topic MySQL Transaction / Table Locks
Hi,

Found this thread while searching for JSON. I am wondering whether JSON is or will be supported.

Thanks in advance,

Best regards, Alexander
Last edit: 3 years 1 month ago by Alexander Deliyannis.

Please Log in or Create an account to join the conversation.

More
3 years 1 month ago #3545 by FlowHeater-Team
Replied by FlowHeater-Team on topic MySQL Transaction / Table Locks
Hi Alexander,

I´m afraid until now JSON is just planned but because less inquiries currently not implemented yet.

Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.

Please Log in or Create an account to join the conversation.

More
2 years 5 months ago #3587 by Gerhard Samstag
Replied by Gerhard Samstag on topic MySQL Transaction / Table Locks
Hi Robert,

I would also be interested in a json import adapter. Is it still just planned?

Many regards
Gerhard

Please Log in or Create an account to join the conversation.

More
2 years 5 months ago #3589 by FlowHeater-Team
Replied by FlowHeater-Team on topic MySQL Transaction / Table Locks

Best wishes
Robert Stark

Did this answer your question? We would be grateful if you provide a brief comment as feedback. It may also help others who may have encountered a similar problem.

Please Log in or Create an account to join the conversation.

Time to create page: 0.401 seconds

other Languages

de

FlowHeater Home

de en

Imprint/Contact

Privacy Statement

Copyright © 2009-2023 by FlowHeater GmbH.
All rights reserved.

Follow us on

twitter  facebook

YouTube

 de en