bulk editing of FHD format

HI Robert
Just a wierd one.
I have a folder with 2000 folders, each is the same exact name as the csv files which are also 2000 files.
Is there anyway to duplicate a fhd file but iterate the settings within the file
file 1.csv
folder to search is
file 1
output is
file 1.json
then the contents of the folder have other different named folders but still the same structure file x.csv = folder x + file x.json
It would be handy to create 2000 + fhd files then with the iterated settings 

 

Hi Brian,

You just have to use FlowHeater Parameter instead of static names 😊

With this you can create dynamically definitions. Like I mention in other question, I miss some detailed information to create a brief example how it works.

For example)
“root directory”
   “file 1”
   “file 2”
   “file x"
   file 1.csv
   file 2.csv
   file x.csv

In this case it would be very simple to procced all CSV Files with the Batch Module like this

    FHBatch.exe /ReadFileName C:\Data\File*.csv convert-csv-to-json.fhd

That’s all. As soon as you provide detailed information I´ll create a brief example.
 

read side = input file name
the file block in the interface which searches the folder X which is linked to the file name (so dynamically changes as you have done)
and finally the write output is same file name for read just json adaptor and file X.json

That would save alot of work making 2000+ of these files. lol

Hi Brian,

That’s quite simple

1. Get the current file name from the READ TextFile Adapter with the System Heater
   
2. Extract just the file name without extension with the Path Heater
   
3. Set a FlowHeater Parameter $output-filename$ with the Set Parameter Heater
   
4. Take this parameter as part of the JSON file name into the JSON Adapter

Please find attached a brief example.


 

 

cool.
what about the input, is it another set? and named from the set. I have 2000 iterated csv files, file 1.csv file 2.csv etc etc
and how do you launch this ? CMD?

the idea is to take 2690 files 
assign a batch of 336 per batch
run 8 copies across 8 cores.

dynamically read from a list of file locations split up into 8 files containing 336 csv files 1 - 336, 337 to etc (8 files)
output dynamically the input file names as json (done now)
so was thinking of 8 different text files, 336 files listing them x 8 batches making for 2690 files in total
max 100k rows with each csv file
then run batch part x 8 across the pc. or at least 4 anyways. 

 

Hi Brian,

Like I mentioned before. The Batch Module take care about the file name and change the file name into the definition during the execution.

You have to use Wildcards to start several batch jobs, e.g.

   FHBatch.exe /ReadFileName “C:\Data\File 11??.csv” convert-csv-to-json.fhd
   FHBatch.exe /ReadFileName “C:\Data\File 12??.csv” convert-csv-to-json.fhd
   and so on
 

So as long as the output, file location for the html is all done.
the actual cmd dictates the input file name?
and then it should do 100k batches.
Does this scale with cpu cores? its for later on as we have 4 machines = 32 cores (or 32 batches), approx 1.6 billion domains to do... (51tb ish of html data)
 

FHbatch.exe /ReadFileName /MaxRows 100001 "H:\excelfiles\file 1?.txt" file 2.fhd

Is this correct
file location is already in the fhd file and the output is set as $output-filename$.json

Does the cmd have to be in administration mode?

H:\FlowHeater V4\BIN>FHbatch.exe /ReadFileName /MaxRows 100001 "H:\excelfiles\file 1?.txt" "H:\excelfiles\file 2.fhd"

FlowHeater (R) Batch Modul (64 Bit): Version 4.4.1.1 Beta2
Object reference not set to an instance of an object.

H:\FlowHeater V4\BIN>

What am i doing wrong here. cmd is in admin mode

H:\FlowHeater V4\BIN>FHbatch.exe /ReadFileName "H:\excelfiles\file 1?.txt" /MaxRows 100001 /WriteFileName "H:\excelfiles\file 1?.txt" "H:\excelfiles\file 2.fhd"

FlowHeater (R) Batch Modul (64 Bit): Version 4.4.1.1 Beta2
Illegal characters in path.

the json output is $output-filename$ in the fhd.

Im confused on 
A the order or a correction of the above
B do i need to call the SET, variable from the FHD as the input file name = output file name.json. 
FILE in the FHD matches the folder of the html files from the name of the input file
Cheers

Hi Brian,

The option "/WriteFileName" doesn’t support wildcards, this is just for READ. For the WRITE site you have to use FlowHeater Parameter instead.

I don´t think you want/need the option "/MaxRows"? With this you limit the processed records at all. This option has the same effect like into the Designer , see screenshot below.

Yes, the Batch Module take care about the current processing file name. In case you use the System Heater and the Set Parameter Heater to get/set the folder name, like mentioned, everything should work

See also your other topic: excel to sqlite3

 
 

do these wildcards support [1-9][0-9] 
for example 1?? = 1 to 199
2?? = 200 to 299?
but how about 1 to 99?

Hi Brian,

Only the common Windows OS wildcards "*" and "?" are supported.

You may use
   File-?.html      to process 0-9
   File-??.html    to process 10-99
   File-1??.html   to process 100-199
   and so on.
 

the 1? i get 11 files 1 + 10 to 19
the 1?? = 1 10 1 to 99 11. i get 111 files
something bugged there.

1?? should be 100 to 199 
FHBatch.exe /ReadFileName "H:\excelfiles\File 1?.txt" H:\excelfiles\1csv.fhd = 11 files (should be 10 to 19) its 1 + 10 to 19
FHBatch.exe /ReadFileName "H:\excelfiles\File 1??.txt" H:\excelfiles\1csv.fhd = 111 files should be 100 to 199
FHBatch.exe /ReadFileName "H:\excelfiles\File 2??.txt" H:\excelfiles\1csv.fhd = 111 files should be 200 to 299
 

Hi Brian,

Looks like a general Windows OS handling issue. The behavior is just the same like into the Windows console. You could test it with dir "File 3??.txt" in the CMD.exe console window.

Workaround) You may process first all files “File ?.txt”. Afterwards delete these files and proceed with “File ??.txt”, and so on.
 

it would be better in [1-9] = 1 to 9
[1-9][0-9] 10 to 99
[1-9][0-9][0-9] 100 to 999
[1-9][0-9][1-9][0-9] 1000 to 9999

[=2][0-9][0-9]
200 to 299
[][0-9][0-9]
300 to 399
=2 = non sequential or fixed single digit number and each group with x-x = sequential series of numbers between 0 and 9
etc
add those switches for precision and isolating groups with ease
or better how about a batch scheduler interface allowing to assign independent series of files depending on max core amount of cpu and available ram.
I run 6 copies at once, and each does say 200 files. Might as well make the most of multi core. 
15 days (single core) vs 3 days (multicore)
 

Hello Brian,

I'll put the request on the list of possible new features. However, I can't yet say when this will be implemented.