error on json export

Hi Robert

I keep getting this error I have changed the value or size for each column. Im not sure if thats what it is talking about or the source / read size = 0 for each column, and json write to length to 999999999

I got the json to work fine in mongodb up until a file is 16mb or large which i assume is a limit in your software too?
Mongodb settings are now at 99mb max size for the html just in case. SO import into final database should be fine here. So far it has been except larger files. I have plenty of spare ram 46gb and terabytes of space.

 

Cheers
Brian

having issues uploading 665kb image file

Hi Brian,

Images are restricted to max. 1 MB and a resolution of 1920x1080 (Full HD), perhaps this is the issue?

Related to your JSON Adapter error. Could you please post your created definition and some test cases, Thanks?

Please note: A field length of 0 means no cutting or filling, the length is flexible in FlowHeater.

 

its mostly text and html files, the max size is around 9800 kb for one website.

I will upload all the files via email and you can test it. seems to have a wierd limit on memory usage for json. or some settings are wierd.

 

email sent with links to files, keep private if you can please. Thanks

Hi Brian,

Thanks for the example data, with this I found the issue. As soon as a fix is available I´ll get back to you here.

 

I will be testing out 87 million html files approx. out of 280 million records via 2700 csv files. 
So plenty of testing can be done. Will report anything in the future I come across with your software. Really needs to be put to a heavy test. Even maybe 8 copies running at once to max out the cores and ram. Should be doable in 48gb spare ram. In terms of a plan.

Hi Brian,

In the current beta version, the issue has been fixed. You can download the fixed version here: Download Beta Version

Now the JSON Adapter supports on WRITE site files greater than 2 GB, only 64-bit version!


 

yes its working... phew lol
Will test some more, mongodb seems to like the json just fine.
Will update later this evening.
Cheers
Thanks for the fix
Brian

Mongodb error with regex it works fine
without it, it doesnt. so by removing any or escaping : via regex
just one issue its massively slower.
2min 51 without regex
11 mins ish with regex.
If its compatible json it should by default handle : in the content, especially html files.
Should fix this then its 100% compatible json.
 

Regex setup, its already in the file.

it might also be the : in the column names on import from text adaptor. you should fix or have a means to remove special characters on import. removing such things. then it will auto connect the other side saving a tonne of time and mistakes.
testing now with them removed for both read and writes 

nope tested, same error on import for mongodb.
only with regex taking : out of the html or escaped can it import. 

Mongodb error with regex it works fine
without it, it doesn't. so by removing any or escaping : via regex
just one issue its massively slower.
2min 51 without regex  does not work with mongodb / json import
2min 55 with with regex. bracket : bracket * and replace with escaped \: does not work with mongodb / json import
17 mins ish with regex. bracket : bracket * and replaced with nothing. works fine with mongodb import as json
If its compatible json it should by default handle : in the content, especially html files.
Should fix this then its 100% compatible json.

removing the : from the columns test seems to work fine for mongodb.
Would be nice to have an adaptor for just mongodb (json) as its good to advertise with mongodb adaptor on your website too.
I tried other ETLs and they process at bloatware level
x etl 1 53 mins 100k records 
x etl 2 37 mins 100k records.
x etl 3 lost the will to live lol

Yours destroyed them all in performance, just some fixes here and there for text to mongodb / json would be awesome. the difference of 5 times faster like for like. is a good start but json issue 
for mongodb import should be fixed. Nothing out there to touch it unless we go full c sharp scripting.

 

yes the : in the write side is not causing the issue.
but the =" in the columns are, importing fine now
So the rel=" are the issues.
So on the json you should have a autoremove of "

 

I would strip those out only on export. That way the autoconnect works regardless, then auto removes the " and works out fine for json/mongodb

 

Hi Brian,

Thanks for your analysis. I´ll fix the issue with the JSON field name and the sign " with the next version.

Please note: The character “:” is a valid sign for JSON field names and of course also string content. This must not be quoted.

 

awesome, no regex, 5 min conversion per 100k records into memory.