Easy Data Transform v1.32.0 now available

Easy Data Transform v1.32.0 is now available for Windows and Mac. Find out more and download the release at:

Please let us know if you have any problems with it. Reply to this post with any feedback.

1 Like

Hello - I am experience repeated freeze/crashes with 1.32.0 on an ARM Mac M1. While using the app it will freeze and become unresponsive.
I’ve tried rebooting and reinstalling 1.32.0 from a fresh download.
Not sure if this is isolated or a potential issue.
Regards,
Mark

@mwb6
Is there any pattern to when these freezes occur? Can you reproduce the problem reliably?

After further review, I think that it is not a freeze problem but a very long delay - perhaps related to data set size. If I wait long enough the app will eventually become responsive again and function ok. If I don’t do anything for a minute or two - it appears to lock again and then repeats the delay process.
I’m working with a fairly large text file with 9000 rows but will experiment with smaller files.
Thank You,
Mark

9000 rows is nothing for most transforms. It should do 9000 rows in a fraction of a second in most cases. So it might be a bug. Can you please email the .transform file and inputs to us so I can investigate:

What do you do with the data?
I have from Time to time a problem, as the result of transformation is written into a file (Excel overwrite). Sometimes the file size grows (more than 5 to 10 times) of the necessary size. When this happens the process slows down, I do it most of the time in batch. After deleting the output file the transformation gets much faster. The logging in EDT shows which steps are requiring how much time. With this I recognised it was the writing and then I identified the above issue.

There is an issue with how data is stored by Excel. As I understand it, Excel stores a dictionary of strings. Every time we write to the Excel file this dictionary potentially grows, as it may now include strings that no longer exist. We don’t have any way through the third party library we use to shrink the dictionary. But you can do it by deleting the Excel file manually, before writing to it. Is this a big issue for you @Olaf ?

No real issue for me, after I identified what was going on. As said I do it in batch mode most of the time, so it is easy to add a file deletion in the script. I have some already as the EDT transformation create a dynamic set of files (acv) in addition to some Excels (about 150 files) each time.

Andy, does this dictionary effect happens with Excel Append, too?

Edit: I meant command line modus and not batch mode, evens it includes batch parts, too

Yes, every time you write to an existing Excel file, as I understand it.

The beach ball happens most times that I move away from the app and come back. I’d like to try to reinstall 1.31 - where can I download it from? I can’t find older versions on the website. I never experienced this before which is why I reported it. If I run into the issue with 1.31 I’m going to have to look for a problem on my Mac,
Thank You

You can access older version release pages via Easy Data Transform News .

E.g. Easy Data Transform v1.31.0 .

Hi,

For Excel file size and related issues, instead of output in Excel format, better to output in CSV format and then read it in Excel and save as xlsx format from Excel. This way the file size will be what it should be and hopefully no other issues.

That isn’t very convenient though, if you need to have it in Excel format. It is on the todo list to look into this a bit more.

Hi,

That was a suggestion for the mean time while you are looking into the fix, to have the process run fast and get the Excel output from the final CSV.

1 Like

@Olaf
There is now a fix for the issue where overwriting a sheet of an Excel file causes the file to grow continually. Please see:

The fix should also improve the efficiency of appending to sheets, because each string will now only be stored once in Excel’s internal string dictionary.

2 Likes

I will check. Version is loaded and my command line script is changes, so that the XLSX filles are not deleted any more before running the transformations. I will report my observations.

1 Like

I did execute the transformation about 5 to 7 times for my regular reporting as usual there are different xlsx files to be overwritten, between some k and 13 MB. So far I don’t see any size increase. But negative effects in the past came up after some days only. I will check further in the next days.

1 Like

Another very positive observation: By accident I deleted some of the Excel files I observing with the new version, which were generated with the 1.31.1. The were not growing with 1.32.2 but the fresh new generation with 1.32.2 generates for the old large files now ones which have only 1/15th to 1/20th of the size. So this seems to be a very efficient fix.

1 Like

Those Excel string tables can get quite big!