Is there a way to turn off the auto calculation? Im doing some pretty big conversions and I dont need/want it to run every time I touch something. I get the concept of why it does it, but man would it be great to be able to disable that while you are building.
1 Like
I have the same problem. I’d like to (have the option to) trigger the calculation and output manually.
There is a preference setting to delay the right pane processing if that helps at all.
1 Like
I have the same issue, but there is maybe a ‘cake and eat it’ solution. It is very useful to see the results of whatever operation you have defined at each step of the way.
If the system were to perform the newly added transform on the first entries in the data set and show those, you would be able to detect any obvious errors.
This would work because (except for the following caveat) entries are by definition serial, and so transforms would be applied to the right data in the right order as each new one is added.
However, it’s non-Markovian in that any given transform operates on the chain of preceding transforms, so for example it may be that an operation like ‘sample’ and several others would not work - I don’t know the internal mechanism. However, the ‘small set’ idea may work for a great many transforms.
Having this optional would be important, because for may smaller jobs it’s not relevant. Even with larger ones it’s an irritation when one makes a mistake, but objectively if I look at the figures, EDT is pretty efficient. Here’s some data of the transforms from one of my jobs
Reading a 640Mb input file
Dataset rows:
4,320,967
Dataset values:
56,172,571
Processing time:
19.68 second(s)
Spilt and stack
Dataset rows:
8,641,934
Dataset values:
86,419,340
Processing time:
6.144 second(s)
Looking up values in another file to enrich the first
Dataset rows:
8,641,934
Dataset values:
95,061,274
Processing time:
3.63 second(s)
Writing out the results file
Dataset rows:
8,641,934
Dataset values:
95,061,274
Processing time:
13.202 second(s)
I think they are pretty good timings, and liveable with, but the pause / or first n solution would make something good even better.
1 Like
I always switch off writing the output file until everything else is done - that helps
1 Like
Have considered using the Sample transform https://www.easydatatransform.com/help/1k/windows/html/index.html
To temporarily limit the amount of data you are using as you build and test your transform?
1 Like
The ability to turn off the automatic triggering of transforms is on the wishlist. Until then, for large datasets, I recommend (as noted above) that you use Sample to reduce to a smaller number of rows and disable output. You can disable the Sample and un-disable the output when you are ready.
See also:
https://www.easydatatransform.com/help/1k/windows/html/how_do_i_handle_large_data_sets.html