Re-enabling columns in an early “Remove Cols” step affects downstream transforms

[Phil]

Re-enabling columns in an early “Remove Cols” step affects downstream transforms

Hi everyone,

I’m using Easy Data Transform 2.13 and have run into an issue when modifying an existing workflow.

I remove unnecessary columns near the beginning to keep processing fast. Sometimes I later realise that I need one of those columns and re-enable it in the early Remove Cols step.

After doing this and then selecting another column, some downstream transforms seem to lose or change their column selections, affecting the results.

Keeping all columns until the end would avoid changing the early selection, but that comes with a performance cost for my datasets.

Is there a recommended way to re-enable a previously removed column while preserving the existing downstream settings?

[Olaf]

I would upgrade, ether was some fix on downstream topics in 2.14. Afterwards I would restart with the transform file where you had the “wrong” removes in and retry to keep/reactivate the former disabled columns.

If you have a unique key in the table you can alternatively try to lookup the removes column or use join.

[Admin]

@Phil

After doing this and then selecting another column, some downstream transforms seem to lose or change their column selections, affecting the results.

Do you have a specific example of this we can investigate? Ideally with a small dataset.

Also, I suggest you update to v2.15.0.

@Phil

Keeping all columns until the end would avoid changing the early selection, but that comes with a performance cost for my datasets.

We recommend using Sample to remove rows if you want to improve the performance of large datasets, rather than removing columns. You can check disable sampling when you want to disable sampling to get the full results.