When you make changes in Easy Data Transform this can cause columns to be added, deleted or re-ordered. To keep everything consistent we need to propogate these changes downstream to other transforms and outputs. This can get very complicated, especially as the changes can depend dynamically on the dataset. We are aware that there are some cases where Easy Data Transform doesn’t correctly propogate these changes, meaning that some column parameters can end up referencing the wrong column. We are rewriting the core of Easy Data Transform to fix this issue. We hope to have new v1 release with this fix in the next few weeks.
In parallel we are also working on new features for v2.
The column propogation issues are not very common. We would get the occasional report, but no-one was able to tell us how to reproduce the issue. We finally managed to reproduce the issue ourselves.
You can generally avoid these issue by not making changes to columns near the top of the graph. But sometimes you do need to make changes upstream, so we are keen to fix the issue.
Is it possible, that if changing anything upstream disturbs things in the down stream transformations, then those effected transformation could be highlighted? so it could be spotted easily for the fix, when one has dozen of transformation in the solution?
Possibly. But I am not sure how that would work. Any changes potentially affects every item downstream. We don’t know what items would actually change until we know the change.
We are still working hard on completely rewriting column propagation in Easy Data Transform. A bonus of the complete re-design is that we should be able to ‘remember’ column selections between deleting and restoring connections. Here is a teaser:
There is still a fair amount of work coding and testing before we can release the changes though.
