Column propagation issue

When you make changes in Easy Data Transform this can cause columns to be added, deleted or re-ordered. To keep everything consistent we need to propogate these changes downstream to other transforms and outputs. This can get very complicated, especially as the changes can depend dynamically on the dataset. We are aware that there are some cases where Easy Data Transform doesn’t correctly propogate these changes, meaning that some column parameters can end up referencing the wrong column. We are rewriting the core of Easy Data Transform to fix this issue. We hope to have new v1 release with this fix in the next few weeks.

In parallel we are also working on new features for v2.

3 Likes

Curious to know whether a [partial] workaround for the present would be to make column changes as late as is feasible in the chain?

I have not encountered any problems myself, being a recent purchaser; just wondering.

The column propogation issues are not very common. We would get the occasional report, but no-one was able to tell us how to reproduce the issue. We finally managed to reproduce the issue ourselves.

You can generally avoid these issue by not making changes to columns near the top of the graph. But sometimes you do need to make changes upstream, so we are keen to fix the issue.

This downstream propagation also hit filters hard .

Hopefully that will be fixed by the re-design. I hope to have something customers can try towards the second half of July. But no promises on dates.

1 Like

@Admin

Is it possible, that if changing anything upstream disturbs things in the down stream transformations, then those effected transformation could be highlighted? so it could be spotted easily for the fix, when one has dozen of transformation in the solution?

1 Like

Possibly. But I am not sure how that would work. Any changes potentially affects every item downstream. We don’t know what items would actually change until we know the change.

Note that we are still working on this. Progress has been slow and interrupted by holidays, but steady. We hope to make the changes available soon.

We are still working hard on completely rewriting column propagation in Easy Data Transform. A bonus of the complete re-design is that we should be able to ‘remember’ column selections between deleting and restoring connections. Here is a teaser:

Animation

There is still a fair amount of work coding and testing before we can release the changes though.

2 Likes

loving the teaser :smiley:

1 Like

We’ve been working on this for several months now. Nearly there!..

1 Like