Adding columns to flow will be removed by following Remove Columns Transform

Feature request
1

Hi,

if I somehow add a new column in an existing flow, where there is a “remove columns” transform later in the flow, the newly added column will be removed.

At least for me, that is not expected behavior, as I want to work with the newly added column…

Any chance for a change here?

Thanks, Jochen

2

are the new columns somewhere in the front part of the table? Keep in mind internally EDT works with column numbers and not column names so if you remove column “5” it doesn’t matter is this is now a different named column it will be removed. I had similar issues quite often. I strongly recommend to use schema when reading in files and set new columns at the end of the table. In most cases this helps.

3

The current behaviour of the Remove Cols transform is that it only passes columns that the user has checked. So, if you add more columns before the Remove Cols transform, they won’t be passed. What we should probably do is add an option to the Remove Cols transform to give you the option whether to keep or remove columns that have not been explicitly checked. For backward compatibility it would have to default to removing new columns. It is somewhere on the wishlist.

1 Like