Downstream Filters

I sure hope there is a fix for this madness. I will have a big transform set up and then realize earlier in the transform I need to add a lookup table that is needed for an additional data point. the problem is, when I do this it breaks all downstream filters, EVEN THOUGH the lookup column added to the data is far far past where the filters are. Lets say the lookup column become 25, but the downstream data filters are 13 and 16 and 20. ALL of those no longer have the filter calculation in them.

And while im at it. Is there a way you could use the X to remove one of the filters and not start over every time you tap it.

please let there be an answer to that first one. it almost makes this unusable.

I guess the real question is why do the downstream filters break??? that column its referencing is still there!! And if its not, just hold the data and let me make a small adjustment to the input field instead of for the whole freaking thing.

There are some known issue when you insert or delete a transform between existing transforms. I am planning to look into that today. It would be helpful if you could post a simple example that illustrates the issue.

I think you need to click ‘remove selected’ rather than ‘remove all’.

Thank you for the “single filter”. that was a dumb miss on my part.

1 Like

How would I best post an example? here is a partial of the one im working on. if you look at the top right, there are a lot of filters and lookups. if I break any link on the bottom left, even if I reattach, some of those filters go blank and the lookup loses the first value.

it would be nice if it just kept what it had until it saw the new data feed and then made the decision to break if the same requirements were not there. I would want that.

I strongly recommend not to break connection, if you can avoid it. Because that then nukes any column related parameters downstream (and it is hard to see how it could be otherwise). Instead try to delete and insert transforms without breaking the connections.

Also you can change the file and input item points to without deleting the input item.

To add a transform to a connection:


1 Like

Thanks. this helps a bit. What do I do in this situation? I added a lookup and some filters and then want the output of those filters to run back to the output. I have noticed that filters dont offer the + option to run an arrow out of them. Why is that?

It isn’t clear what should happen if you connect multiple transforms to 1 output. So an output item has a maximum of 1 upstream transform item (or input item).

If you delete the connection from ‘remove cols’ to your output you will be able to an input from one of the ‘filter’ transforms. But you have lost the original connection.

If you want to send the data from the ‘filter’ to a new output file, just add a new output with the filter selected.

If you want to send the data from the ‘filter’ to an existing output file, use a ‘stack’ transform to merge the various outputs and then output from the ‘stack’ transform.

Its honestly just being lazy. I get why you cant put many into one. I would break the top arrow and then use the top filter and drive that to the output. it just saves me from having to do all of the output steps (format, location, etc). its not a huge deal. I use the stack a lot and it makes sense why a filter would not allow an output as it could cause a rip in space and time.

whoa…nevermind. I just had the + pop up on the filter. weird, I swear it wasn’t their earlier!