Using file wildcards in batch mode may not cover what you want to do. In that case you might have to do it in 2 steps, for now. E.g. use the batch feature to aggregate all the data into a big CSV. Then create a second .transform to process the big CSV.
Perhaps you could provide a bit more detail about what you are trying to do.
Hi
I’ve already given the detail in the quote below or do you really need more?
Just making sure I understand correctly. So you need to aggregate all the JSON data files into 1 dataset before you can do the Gather and/or Spread?
I suggest again please take a closer look at the link I provided for a similar “almost essential within workflow pipeline” iterator functionality available in ArcGIS modeler.
It’s really not the same-same as the current ability in EDT to only be able to iterate/batch as a wrapper the whole transform.
I realize being able to iterate within an element of a .transform not the same as iterating the whole .transform. But I want to ensure I understand the problem correctly before I consider any solutions.
Other way round… I use Gather and Spread to parse the multiple json files to first one big CSV. Works better than trying to use Stack alone when you have tens of thousands of files with many different field headings.
But also this was just one general example of where a “within-transform iterator” operator would be useful…
Having such a capability could potentially be a helpful alternative in any situation where EDT users currently have to resort to the batch mode.
A “within-transform iterator” also potentially adds a lot of complexity. But I will have a think about it.
Many Thanks.
I suggest maybe also have a look at the functionality of the
Bubble.io equivalent plugin iterator function available here:
Iterator Plugin | Bubble
Or the Mendix loop function here:
https://docs.mendix.com/refguide/loop
Essentially this copy request is asking for a way to either embed or link transforms. Basically the ability to “nest transforms…”
For example, if EDT had a processing step to “Allow jumping to and automatically running another outside transform and then returning the result to continue” then that would be great.
Idea is to allow the user to be able to establish a “Library of useful mini-transforms” that could then be referenced and re-used.
For example:
I have several “EDT transform branches” that I typically use again and again in a lot of different scripts. At the moment I must manually edit and re-create those standard “transform branches” from scratch each time I need them in a new transform script.
It would be great if I could either:
Copy-and-paste the needed “transform branch” into the new transform from my existing transforms whenever needed.
Or save that “transform branch” as a ready-to-run transform in a library
and then have an EDT operator that could find it in the library and run it and return the result.
Essentially wishing for the capability to be able somehow to package needed EDT mini-branches as a “new operator” that could then be re-used and dropped into a transform wherever needed allowing users to create their own re-usable EDT processing operators.
A “Master” EDT Transform could then be created that simply kicks off a series of stored library mini-transforms in the right processing order.
Onwards and upwards 
Being able to copy and paste a group of items from the center pane between different .transform items would certainly be useful and is on the wishlist already. But note that any column-related options in the transforms would be unset, unless you also copied and pasted all the inputs the transforms were connected to. Also there are a lot of other details to work through.
Once we have copy and paste working it should also be possible for a user to build up a library of standard transform sequences. E.g. select a sequence whitespace → case → filter and copy it into the library as “cleanup”. You can then copy that sequence from the library in a click or two, much as you would add any other transform. You would need to be able to edit/manage this library and possibly move it from one machine to another.
Being able to embed one .transform inside another is an interesting idea that we have thought about a bit. I don’t know that it would be practical to pass the data directly from the parent .transform to the embedded child .transform and back. So it might be down to the user to ensure that the input files for the child match the outputs from the parent. Or maybe we could do somethign with aliases, as we do with batch processing. But that is starting to get complicated.
Currently we are concentrating on smoothing off some edges on existing features before adding any major new features.
Support for Applescript and Javascript for Automation would be great. ( I realize you support command line operation and for the most part that can accomplish the same thing - but Applescript or Javascript would be simpler and/or more powerful.)
I will make a note of the request for Applescript/Javascript automation. However this would only work on the Mac version. I’m not sure what the closest Window equivalent is.
I think the Replace transform is the one I would most like to copy between transforms. I have a conversion that is pretty easily done on an output from an IP address that translates to a site name. It would be great to not have to rebuild it for every transform. I’m not exactly sure I need it every now and again so using a template hasn’t been effective but I’m going to try to start doing that. just being able to copy all the replace data would be awesome.
@kozborne
Are you aware that you can copy terms from one Replace and paste it into another?
Also you might be able to use a Lookup transform, instead of Replace.
was not aware, will start using. and will evaluate the lookup transform. Thank you!
Lookup might work better for you. You can use a separate input file for the lookup values. But note that it requires an exact match, whereas Replace can replace part of a string.
You are right Lookup is very nice and will solve several of my needs, the copy function is also very nice.
I see I never replied! For the record, I love the notes feature, I use it all the time and find it really useful when revisiting old projects. Thank you.
As a feature request or suggestion, it would be fantastic to have a mechanism similar to Excel’s functionality for multi level pivots. For instance, the ability to drag and drop the pivot column (or row) for easy organization, and the option to nest aggregations for a more flexible and intuitive data summarization experience. This could greatly enhance the efficiency and user-friendliness of the tool. Thank you!"
