@Prashant, another query, if I may. Are file names duplicated in any single run to populate the ERP, even though duplicates may occur over time? If not, then you can create a hash from a concatenation of file name with metadata such as $(CurrentDateTime) or $(FileUpdatedDateTime), or both. That way no seed entry is needed and source length is sufficient for the hash to work well. A million images is only 2^20 so allowing birthday conditions you should still avoid collisions. If that is at all unclear I can write an example late today (Oz time).
Always appreciated !! Thank You in advance
Using bits of my earlier procedure with the Regex from Anonymous,
Combine data unique to this run with current date & time, so the combination will be unique across all runs. Hash that and reduce to desired character count then arrange formats.
It appears you can take only one Metadata item. Stamp would presumably work as well.
Unique ID.transform (3.7 KB)
I did some experimenting and it looks like UUIDs/GUIDs are relatively straightforward to generate.
It uses DCE v4 (random) on Mac and Windows GUIDs on Windows. We donāt plan to support other variants of UUIDs.
The values generated are different each time you run the UUID transform and there is no way to control the values (no seed).
1 Like
WOHOO !! Thanks , I love the touch of Without Braces option.
If it is to be used as a unique reference, e.g. to anonymise data, is that not just a sufficiently large hash function of [some of] the data?
That would be pseudomized, not anonymizedā¦ anonymous data by no way can be decoded back to the original. That is not true for a hash valueā¦
1 Like
You can reverse a hash by generating all possible inputs and storing the hash for each. Then doing a reverse looked of the hash value to get back to the original. This is why passwords are usually stored as āsalted hashesā. But it would be pretty hard to make this work for unsalted hashes, if the input was going to be more than a few characters.
Hashes are repeatable by definition, or else no password verification would be possible. No-one asked for Argon 2id here, and that is merely a distance on the hardness scale.
My context was not even that level of pseudonymy given I also discussed retaining a separate (securely held) file for reverse lookups, no data-crunching required. That is a process for separating current researchers from personal knowledge less critical in the future, or for blinding in social or medical research where identification of the original respondent may still be useful (and acceptable) afterwards.
UUIDs (GUIDs) are now supported in Easy Data Transform v2.5.0. See:
2 Likes
We are no longer at the mercy of Random !! True successor to always random arrives 
1 Like
If you need to pseudonymize data, then please see:
Question.
So, say I need to create a GUID for each shift of an employee on a timekeeping file so that I can then reconcile off of that GUID to make sure I do not double pay a shift. Would I need a āmasterā file of all UUIDs from previous to lookup against for future files?
Currently, Iām using a concat of date,EID,timeofstart but this can still get kinda funky if I have an employee in the ER that works multiple shifts in a day starting at the āsame timeā. Which would sometimes try to filter out a shift I need to stay in the file.
I feel like there must be a better way than how Iām currently doing it.
Hard to say without knowing a bit more. But UUIDs are regenerated each time you process the transform. Maybe you can just supplement your current concat with Row Num and the file name from input Meta Info.
Why not add to it, shift name or however you define shift to it as well, this way, employee record will always be unique, even if he/she is working several shift and you donāt have to worry about double pay a shift and it will keep filtering simple as well.
If you can provide sample data and output you like, it would greatly help providing the solution or suggestion to achieve your desired result.
1 Like