Before EDT it took long to get a good set of data in the appropriate format. Today, thanks to EDT, I have tons of relevant data, even sets of data I had not before. Hence, storing data -in a smart way- becomes slightly a problem. I use Excel/CSV to drop data (when I am lazy) and Access (for critical data). But the truth is I am not satisfied with both of them.
So my question is, do you know a software that is straight to the point, easy to handle, visual & powerful and… fun (like EDT is) not to ETL but to store data this time. A software that works perfectly with EDT to inject and retrieve data in an eyeblink. For my own usage. No matter to pay for it or not (as long as the price is in the same range as EDT). Any feedback is welcome, thank you in advance.
PS: I am not admin, so a portable version (like with EDT) is definitely a plus.
It depends a lot on what type of data it is and what you want to do with it.
In some cases it is sufficient to just store folders of CSV files, with each folder having files with the same schema ( column name and order) and with the folders and files having explanatory names so they can easily be searched.
This free tool is very good for searching file and folder names on Windows: https://www.voidtools.com/
(much better than Windows search, which is awful)
If you ever do find a perfect match , let me know .
EDT has been GodSent.
I also use ModernCSV quite a lot , it allows you to “PIN” folders aka database tables in one simple view in tree pane , very helpful . Works across OSX and Windows , major plus for me
I personally use SQLite databases for all sort of things, it is easy fast and conforms to latest SQL Standards.
SQLite Expert is a tool that is build to provide you the access to the sqlite databases with ease
You can read more about SQLite on www.sqlite.org most widely used database in the world, all your applications on mobile use sqlite behind the scenes to store you data.
I was thinking on suggesting @Admin to bring sqlite in the fold of EDT as well, and use sqlite to store transient data between transforms and it could make so many things simpler and fast.
I think the main reason to store intermediate data in a database would be to be more efficient with memory, so you can handle larger datasets. I don’t think it would be faster than storing in memory. Also SQLite is an OLTP database, optimized for adding data. It might be better to use DuckDB for intermediate data. DudkDB is an OLAP database, optimized for reading data.
Currently workflow being export to CSV , do operations & import back . This is what I do with tonnes of QuickBase related changes (taking care of primary key or [Record ID#])
See anyways EDT makes it very easy to run database like queries
You need to ensure your master/child relationship are OK then you can lookup using join , you can summarise using unique , pivot etc. I love the concept of doing flat files to “db” structure and vice versa.
You might take a look at GS-Calc & GS-Base at: https://www.citadel5.com/, they are both really inexpensive, very small in footprint and can manage many different file types (including SQLite) and the developer is very responsive to questions and open to ideas to extend his products. Tell him I sent you, i don’t get anything for that other than he knows who is spreading the word on these under-sung products.
Hello, in a first phase I plan to follow @prashant. I am going to use EDT in combo with (flat files and) ModernCSV (buying a basic licence). Because it is the fastest and the more versatile possible solution (a portable version is also available).
When departments around me have a sharp view on their own data structures, then only, I move towards a solution that is more specialized, such as suggested above.
Thank you gentlemen, it has been an interesting debate. Regards, Oliver
