Bartleby - Copy to multiple destinations - check file integrity and generate report

[Alexandre Westphal]

Hi everyone,

Just a quick message in case it can interest someone.

I am in a middle of a doc project that involve a loooot of copy/checksum to multiple destinations. AND I wanted to learn a little bit of Rust language. That being said, I also wanted to see how I could go with Claude Code as a learning tool on a practical software build.

And so I made Bartleby : https://github.com/colorgrain/Bartleby

It is still in a very early Beta stage but it works on Linux Mint (my main workstation) and on Windows on another partition of my drive. I didn't test it yet on Mac OS. The code was heavily written with the help of AI. It is under review now (my goal is to go out of beta this summer). If any of you want to give it a look, your welcome (the comments are pretty long and exhaustive but it is a part of the learning process). It is released under the GPLv3 License. Please, should you decide to have a look into the code, be kind and remind that I am not a developer

What the software does right now :
-Copy one source to N destinations.
-Possibility to queue several jobs
-Check (or not) the copy with MD5 or xxh3 algos
-Create reports in CSV, PDF or HTML
-Reports can be customized with Names, Contact, Logos and "color theme" to match one's company logo
-The interface was build to look as native as possible on the 3 OS (Gnome, MacOS and Windows) and to follow the light/dark mode upon startup.

You'll need mediainfo CLI and ffmpeg to generate the metadata of the reports and the thumbnails for the copied videos


Cheers everyone.
Alex

 

[Lois Plagnard]

Bonjour Alex,

I gave it a try.
I love the reports ! The only thing that doesn't work for me are the thumbnails. I do have ffmpeg installed and I run a Macbook Pro M2.

When it comes to feedback :
- Progress bar and average read and write speed seems to wait for at least one file to copy to get an accurate reading. If I have large files from long takes it might look like the app is stuck until the second file is written.
- For the interface I think many would prefer to have source on the left and destination on the right rather than on top of each other. But it's more a matter of taste

Other than that it's really great, I applaud you for making it open source

 

[Alexandre Westphal]

Salut Lois,
Thanks a lot for your feedback !!! I didn't have a chance to try it on Mac already, so I am happy that it is kind of working at least. I'll try to install it on a Mac but it likely won't happen before next week at best. I'll let you know what I can find about those thumbnails.
There is a beta 2 online now with some tweaks on the copy engine and the read/write speed display under the progress bar. Maybe it can change things... I don't know.
As I told you, the app is still in its very early days so...you can expect rough edges.
Don't hesitate to keep me in touch if you experience any other issue.
And for the UI, I can think about it (it is not the biggest change to make).
Thanks again !

Bartleby Beta 2 Release Notes:​

1. Performance: The copy engine has been optimized for faster transfer speeds.
2. Bug Fix: Resolved an issue in the conflict manager where selecting "Skip" for a file already present in one destination would incorrectly skip all files for other destinations during multi-destination copies.
3. Conflict Manager Improvements:
   The manager now displays the file size and modification date of existing files in the destination folder.
   Files are now skipped only if both the size and modification date match (using the original source date, not the copy date). This is particularly useful if a copy operation was interrupted unexpectedly.
4. New Feature: Added a toggle to copy the source folder itself into the destination, rather than just its contents. This is useful for workflows involving daily folders (e.g., sound recording sessions) where you wish to preserve the folder structure without formatting the device daily.
5. Controls: Added Pause and Resume buttons.
6. Logging: Logs are now saved automatically.
7. UI: Minor interface improvements.

 

[Tony Sal]

Alexandre,

If you aim is to target those who use media management apps on set you must have both a Windows and Mac OS version. Linux workstation are not widely used for on set media management.

I suggest comparing existing digital asset management (DAM) software such as Pomfort Silverstack and Offload manager, Shotput Pro, Yoyotta all software apps I have used for on set as well as in the color suite.

 

[Alexandre Westphal]

Hi Tony,
Thanks for your message and your advice.
My idea first was to make something I could use myself on my machine so the Linux version was essential. But indeed, my plan is to release Windows and Mac versions as well (that is why I choose Rust+Tauri).
The idea is definitely not to compete with Silverstack but more to build something that would stay simple , light and straightforward for backup /verification/report. Not sure I want to make it a metadata manager / file reviewing software and implement a library in the interface (too much work and this is the kind of stuff for which Resolve can be used maybe ?).
I'll keep you posted with future modifications/releases.

 

[Alexandre Westphal]

Hi everyone.
I just released a new Beta, with (Thanks to Lois !) and horizontal layout. Log is now an optional right panel. There is also now a "comment" section to add text to your job. And a bunch of improvements.
I'll dig into the MacOS version this week.

GitHub - colorgrain/Bartleby: Copy software with md5 verification and reporting tools

Copy software with md5 verification and reporting tools - colorgrain/Bartleby

github.com

Cheers.
Alex

 

[Tomislav Stojanovic]

Nice! Any chance to add an option to create MHL file?

MHL: Media Hash List | Improving Data Integrity in Digital Media Workflows

 

[Alexandre Westphal]

Hi Tomislav,
I am not familiar with MHL but from my understanding of their website it is checksum wrapped into and XML with additional information so it shouldn't be too hard to implement in the next Beta.
Could you tell me a bit more about your workflow with it so I can try to see the best way to make it useful (is it something you're often asked for ? do you just bundle it with backups or do you use it also to proceed to verification and do you need it to read it afterward ?).

The next steps for now are improving the progress bar and read/write display (which looks stuck during copies for now because they display average values for each file). And checking why Lois couldn't see thumbnails in the reports on MacOS.

Cheers,
Alex

 

[Tomislav Stojanovic]

Hi,

MHL is standard for all different offloading and backup softwares. For example:
- YoYotta
- Shoutput
- Hedge / offshoot


It is standardized format to white hashes to. Pomfor SealVerify and MediaVerify are free and can read MHL, calculate hashes and compare for whole folders... So pretty good to have it

 

[Alexandre Westphal]

Hi everyone,
Still in beta (and will stay in beta a while), but some bugs are fixed and a bunch of new features are shipped with the beta 7 on github : https://github.com/colorgrain/Bartleby/releases/tag/v0.1.0-7
Works for me on linux and windows. Installation should be possible via homebrew on MacOs: https://github.com/colorgrain/homebrew-bartleby
Didn't have a lot of time to do the testings on MacOS.

The app now support ASCMHL (creation/verification), thanks @Tomislav Stojanovic
If one of you own a Silverstack/Yoyotta/offshoot license and have time to test the MHL files generated via Bartleby, I'll be happy to know the results. I tried to be fully compliant with the specifications but ASC docs can be heavy sometimes and I might have missed something.

There is a verification windows that support all checksums and MHL manifests (including generational iterations).

There is also a left panel with 2 tabs : a file explorer (with a right click menu to add source/destination/create folder) and a preview of the folder structure associated with each job.

I've also added the possibility to create folder structures into the destination with presets (#presetname) and custom variables (%variables) including date, shooting day, project name or custom variables with several values possible (%CAM can be associated to CAMA, CAMB and so on for example). All of that can be set in menus or directly in a field at the bottom of each job card.

Regarding @Lois Plagnard issues with thumbnails not displaying on Mac OS in reports, I tried to solve this by adding ffmpeg and mediainfo as sidecar executables inside the app itself. It should prevent any path issue related with previous installs. There still might be signature issues of MacOS... I am not a specialist of this part but any feedback is welcomed.

Here are some screenshots.

Cheers,
Alex

 

[Manuel López]

I often have to back up projects, proxies files, renders, etc., usually to two external hard drives. I’ve been looking for an application that would let me do this simultaneously on two drives, but so far without success.
So I decided to give Bartleby a go. It’s not bad, but I had a bit of a scare at the start. I got a message saying the drives weren’t empty, so I aborted it straight away, just in case it deleted the content I’d already saved. I did a test on a drive with some unimportant files, to make sure it wouldn’t delete anything, and indeed nothing was deleted... I don’t know, I don’t really understand what that initial message is for, or does it perhaps expect that every time I make a backup, it should be done on completely empty drives...
In any case, I can see it works, but I’m annoyed that the additional files it creates (ascmlh, reports, manifest, etc.) have been left lying in the root directory; wouldn’t it be better to save them all in an additional folder within the copied folder?

Windows 11
Thanks

 

[Tony Sal]

I often have to back up projects, proxies files, renders, etc., usually to two external hard drives. I’ve been looking for an application that would let me do this simultaneously on two drives, but so far without success.
So I decided to give Bartleby a go. It’s not bad, but I had a bit of a scare at the start. I got a message saying the drives weren’t empty, so I aborted it straight away, just in case it deleted the content I’d already saved. I did a test on a drive with some unimportant files, to make sure it wouldn’t delete anything, and indeed nothing was deleted... I don’t know, I don’t really understand what that initial message is for, or does it perhaps expect that every time I make a backup, it should be done on completely empty drives...
In any case, I can see it works, but I’m annoyed that the additional files it creates (ascmlh, reports, manifest, etc.) have been left lying in the root directory; wouldn’t it be better to save them all in an additional folder within the copied folder?

Windows 11
Thanks

Manuel,

I recommend comparing DAM applications available from Imagine Products- Shotput pro (Perpetual license purchase), / Yoyotta (12 month or 1 month to 5 year subscription license purchase )/ Pomfort - Silverstack, Offload manager (temporary or yearly subscription purchase.). All these apps allow for downloading to LTO or multiple hard drive or SSD destinations. I have used all three of these DAM applications as each offers the user a choice based on their specific needs and requirements. I agree saving the reports files should be within the subfolder where the original camera negative (OCN) files are downloaded to as this is possible with existing DAM applications.

ShotPut Pro - Offloading video, audio, and image files

An essential tool for any media professional's arsenal, ShotPut Pro allows you to manage your files with ease. With lightning fast verified offloads and media focused PDF reports, you can ensure production stays on time and on track.

www.imagineproducts.com

YoYotta | Pricing

YoYotta create workflow software for production and post

yoyotta.com

Store - Pomfort

See the store to purchase Silverstack, the clever media management for film productions, and Livegrade Pro for professional Look Creation.

pomfort.com

 

[Manuel López]

Hi Tony.
Yes, I’m familiar with those programmes because I’ve also worked as a data wrangler. And they’re overkill for what I need—I simply need to back up the material generated by my work: dailies, proxies, VFX renders, masters, etc. I have a workstation with several internal NVMe drives and I regularly need to free up space. I usually back up the material to two external hard drives, so if I can do it simultaneously, I save time. That’s all there is to it.
At the moment I’m using BeyondCompare, a very good synchronisation tool that allows two copy sessions at the same time – it’s not quite the same as doing it simultaneously, but it works for me.
So, seeing this programme, which supports multiple destinations, has piqued my interest. The fact that it includes verification is also a significant bonus.
I know the primary purpose of the application is to download original camera footage, but I suppose it can also be used for any other material stored on a drive, or am I mistaken?
I’ve only run a couple of tests, and this software works well, seems fast, and with a few requests I’ve made to the developer, it could be a good solution for my needs.
Thank you.

 

[Alexandre Westphal]

Hi Manuel,
Thanks for your feedback !
Don't forget the app is still in an early beta stage and use it accordingly. Any testing like this is really helpful to understand how the app can be used, corrected and eventually be of some use to anyone.

Not sure to understand what you where trying to do. Did you want to copy files/folders to the root off a drive already populated with other stuff ?

Actually the warning displayed is intentional. As Bartleby is made mostly for offloading cards and backup media files, I thought in 90% of the case, anyone would use a dedicated folder to copy everything (by day, shooting date, media type etc...). Hence, the warning because an already populated folder as a destination is often the sign of an error in the destination path, or a previous copy that failed for example. But maybe the warning should be more self explanatory regarding what the software does.

Indeed, it doesn't delete anything. It will copy the new files along the old ones. The only case a second popup will be displayed is if there are files with the same names already in the folder. Then it will compare names/size/modification date and propose to cancel/skip similar files (and replace only the mismatch)/replace all. I think I'll add another option here for keeping all files and renaming the mismatch files (the new ones could have a suffix for example).

As for the specific folder for reports and checksum it is a good idea and I can implement it in the next beta (maybe as an option in the settings menu). But for the ascmhl manifests, according to the spec sheet (if correctly understood by me of course), they should be in an ascmhl folder in the direct path to the files that are copied/verified, not in custom parallel subdirectories. I think it is because it wouldn't be standard and transparent for other apps (impossible to find and read them for the next generations when performing new copies or verifications for example).

Cheers,
Alex

 

[Manuel López]

Hi Alex.
Thank you very much for your work; I really appreciate what you’re doing.

I understand that the tool’s main purpose is to download footage from a camera, but as I mentioned before, I think it can also be used for other tasks.
In my case, it’s simply copying folders between drives. There are loads of applications that do this, but I don’t know of any that do it simultaneously. When you need to back up, say, 500GB to two external hard drives, being able to do it simultaneously saves time, and if the copy also includes verification, so much the better.

I can see now that the warning only appears if you don’t tick ‘Copy Folder Itself’; it was my first test and I didn’t spot the option, and I also assumed that the source folder was copied by default. If you’re copying folders, you take that for granted.

One question: if you’re copying to multiple destinations, the verification is also done on each destination, isn’t it? I assume that if that’s correct, it takes longer.

I’ve made several requests on GitHub; if you could implement them, that would make the tool more efficient and versatile, in my opinion.
The folder for additional files is simply for better organisation. In my case, I don’t use ASCMHL, but I do use MD5 verification.

Thanks
Regards

 

[Alexandre Westphal]

Hi Manuel,
Wonderful, thanks for taking the time to run tests and to talk about it. If the app can be useful to different kind of users it is even better. That is great if you've made the requests on Github, I'll have a look at it as soon as possible.

Regarding the warning and the "copy folder itself" checkbox, let me explain a little bit more the logic of this as it was conceived first and I'll see what can be done to improve that behavior.
As you noticed, I started to develop the app when I had a lot of offloads and backups from different kinds of recorders/cameras recently (Sounddevice, Atomos Shogun, Sony camera, deity recorders and so on, for the same project). So for the Sony camera for example you might just want to copy what is inside the memory card (also because the card itself often has a useless name). And so you would add the PRIVATE and AVF_INFO folders into, let's say, a CARD 01 folder. In this case, the warning was just made so if you mistakenly copy these folders into the wrong (and probably already used) folder, you have the time to cancel the copy, so it prevent altering the folder structure of the card/drive. On the other hand, on the Atomos Shogun for example, the SSD drive is properly named SHGNXX, and is populated by files named after the same patern (SHGNXX_...). So in this case you might want to copy the drive itself "as" a folder. "Copy folder itself" only tell the software "don't take what's inside the chosen folder or drive, but the whole folder/drive itself". So obviously, as you are copying everything, there is no risk to mix up the folder structure of the card/drive with other files. Everything will be kept intact. And so, no need for warnings.

Regarding the verification, you are right. Everything is verified on the source, and on every destination. Every file is read sequentially, but the soft reads in parallel the source and the destinations. So in practice, you are mostly limited by the speed of your slowest drive (because the comparaison of checksums must happen at the end of the process).

Also note that for each plateform, the code forces the verification disk to disk (and prevent any verification from cache files created during the copy). In this release, the process is not optimum on exFAT drives on Windows (fix on the way) and the read/write speed in this case is now slow.

Just one question on my side : if you use MD5 verification, where do you put your checksum files ? Do you put them in a separate folder with a custom name, not in the same folder than the checked files ? And if so, how do you manage the files path in the .md5 file ? Is it an absolute path (I don't see a relative path working on this case) ? (if so the issue is than if you happen to copy the folder to another place/drive/computer because the path wouldn't be good).

Thanks again,
Alex

 

[Manuel López]

Yes, I understand. You’re right, the verification file can’t be placed in another folder if that prevents it from working properly.
But in my case, where I only copy folders, it can be placed inside the copied folder, without the need for an additional folder.

At the moment, using the ‘Copy folder itself’ option, those additional files end up outside the copied folder; I think they could be stored inside the copied folder without causing any issues for the verification file, or am I wrong?

I have a script (created by ChatGPT) that I use to generate a manifest.md5 file, which verifies any file or folder located in the same root directory. For example, I have a folder called Material_01, and inside it other folders and files; the manifest.md5 file in the inner root of Material_01 verifies everything inside the folder perfectly, without any issues.

Therefore, if organised within another folder, that wouldn’t work; but if placed in the root of the copied folder, I think that would be a better solution than what we have now, where these files are left outside the copied folder.

I hope this makes sense to you. Apologies as English isn’t my first language and I may not have explained myself clearly.
Best regards

 

[Alexandre Westphal]

Hi Manuel.
Thanks for your input again.

I am too busy to dive into this during the next two weeks but I'll work on this to give more choices in the next beta for sure

Cheers,
Alex

 

[Lois Plagnard]

Salut Alexandre, I've tested your latest release.
The thumbnails is now working properly on my mac. The verify function for the checksum always returns missing hash errors, not sure if I'm doing something wrong.

I think as a simple software for DIT use on set it's very good, and it doesn't need to compete with the major . From my point of view to implement in our workflow the most critical would be the stability and keep working on the interface/ease of use. You could add things like estimated time remaining, notification when a job is done, etc...

If I can I'll try it on our next shoot so I can see if it works properly.

I'm also curious about the speed of the phase two checksum. For all I know I think some backup software (can't remember if it's Silverstack) are able to perform at least a part of the checksum during the copy itself, that speeds up the verification once the copy is done. It might be worth experimenting ?

Best,
Loïs

 

[Alexandre Westphal]

Bonjour Lois,
Thanks for your tests. Cool to know that the thumbnails are there on MacOS too. Adding ffmpeg and mediainfo as sidecars seems the reasonable option.

As for the verification failing... it is very likely a path issue. Can you open the checksum files in a text editor to check if the path to the files is correct ? It should be a relative path starting at the same level than the checksum. (that is also why the checksum file cannot be moved).

I'll check the code when I'll have time to see what happens on MacOS.

Regarding the speed of the verification process, I'll have a look when the features will be locked for a while. The main thing I wanted to avoid at first is to read the files for the verification from the buffer to ensure a true disk to disk process (avoiding false positive with an issue occurring between the buffer and the disk when writing files).

I agree with you regarding the improvements to be made.
Beyond general stability of course, notifications are on the list (they are already here on the OS, but on the phone, it would be a great addition I believe).
I'll work on the checksum/report location and naming (though it raises organizational challenges)
And some interface enhancements (remaining time / a source size indicator instead of the source drive size, thanks to Manuel's idea etc...).

Cheers,
Alex