External HDD preparation best practices

LUKS/LVM from GParted (as well as setting up passphrase protection) seem to be available from GParted.
http://thesimplecomputer.info/full-disk-encryption-with-ubuntu

tl;dr:
I'll probably partition the drive, but strictly for data organisation (video, documentation, pictures, audio).
I'll decide the size of each partition in relation to my existing data.
Yet for sure, backing up the computer's data, I'd rather do that on a smaller drive (more writes, it feels safer on a smaller drive since less data/memory is involved). And I'll definitely use a passphrase. Last but not least, I'll have to set up the partitions to be resizable.

Questions left:
I wonder if I should keep, I don't know, a third of the drive for backing it up on another partition on this same drive.

Also, ext4? btrfs?

And defragmentation: should I worry about it?

Also, do you know any cataloguing software? Should I use one in that case?
It looks like Libo Base is an option, but I don't understand how stocking data in a drive (which will be unplugged most of the time) and a database and a way to retrieve that data (yes, I don't know exactly how this works) is better (faster, cleaner, safer) than looking for it from a standard file manager.
Also, does that mean that I'm wasting time organizing my data into folders on my PC?

We are not talking about a disk that will contain a operating system, right?

To organize the files there is something called directories (aka folders), which do not bring any partitioning issue. File searches can be restricted to a directory.

One directory can be for the backups, e.g., with Trisquel's "Backups" tool in the "System Settings". However the backups may end up occupying the whole disk. "Backups" does that (and automatically removes the older versions of the files when there is a shortage of space). That said, since it backups in an incremental way, it may take quite some time to fill up the whole disk. Well, it depends on how much data you back up and how often those files are modified...

Anyway, backups may end up filling the whole partition (it is "normal"), it makes sense to have a separate (and large enough) partition for files you want to carry, besides the partition (the rest of the disk) for the backups. And I am talking of backing up files from another disk (and the external can be backup on the other disk: "cross backups"). Indeed, although you can backup on the same disk, it is not particularly safe: the disk (the hardware) can fail, suffer from an accident (it falls), it can be stolen, etc. In such eventualities, the data on all partitions are lost at once.

Modern filesystems do not suffer from fragmentation problems until they are almost full. You can choose any filesystem and they will be readable from any operating system with a kernel that supports the type of the filesystems. XFS is often said to more efficiently deal with large files (megabytes already is considered large). However XFS cannot be shrunk. That would be a problem is you choose to have several partitions but oversize a partition with an XFS filesystem and need more space for another partition. Btrfs is efficient but it still is rather new: the probability to lose data, although very small, is higher than with a more mature filesystem type such as ext4. That said, it is users people do not adopt btrfs that it keeps on suffering from this novelty issue (it actually is not that new anymore).

So, if I were you, I think I would go with two partitions: one for files to carry (I do not know how much you want to carry but you had better over-dimension than under-dimension that partition), organized in folders, and the other partition (the rest of the disk) dedicated to the backups. Both can be formated in ext4 but you may prefer NTFS for the first partition (so that you can share files with Windows users).

I'llsave the really important data on some DVDs

Notice that DVDs are not eternal either.

Why would a cross backup matter since I have the original data on the computer's disk anyway?

If all files on the disk are stored on another disk as well, it does not matter. I thought that you wanted the disk to have the only copies of some large files and a backup of these same files on the same disk.

I want one of the smaller drives to be used as a backup mainly, and partly as a storage extension

If it is smaller, you will probably not have enough space to backup the larger disks. Backups can be compressed ("Backups" compresses by default) but, files on personal computer (movies, pictures, music, LibreOffice documents, etc.) usually are in compressed formats. As a consequence, basically no space is gained by compressing what already is compressed.

But how much space to leave for the backup partition? Should I double or triple the amount of disk space I expect to backup?

That looks reasonable. With an incremental backup ("Backups" does that by default too), you will be able to go quite far in the past (recover files removed quite some time ago or as old versions of files that were modified).

Yes, I thought folders would be enough, but reading about collections managers (basically database management with indexing if I get it right), I thought this might be faster/more convenient.

Well, that may be my ignorance speaking but I do not believe that searching in a folder would be faster if that folder is alone in a partition. I am talking about a search restricted to the folder in question of course.

For backup, I need to find out inside the software I will choose.

With "Backups", it is a box to check. But if you encrypt the partition, it looks useless to also encrypt the backup.

Ah sorry, I didn't explain properly:

A is my PC HDD, with the OS within, and some data of course.
B (as in "Backup) is the external backup drive, either the same size, or a bit larger than A. It's smaller than C though.
C is the huge HDD used for storage, unplugged from the PC most of the time.

-

So there are two separate ideas, because B should have the same files as A since it backs it up (so it's ok if either one crashes), but for C, I was thinking of the DVDs for backing up a few specific files. Though these files will probably be on A and B too, so I guess more is really overkill.

I was considering indeed (at first) backing up all the files of C inside itself, and you didn't recommend it (plus It would reduce the space for data by half). But it's true that DVDs don't last for too long.

-

Well, that may be my ignorance speaking but I do not believe that searching in a folder would be faster if that folder is alone in a partition. I am talking about a search restricted to the folder in question of course.
Yes, I too doubt partitioning would speed up anything.

-

With "Backups", it is a box to check. But if you encrypt the partition, it looks useless to also encrypt the backup.
Ah, that's simplifies things, thanks for the info.
DejaDup has some bad press though (maybe because of the user, but still): https://www.reddit.com/r/Ubuntu/comments/4blx2o/should_we_replace_deja_dup_as_the_default_backup/

The only bad thing I found about Back In Time for now is this:
There's a "gotcha" with backintime - "dot" files are excluded by default. If you want your home directory's dot files, use backintime's Settings->Exclude and remove .*
https://askubuntu.com/questions/2596/comparison-of-backup-tools

EDIT: It seems it's been fixed (https://github.com/bit-team/backintime/issues/315).

-

So I suppose I should format C as ext4 with encryption.
I should format B as ext4 (no encryption) and set the software to backup on B, with encryption from the software.
And maybe get a few files on DVD just in case the 3 HDD crash all at once (obviously highly unlikely).

That means a whole bunch of passphrases. I probably need to learn how to use gnome-keyring and Seahorse so I don't have to memorize them all.

Did I miss something? Is there a better idea? Is there a way to backup C as a whole, like using yet another huge drive and clone it there periodically (that seems quite complicated)?

Just to say there is a preacution while formatting SSD hardrives, if you plan to buy one. Mine is broken since I formatted it in fat32. Now it takes five hours to copy 2Go of date from the disk to the computer.
I've read somewhere you must format that type of disk with care, letting some space in the partition, or upload the firmware or something like that, I can't retrieve the page.

Thanks Sasaki, definitely worth knowing.

About encrypting an external drive, it seems gnome-disks is a tool to use, not GParted (that previous link I posted was about an OS install): https://help.ubuntu.com/community/EncryptedFilesystemsOnRemovableStorage

I have a hard time finding tutorials about using gnome-disk-utility (aka gnome-disks, also disks I think).
There's one small thing that worries me a little, it's that at first I had one unique partition, and now I have two.
It might be normal since here (https://askubuntu.com/questions/805485/how-does-luks-work) it's talked about LUKS bringing a layer.
and in the gnome-disks window, I have indeed the top layer with a lock symbol.

Here's what I did:
1. I erased existing partitions, deleting the manuals that were in there (all pretty much useless plus I backed it all up just in case).

2. Having one partition left, I tried the menu "wheel" icon at the top, I assume it's for the whole disk, sinche there's the same wheel when selecting a partition, so that's inside the window, not at the top.
Formatting indicated a progress bar. But no LUKS, so I canceled it.

3. Then I tried the same format action but on the unique partition, from the menu inside the window. There I could select the LUKS+ext4.

-

The only indicator that something is going on is a small waiting wheel (like the mouse indicator) next to the drive name in the left column.
But I know it will take about 4 days, because there was a progress bar when I tried at first to format only.

So I'm hoping this is going well.

It's been 5 days now, without still no indication of how long is left for formatting/encrypting.

It's basically this issue: https://askubuntu.com/questions/19049/disk-utility-running-for-a-long-time

Any idea?
For now I'll leave it running.

For now, it is unlocked, and when I try to lock it from disk-utility, I get:

Error locking /dev/sdX: Command-line 'cryptsetup luksClose "luks-bunchofalphanumericalcaracters-andsomemore" exited with non-zero exit
status 5:device mapper:remove ioctl on luks-samealphanumericalnumbers failed: Peripheral or resource busy
-
and then from "device-mapper" it loops.


EDIT:
It might be linked to this https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/484429

I know why I can't unplug it or lock it, because the formatting is probably still going on. the problem is that I don't know for sure it's still going on, and for how long.https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/484429

htop shows 4 gnome-disks processes, 2 indicate 0 in the TIME+ column, one seems always stopped and shows 12 hours, while the remaining active process shows 65 hours and starts and stop every few seconds, irregularly.

I syuppose it's still working, but it's only a guess.

SMART data shows zero issues on the drive btw.

I can't seem to find any log about gnome-disks, that would hopefully say if everything is going fine or not.

Ok, so I ended up stopping the formatting, and did a new one (but fast this time) on the whole drive, and did it again with LUKS+ext4. Plus it's a brand new drive, so it should be more than fine.

It's troublesome that it keeps going forever though, and also that there's no progress indicator/bar/%.

anyway, this problem aside, I tried to use the CLI to first unlock it, then mount it, then unmount it, then lock it back.
A fine example of wasting time on the CLI (plus I couldn't make it work, with more than half an hour of research).

So I tried on Nautilus. Prompted for my pass phrase, worked.
End of the story.

Just make sure to clic on "immediately forget the passphrase", because if you don't, you'll be able to unplug it and back without the PC asking for the passphrase (at least until you restart the PC).

So in a nutshell, I had troubles with slow formatting.
Maybe I should have done it on the whole disk first instead of from the partition.

Also, on my Trisquel netinstall, I don't have the authorization to open the external HDD.
I don't have any other software blocking this.
I'll try again after restarting.

EDIT:
For those interested, this worked (the first line, about unlocking. I could then mount with a file manager).

Also fdisk says my drive has a wrong partition table, even if it appears on the file manager when plugged.
I think my Trisquel install is a mess...

Just so you know, I noticed that after the drive being plugged for a while, the option for safely removing the drive disappears :(

Gotta go through gnome-disks to stop the disks from spinning for a while after unmounting. I suppose this should be a bug report to whoever keeps taking care of this software.
Should this be a distro-based bug report, or should I find out who deals with Nautilus?