Re: [LUG] Optimisation for backups?

[comrade meowski <mr.meowski@xxxxxxxx>]

On 02/10/2020 23:41, Simon Waters wrote:
> Been too long out of hands on system administration.
*appears in puff of magic smoke*

> # cat /sys/block/sda/queue/scheduler
> [mq-deadline] none
You're not that out of practice as you've immediately gone to the right 
place :]
The BFQ IO scheduler is ideally what you want. Tailored for desktop user 
responsiveness over any other consideration. Unfortunately not available 
in super-conservative 4 series Debian kernels but you might have CFQ 
available?
Your SSD scheduler can be left as is - you didn't actually specify the 
nature of your backup drive though. Internal SATA HDD(s) I presume? The 
problem is massively intensified through USB obviously as everyone who's 
ever had their powerful Linux box unaccountably judder and lag through a 
simple USB disk copy knows all too well.
You need a kernel with proper IO scheduler support though and you 
probably can't have that without upgrading to the 5.x series from 
Debian, building your own or installing an alternative. You want either 
CFQ (technically depreacted) or ideally, BFQ. Set your backup drive(s) 
IO scheduler to the desired choice like so:
echo bfq | sudo tee -a /sys/block/sdX/queue/scheduler

I set my workstation like this every boot, for reasons*:

echo none | sudo tee -a /sys/block/sd*/queue/scheduler

You're doing yourself a bit of a mischief here by being so conservative 
with your kernels. Don't use old kernels unless you specifically have to 
for valid reasons. Newer kernels exist for a reason and are no less 
stable than old ones, use them!
For Debian I hugely recommend Zen or Liquorix 3rd party kernels. 
Investigate them and make your own mind up but these aren't silly 
"riced" kernels made up by kids for fun - they are expertly built 
advanced kernels. If you don't want to make your own but want modern 
hardware support and features these are the go to choice.
The IO scheduler is the obvious first thing to change - tuning it would 
be unwise before correcting or tuning other aspects of your system first 
though. Second would be to correct the CPU scheduler. I boot my kernel 
like so:
comrade@failbot:~$ cat /proc/cmdline
BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.8.0-pf6-coldwar+ 
root=ZFS=rpool/ROOT/ubuntu ro bmq.timeslice=2000
BMQ is to CPU scheduling what BFQ is to IO scheduling. The distro 
defaults for all Linux variants I know are poorly chosen "best average 
fit" for both of these important components.
The *reasons* I mentioned above for setting ALL of my workstation's disk 
schedulers to "none" - normally a poor choice - is because I use ZFS and 
ZFS has it's own much more sophisticated internal IO scheduler so I set 
"none" to allow it to take over.
The rules is:non-ZFS systems or disks not handled by ZFS have their IO 
scheduler set to "none" if SSD/NVME and "bfq" if HDD. This is for 
"responsive" systems, aka PCs. Servers can be different if needed, but 
usually can be treated with the same rules and will perform better than 
the defaults.
Speaking of ZFS, it's trivially officially installable on Debian and has 
been for ages. It solves all of your issues and countless more you 
didn't even know you had. Your tar backup routine is horrible! ZFS 
optionally transparently compresses and is a much better choice 
specifically for bulk backup/data disks.
So I would:

Install a proper kernel (at least for testing options if nothing else)
Switch backup disk(s) to BFQ scheduler
Evaluate switching CPU scheduler to BMQ

Next:
Buy 8Gb more RAM for that system and another HDD
Install ZFS and use it manage that the new HDD
Compare ZFS vs your old system
Promptly switch all bulk storage to ZFS as inevitable result

System tuning is very definitely my cup of tea and there's a lot of 
other sysctl stuff to start on (vm.swappiness= etc etc) if you want to 
really get into it but pick the low hanging fruit first. Upgraded kernel 
provides many advantages including more sophisticated schedulers and 
tuning bits so you're going to need that as bare minimum.
Actually, looking again and more carefully at your system specs, exactly 
what you're doing and the fact you have a 4Gb swap partition active on 
that 8Gb of RAM then yeah, it's not surprising your poor computer isn't 
very happy with you. I would definitely expect it to be pretty badly 
impacted if you've just got default everything's set. Ouch!
tar cfz /usbmountpoint/SOMEDATEDFILE.tar.gz /home

is a rude thing to do to that poor machine if it's 200Gb and all disks 
are encrypted as well. I'm not going to tell a man how to do his own 
backups but I humbly suggest you could do a lot better, as you suggested 
yourself. Please at least consider incremental rsync you monster :]
Apologies, that was as "brief" as I could possibly keep it believe it or 
not.
--
The Mailing List for the Devon & Cornwall LUG
https://mailman.dcglug.org.uk/listinfo/list
FAQ: http://www.dcglug.org.uk/listfaq