I had the challenge to move 600 GB of an NFS mounted nextcloud primary storage to a Hetzner storagebox. NFS usually works nice on internal networks, it is not encrypted, but the server has got too much load with thousands of users and writing to disc tuned out as a performance bottleneck with large i/o waits.

As we do not know yet how much storage we need in the next years, we have ordered a Hetzner Storagebox. Starting from a few Euros, you get Samba, Webdav or sftp connectivity and you can set up sub-accounts. A first performance test showed that the storage box mounted with webdav resulted in a writing speed of 106 MBit/s - compared to 1-2 MBit/s we have seen with NFS mounted to a local disc at digital rush hours.

Here are my entries in /etc/fstab

# integrating hetzner storagebox webdav -  slow performance
https://u111111-sub1.your-storagebox.de /mnt/storagebox_davs davfs _netdev,user,uid=www-data,gid=www-data,auto,rw,file_mode=0660,dir_mode=0770 0 0
# integrating hetzner storagebox sshfs - good performance
u111111-sub1 [at] u111111-sub1.your-storagebox.de: /mnt/storagebox_sshfs fuse.sshfs defaults,allow_other,uid=33,gid=33,port=23,reconnect,ServerAliveInterval=30,ServerAliveCountMax=10,auto_cache,cache_timeout=115200,attr_timeout=115200,_netdev 0 0

For webdav you need to add a line in /etc/dvfs2/secrets:

https://u111111-sub1.your-storagebox.de u111111-sub1 myStorageSubAccountPassword

I started to copy the content of the local NFS mounted directory and could follow the progress in the log file. The first run took about two days:

rsync -Aaxz -e 'ssh -p23' --log-file=/root/rsyncdatamigration.log --times --size-only --delete --exclude='.DAV' --exclude='authorized_keys' /data/nfs-mounted/nextcloud/ u111111-sub1 [at] 111111-sub1.your-storagebox.de: &

Now the switch should happen early in the morning. I put nextcloud in maintenance mode, and run again the rsync for syncing the rest which had changed in the meantime. Now both storages had the same content.

Trial 1: webdav failed

In the nextcloud conifg I changed the data source location:

  'datadirectory' => '/mnt/storagebox_davs',

Putting maintenance back to false showed errors first. Permissions of some folders were not www-data but root, remounted, were then okay, without having uid+gid touched, scary. Launching again let immediately start several sync clients. the webdav process went up to 100% and files appeared in the web interface very slowly, it took several minutes. No, webdav is not made for heavy load scenarios, the author admitted in a support forum. Well, I did not give a lot of cache space, but that would not help either when nextcloud steps through more than 100000 files regulary.

Trial 1: sshfs performs nice

Which other option did I have? Samba was suspicious to me, as it was rather made for the Windows world than for Unixes. So I wondered if there was a way to simply mount a scp / ssh account. And yes, there is sshfs as a fuse file system extension.

apt install sshfs

To get access via ssh keys you put a file authorized_keys with the public key of the server into the storage box' sub account directory, see the Hetzner WIKi for details. Mounting the directory should all files again and so I dared to change again the storage location in the nextcloud config during a maintenance window:

'datadirectory' => '/mnt/storagebox_sshfs',

Again, several sync clients started and CPU load remained normal, sometimes showing ssh and sshfs with a 3-10% load. Well, all traffic is being now de- and encrypted, which costs some resources.

After half a day of smooth operation, we could observe a nice performance gain in nextcloud, the violet munin clouds of i/o wait shrinked dramatically and othe NFS services on the same machine got more time to breath.

sshfs seems to be perfect to mount any file based storage. You just need ssh access to a server, no need to install or maintain any daemon software. It is secure, reliable and integrates well into the linux ecosystem and its kernel. Performance compares well shows this report. 

If I would set up another  large nextcloud server from scratch, I would consider to use a block based storage (such as minio). This gives more flexibility and scalability and is natively supported as a primary storage in nextcloud. Migrating to it would be a longer story which is being told by somebody else here.