[ceph-users] How to troubleshoot rsync to cephfs via nfs-ganesha stalling

[Daniel Gryniewicz]

This isn't something I've seen before. rsync generally works fine, even
over cephfs. More inline.

This rsync command fails and makes the local nfs unavailable (Have to
stop nfs-ganesha, kill all rsync processes on the client and then start
nfs-ganesha)
rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*"
--exclude "isos"
/localpath/CentOS7-x86_64/
When I do individual rsyncs on the subfolders
-rw-r--r-- 1 nobody 500 14 Nov 25 17:01 CentOS_BuildTag
-rw-r--r-- 1 nobody 500 29 Nov 25 17:16 .discinfo
drwxr-xr-x 3 nobody 500 8.3M Nov 25 17:20 EFI
-rw-rw-r-- 1 nobody 500 227 Aug 30 2017 EULA
-rw-rw-r-- 1 nobody 500 18K Dec 9 2015 GPL
drwxr-xr-x 3 nobody 500 572M Nov 25 17:21 images
drwxr-xr-x 2 nobody 500 57M Dec 9 14:11 isolinux
drwxr-xr-x 2 nobody 500 433M Nov 25 17:20 LiveOS
drwxrwxr-x 2 nobody 500 9.5G Nov 25 16:58 Packages
drwxrwxr-x 2 nobody 500 29M Dec 9 13:53 repodata
-rw-rw-r-- 1 nobody 500 1.7K Dec 9 2015 RPM-GPG-KEY-CentOS-7
-rw-rw-r-- 1 nobody 500 1.7K Dec 9 2015 RPM-GPG-KEY-CentOS-Testing-7
-rw-r--r-- 1 nobody 500 354 Nov 25 17:21 .treeinfo
These rsyncs are all going fine.
rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*"
--exclude "isos"
/localpath/CentOS7-x86_64/Packages/
rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*"
--exclude "isos"
/localpath/CentOS7-x86_64/repodata/
rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*"
--exclude "isos"
/localpath/CentOS7-x86_64/LiveOS/
Except when I try to rsync the file CentOS_BuildTag then everything
stalls. Leaving such files
-rw------- 1 500 500 0 Dec 9 14:26 .CentOS_BuildTag.2igwc5
-rw------- 1 500 500 0 Dec 9 14:28 .CentOS_BuildTag.tkiwc5

So something is failing on the write, it seems. These are the temporary
files made by rsync, and they're empty, so the initial write seems to
have failed.

I can resolf this by doing a wget and moving the file to the location
wget
'http://mirror.ams1.nl.leaseweb.net/centos/7/os/x86_64/CentOS_BuildTag'
mv CentOS_BuildTag /localpath/CentOS7-x86_64/
I had also problems with .discinfo and when I ls this directory on
cephfs mount it takes a long time to produce output.
When I do the full rsync to the cephfs mount it completes without
errors, when I then later do the sync on the nfs mount it completes also
(nothing being copied)

This confirms that it's not metadata related, as this second successful
rsync is purely metadata.

Anybody know what I should do to resolv this? Is this a typical ganesha
issue or is this cephfs corruption, that make ganesha stall?

Writes in Ganesha are pretty much passthrough, modulo some metadata
tracking. This means that a write hang is likely to be somewhere
between Ganesha and CephFS. However, this is a single, small file, so I
don't see how it could hang, especially when wget can copy the file
correctly. Maybe there's something about the structure of the file?
Does it have holes in it, for example?

Also, can you send the mount options for the NFS mount?

Daniel