[ceph-users] Fastest way to find raw device from OSD-ID? (osd -> lvm lv -> lvm pv -> disk)

Hi,

$ ceph-volume lvm list

Does that work for you?

Wido

Post by Kevin Olbrich
Hi!
Is there an easy way to find raw disks (eg. sdd/sdd1) by OSD id?
Before I migrated from filestore with simple-mode to bluestore with lvm,
I was able to find the raw disk with "df".
Now, I need to go from LVM LV to PV to disk every time I need to
check/smartctl a disk.
Kevin
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Kevin Olbrich

2018-10-08 10:09:09 UTC

Hi!

Yes, thank you. At least on one node this works, the other node just
freezes but this might by caused by a bad disk that I try to find.

Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

Alfredo Deza

2018-10-08 17:31:51 UTC

Hi!
Yes, thank you. At least on one node this works, the other node just freezes but this might by caused by a bad disk that I try to find.

If it is freezing, you could maybe try running the command where it
freezes? (ceph-volume will log it to the terminal)

Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Jakub Jaszewski

2018-10-08 19:15:22 UTC

Hi Kevin,
Have you tried ceph osd metadata OSDid ?

Jakub

Post by Kevin Olbrich
Hi!
Yes, thank you. At least on one node this works, the other node just

freezes but this might by caused by a bad disk that I try to find.
If it is freezing, you could maybe try running the command where it
freezes? (ceph-volume will log it to the terminal)

Post by Kevin Olbrich
Kevin
Am Mo., 8. Okt. 2018 um 12:07 Uhr schrieb Wido den Hollander <

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

Post by Kevin Olbrich
Hi!
Is there an easy way to find raw disks (eg. sdd/sdd1) by OSD id?
Before I migrated from filestore with simple-mode to bluestore with

lvm,

Post by Wido den Hollander

Post by Kevin Olbrich
I was able to find the raw disk with "df".
Now, I need to go from LVM LV to PV to disk every time I need to
check/smartctl a disk.
Kevin
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Kevin Olbrich

2018-10-08 21:26:58 UTC

Hi Jakub,

"ceph osd metadata X" this is perfect! This also lists multipath devices
which I was looking for!

Kevin

Am Mo., 8. Okt. 2018 um 21:16 Uhr schrieb Jakub Jaszewski <

Post by Jakub Jaszewski
Hi Kevin,
Have you tried ceph osd metadata OSDid ?
Jakub

Post by Kevin Olbrich
Hi!
Yes, thank you. At least on one node this works, the other node just

freezes but this might by caused by a bad disk that I try to find.
If it is freezing, you could maybe try running the command where it
freezes? (ceph-volume will log it to the terminal)

Post by Kevin Olbrich
Kevin
Am Mo., 8. Okt. 2018 um 12:07 Uhr schrieb Wido den Hollander <

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

Post by Kevin Olbrich
Hi!
Is there an easy way to find raw disks (eg. sdd/sdd1) by OSD id?
Before I migrated from filestore with simple-mode to bluestore with

lvm,

Post by Wido den Hollander

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Paul Emmerich

2018-10-08 21:03:46 UTC

ceph-volume unfortunately doesn't handle completely hanging IOs too
well compared to ceph-disk. It needs to read actual data from each
disk and it'll just hang completely if any of the disks doesn't
respond.

The low-level command to get the information from LVM is:

lvs -o lv_tags

this allows you to map a LV to an OSD id.

Paul

Hi!
Yes, thank you. At least on one node this works, the other node just freezes but this might by caused by a bad disk that I try to find.
Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

Alfredo Deza

2018-10-08 21:33:54 UTC

Post by Paul Emmerich
ceph-volume unfortunately doesn't handle completely hanging IOs too
well compared to ceph-disk.

Not sure I follow, would you mind expanding on what you mean by
"ceph-volume unfortunately doesn't handle completely hanging IOs" ?

ceph-volume just provisions the OSD, nothing else. If LVM is hanging,
there is nothing we could do there, just like ceph-disk wouldn't be
able to do anything if the partitioning
tool would hang.

Post by Paul Emmerich
It needs to read actual data from each
disk and it'll just hang completely if any of the disks doesn't
respond.
lvs -o lv_tags
this allows you to map a LV to an OSD id.
Paul

Hi!
Yes, thank you. At least on one node this works, the other node just freezes but this might by caused by a bad disk that I try to find.
Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Paul Emmerich

2018-10-08 21:51:16 UTC

Yeah, it's usually hanging in some low-level LVM tool (lvs, usually).
They unfortunately like to get stuck indefinitely on some hardware
failures, but there isn't really anything that can be done.
But we've found that it's far more reliable to just call lvs ourselves
instead of relying on ceph-volume lvm list when trying to detect OSDs
in a server, not sure what else it does that sometimes just hangs.

One thing I've learned from working with *a lot* of different hardware
from basically all vendors: every command can hang when you got a disk
that died in some bad way. Sometimes there are work-arounds by
invoking the necessary tools once for every disk instead of once for
all disks.

Paul

Post by Paul Emmerich
ceph-volume unfortunately doesn't handle completely hanging IOs too
well compared to ceph-disk.

Not sure I follow, would you mind expanding on what you mean by
"ceph-volume unfortunately doesn't handle completely hanging IOs" ?
ceph-volume just provisions the OSD, nothing else. If LVM is hanging,
there is nothing we could do there, just like ceph-disk wouldn't be
able to do anything if the partitioning
tool would hang.

Post by Paul Emmerich
It needs to read actual data from each
disk and it'll just hang completely if any of the disks doesn't
respond.
lvs -o lv_tags
this allows you to map a LV to an OSD id.
Paul

Hi!
Yes, thank you. At least on one node this works, the other node just freezes but this might by caused by a bad disk that I try to find.
Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

Paul Emmerich

2018-11-30 20:10:36 UTC

Post by Paul Emmerich
ceph-volume unfortunately doesn't handle completely hanging IOs too
well compared to ceph-disk.

Not sure I follow, would you mind expanding on what you mean by
"ceph-volume unfortunately doesn't handle completely hanging IOs" ?
ceph-volume just provisions the OSD, nothing else. If LVM is hanging,
there is nothing we could do there, just like ceph-disk wouldn't be
able to do anything if the partitioning
tool would hang.

Another follow-up for this since I ran into issues with ceph-volume
again a few times in the last weeks:
I've opened issues for the main problems that we are seeing since
using ceph-volume

http://tracker.ceph.com/issues/37490
http://tracker.ceph.com/issues/37487
http://tracker.ceph.com/issues/37492

The summary is that most operations need to access *all* disks and
that will cause problems if one of them is misbehaving.
ceph-disk didn't have this problem (but a lot of other problems,
overall we are more happy with ceph-volume)

Paul

Post by Paul Emmerich
It needs to read actual data from each
disk and it'll just hang completely if any of the disks doesn't
respond.
lvs -o lv_tags
this allows you to map a LV to an OSD id.
Paul

Hi!
Yes, thank you. At least on one node this works, the other node just freezes but this might by caused by a bad disk that I try to find.
Kevin

Post by Wido den Hollander
Hi,
$ ceph-volume lvm list
Does that work for you?
Wido

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Alfredo Deza

2018-11-30 20:20:54 UTC

Post by Paul Emmerich

Post by Paul Emmerich
ceph-volume unfortunately doesn't handle completely hanging IOs too
well compared to ceph-disk.

Not sure I follow, would you mind expanding on what you mean by
"ceph-volume unfortunately doesn't handle completely hanging IOs" ?
ceph-volume just provisions the OSD, nothing else. If LVM is hanging,
there is nothing we could do there, just like ceph-disk wouldn't be
able to do anything if the partitioning
tool would hang.

Another follow-up for this since I ran into issues with ceph-volume
I've opened issues for the main problems that we are seeing since
using ceph-volume
http://tracker.ceph.com/issues/37490
http://tracker.ceph.com/issues/37487
http://tracker.ceph.com/issues/37492
The summary is that most operations need to access *all* disks and
that will cause problems if one of them is misbehaving.
ceph-disk didn't have this problem (but a lot of other problems,
overall we are more happy with ceph-volume)

Paul, thank you so much for opening these issues. It is sometimes hard
to prevent these sort of "real world" usage problems.

None of them seem hard to tackle, I anticipate they will get done and
merged rather quickly.

Post by Paul Emmerich
Paul