[ceph-users] Fwd: "rbd map" command hangs

[Jay Janardhan]

(Sorry if this is a duplicate message - email server is acting up this
morning).


I'm following quick start guide and have a ceph cluster with three nodes.
When I try to map image to block device my command hangs. This seems like a
kernel hang as the only way I was able to get out is via a hard reset of
the image. The following is my configuration. Any help is greatly
appreciated.

command on the ceph-client node (that hangs):

$ sudo rbd map foo1 --pool rbd --name client.admin

*ceph-client node info:*

$ rbd info foo1

rbd image 'foo1':

size 4096 MB in 1024 objects

order 22 (4096 kB objects)

block_name_prefix: rb.0.1050.74b0dc51

format: 1

Kernel and Ubuntu release:

$ uname -r

3.6.9-030609-generic

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 12.04 LTS

Release: 12.04

Codename: precise


logs from /var/log/syslog:

May 17 14:12:48 ceph-client kernel: [ 128.866445] Key type ceph registered

May 17 14:12:48 ceph-client kernel: [ 128.866453] libceph: loaded (mon/osd
proto 15/24, osdmap 5/6 5/6)

May 17 14:12:48 ceph-client kernel: [ 128.867313] rbd: loaded rbd (rados
block device)



*ceph-node1 node info (node2 and node3 are similar):*

$ uname -r

3.2.0-23-generic

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 12.04 LTS

Release: 12.04

Codename: precise



$ ceph status

cluster df4f503a-04a9-4572-96d3-e31218592cfa

health HEALTH_OK

monmap e1: 1 mons at {ceph-node1=192.168.56.102:6789/0}, election
epoch 2, quorum 0 ceph-node1

osdmap e60: 3 osds: 3 up, 3 in

pgmap v1168: 192 pgs, 3 pools, 1373 bytes data, 4 objects

22739 MB used, 202 GB / 236 GB avail

192 active+clean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140519/0f29e0f9/attachment.htm>

[Ilya Dryomov]

Post by Jay Janardhan
(Sorry if this is a duplicate message - email server is acting up this
morning).
I'm following quick start guide and have a ceph cluster with three nodes.
When I try to map image to block device my command hangs. This seems like a
kernel hang as the only way I was able to get out is via a hard reset of the
image. The following is my configuration. Any help is greatly appreciated.
$ sudo rbd map foo1 --pool rbd --name client.admin

What's your ceph version (ceph --version)? Can you run 'rbd map' under strace,
and when it hangs do SysRq+w followed by SysRq+t and send along strace and
SysRq outputs?

Thanks,

Ilya

[Jay Janardhan]

Thanks for the response Ilya. I need to figure out how to use SysRq on my
Mac. Meanwhile, here is the strace output and ceph version:

*Ceph Version: *ceph version 0.80.1
(a38fe1169b6d2ac98b427334c12d7cf81f809b74)

*?*
Note that IP 192.168.56.102:6789 is reachable from the client node. ?

*?192.168.56.102 is a monitor node.*

?$ ?
ceph status

cluster df4f503a-04a9-4572-96d3-e31218592cfa

health HEALTH_OK

monmap e1: 1 mons at {ceph-node1=192.168.56.102:6789/0}, election
epoch 2, quorum 0 ceph-node1

osdmap e102: 3 osds: 3 up, 3 in

pgmap v1604: 192 pgs, 3 pools, 1373 bytes data, 4 objects

22744 MB used, 202 GB / 236 GB avail

192 active+clean

*strace output:*

map(NULL, 8392704, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7899894000

mprotect(0x7f7899894000, 4096, PROT_NONE) = 0

clone(child_stack=0x7f789a093f70,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7f789a0949d0, tls=0x7f789a094700,
child_tidptr=0x7f789a0949d0) = 1629

rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0

open("/etc/ceph/ceph.client.admin.keyring", O_RDONLY) = 3

close(3) = 0

open("/etc/ceph/ceph.client.admin.keyring", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=63, ...}) = 0

read(3, "[client.admin]\n\tkey = AQDb7HRTkB"..., 63) = 63

close(3) = 0

futex(0x7f789cc741a4, FUTEX_WAKE_PRIVATE, 2147483647) = 0

brk(0x17e3000) = 0x17e3000

futex(0x17a4b84, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x17a4b80, {FUTEX_OP_SET, 0,
FUTEX_OP_CMP_GT, 1}) = 1

futex(0x17a4b00, FUTEX_WAKE_PRIVATE, 1) = 1

brk(0x17e1000) = 0x17e1000

add_key(0x425208, 0x7fffce5220b0, 0x7fffce521fe0, 0x22, 0xfffffffe) = -1
ENODEV (No such device)

stat("/sys/bus/rbd", 0x7fffce522230) = -1 ENOENT (No such file or
directory)

rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], 0}, 8) = 0

rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], 0}, 8) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0

clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD,
parent_tidptr=0x7fffce522060) = 1630

wait4(1630, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 1630

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8) =
0

rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0

rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], SA_RESTORER, 0x7f789c6d84c0}, 8) = 0

rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], SA_RESTORER, 0x7f789c6d84c0}, 8) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0

clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD,
parent_tidptr=0x7fffce522060) = 1633

wait4(1633, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1633

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8) =
0

rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0

rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0

open("/sys/bus/rbd/add_single_major", O_WRONLY) = -1 ENOENT (No such file
or directory)

open("/sys/bus/rbd/add", O_WRONLY) = 3

write(3, "192.168.56.102:6789 name=admin,s"..., 87


?

Post by Jay Janardhan

Post by Jay Janardhan
(Sorry if this is a duplicate message - email server is acting up this
morning).
I'm following quick start guide and have a ceph cluster with three nodes.
When I try to map image to block device my command hangs. This seems

like a

Post by Jay Janardhan
kernel hang as the only way I was able to get out is via a hard reset of

the

Post by Jay Janardhan
image. The following is my configuration. Any help is greatly

appreciated.

Post by Jay Janardhan
$ sudo rbd map foo1 --pool rbd --name client.admin

What's your ceph version (ceph --version)? Can you run 'rbd map' under strace,
and when it hangs do SysRq+w followed by SysRq+t and send along strace and
SysRq outputs?
Thanks,
Ilya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140519/fbc66079/attachment.htm>

[Jay Janardhan]

Ilya, The SysRq is not doing anything as the kernel is hung. Btw, this is a
VirtualBox environment so I used the VBoxManage to send the SysRq commands.
Just to let you know, the system locksup and the only way out is a hard
reset.

Post by Jay Janardhan
Thanks for the response Ilya. I need to figure out how to use SysRq on my
*Ceph Version: *ceph version 0.80.1
(a38fe1169b6d2ac98b427334c12d7cf81f809b74)
*?*
Note that IP 192.168.56.102:6789 is reachable from the client node. ?
*?192.168.56.102 is a monitor node.*
?$ ?
ceph status
cluster df4f503a-04a9-4572-96d3-e31218592cfa
health HEALTH_OK
monmap e1: 1 mons at {ceph-node1=192.168.56.102:6789/0}, election
epoch 2, quorum 0 ceph-node1
osdmap e102: 3 osds: 3 up, 3 in
pgmap v1604: 192 pgs, 3 pools, 1373 bytes data, 4 objects
22744 MB used, 202 GB / 236 GB avail
192 active+clean
*strace output:*
map(NULL, 8392704, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7899894000
mprotect(0x7f7899894000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f789a093f70,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7f789a0949d0, tls=0x7f789a094700,
child_tidptr=0x7f789a0949d0) = 1629
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
open("/etc/ceph/ceph.client.admin.keyring", O_RDONLY) = 3
close(3) = 0
open("/etc/ceph/ceph.client.admin.keyring", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=63, ...}) = 0
read(3, "[client.admin]\n\tkey = AQDb7HRTkB"..., 63) = 63
close(3) = 0
futex(0x7f789cc741a4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(0x17e3000) = 0x17e3000
futex(0x17a4b84, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x17a4b80, {FUTEX_OP_SET, 0,
FUTEX_OP_CMP_GT, 1}) = 1
futex(0x17a4b00, FUTEX_WAKE_PRIVATE, 1) = 1
brk(0x17e1000) = 0x17e1000
add_key(0x425208, 0x7fffce5220b0, 0x7fffce521fe0, 0x22, 0xfffffffe) = -1
ENODEV (No such device)
stat("/sys/bus/rbd", 0x7fffce522230) = -1 ENOENT (No such file or
directory)
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0},
{SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD,
parent_tidptr=0x7fffce522060) = 1630
wait4(1630, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 1630
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0}, {SIG_DFL,
[], SA_RESTORER, 0x7f789c6d84c0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f789c6d84c0},
{SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD,
parent_tidptr=0x7fffce522060) = 1633
wait4(1633, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1633
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f789c6d84c0}, NULL, 8)
= 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
open("/sys/bus/rbd/add_single_major", O_WRONLY) = -1 ENOENT (No such file
or directory)
open("/sys/bus/rbd/add", O_WRONLY) = 3
write(3, "192.168.56.102:6789 name=admin,s"..., 87
?

Post by Jay Janardhan

Post by Jay Janardhan
(Sorry if this is a duplicate message - email server is acting up this
morning).
I'm following quick start guide and have a ceph cluster with three

nodes.

Post by Jay Janardhan
When I try to map image to block device my command hangs. This seems

like a

Post by Jay Janardhan
kernel hang as the only way I was able to get out is via a hard reset

of the

Post by Jay Janardhan
image. The following is my configuration. Any help is greatly

appreciated.

Post by Jay Janardhan
$ sudo rbd map foo1 --pool rbd --name client.admin

What's your ceph version (ceph --version)? Can you run 'rbd map' under strace,
and when it hangs do SysRq+w followed by SysRq+t and send along strace and
SysRq outputs?
Thanks,
Ilya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140519/ddd67475/attachment.htm>

[Ilya Dryomov]

Post by Jay Janardhan
Ilya, The SysRq is not doing anything as the kernel is hung. Btw, this is a
VirtualBox environment so I used the VBoxManage to send the SysRq commands.
Just to let you know, the system locksup and the only way out is a hard
reset.

Well, that's not much to go on. Was there something in dmesg when it
locked up or in response to SysRqs?

Thanks,

Ilya

[Jay Janardhan]

Got the stack trace when it crashed. I had to enable serial port to capture
this. Would this help?

[ 172.227318] libceph: mon0 192.168.56.102:6789 feature set mismatch, my
40002 < server's 20042040002, missing 20042000000

[ 172.451109] libceph: mon0 192.168.56.102:6789 socket error on read

[ 172.539837] ------------[ cut here ]------------

[ 172.640704] kernel BUG at /home/apw/COD/linux/net/ceph/messenger.c:2366!

[ 172.740775] invalid opcode: 0000 [#1] SMP

[ 172.805429] Modules linked in: rbd libceph libcrc32c nfsd nfs_acl
auth_rpcgss nfs fscache lockd sunrpc ext2 ppdev microcode psmouse serio_raw
parport_pc i2c_piix4 mac_hid lp parport e1000

[ 173.072985] CPU 0

[ 173.143909] Pid: 385, comm: kworker/0:3 Not tainted 3.6.9-030609-generic
#201212031610 innotek GmbH VirtualBox/VirtualBox

[ 173.358836] RIP: 0010:[<ffffffffa0183ff7>] [<ffffffffa0183ff7>]
ceph_fault+0x267/0x270 [libceph]

[ 173.629918] RSP: 0018:ffff88007b497d90 EFLAGS: 00010286

[ 173.731786] RAX: fffffffffffffffe RBX: ffff88007b909298 RCX:
0000000000000003

[ 173.901361] RDX: 0000000000000000 RSI: 00000000ffffffff RDI:
0000000000000039

[ 174.040360] RBP: ffff88007b497dc0 R08: 000000000000000a R09:
000000000000fffb

[ 174.235587] R10: 0000000000000000 R11: 0000000000000199 R12:
ffff88007b9092c8

[ 174.385067] R13: 0000000000000000 R14: ffffffffa0199580 R15:
ffffffffa0195773

[ 174.541288] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000)
knlGS:0000000000000000

[ 174.620856] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

[ 174.740551] CR2: 00007fefd16c5168 CR3: 000000007bb41000 CR4:
00000000000006f0

[ 174.948095] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000

[ 175.076881] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400

[ 175.320731] Process kworker/0:3 (pid: 385, threadinfo ffff88007b496000,
task ffff880079735bc0)

[ 175.565218] Stack:

[ 175.630655] 0000000000000000 ffff88007b909298 ffff88007b909690
ffff88007b9093d0

[ 175.699571] ffff88007b909418 ffff88007fc0e300 ffff88007b497df0
ffffffffa018525c

[ 175.710012] ffff88007b909690 ffff880078e4d800 ffff88007fc1bf00
ffff88007fc0e340

[ 175.859748] Call Trace:

[ 175.909572] [<ffffffffa018525c>] con_work+0x14c/0x1c0 [libceph]

[ 176.010436] [<ffffffff810763b6>] process_one_work+0x136/0x550

[ 176.131098] [<ffffffffa0185110>] ? try_read+0x440/0x440 [libceph]

[ 176.249904] [<ffffffff810775b5>] worker_thread+0x165/0x3c0

[ 176.368412] [<ffffffff81077450>] ? manage_workers+0x190/0x190

[ 176.512415] [<ffffffff8107c5e3>] kthread+0x93/0xa0

[ 176.623469] [<ffffffff816b8c04>] kernel_thread_helper+0x4/0x10

[ 176.670502] [<ffffffff8107c550>] ? flush_kthread_worker+0xb0/0xb0

[ 176.731089] [<ffffffff816b8c00>] ? gs_change+0x13/0x13

[ 176.901284] Code: 00 00 00 00 48 8b 83 38 01 00 00 a8 02 0f 85 f6 fe ff
ff 3e 80 a3 38 01 00 00 fb 48 c7 83 40 01 00 00 06 00 00 00 e9 37 ff ff ff
<0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8

[ 177.088895] RIP [<ffffffffa0183ff7>] ceph_fault+0x267/0x270 [libceph]

[ 177.251573] RSP <ffff88007b497d90>

[ 177.310320] ---[ end trace f66ddfdda09b9821 ]---

[ 177.461430] BUG: unable to handle kernel paging request at
fffffffffffffff8

[ 177.464615] IP: [<ffffffff8107c8b1>] kthread_data+0x11/0x20

[ 177.464615] PGD 1c0e067 PUD 1c0f067 PMD 0

[ 177.464615] Oops: 0000 [#2] SMP

[ 177.464615] Modules linked in: rbd libceph libcrc32c nfsd nfs_acl
auth_rpcgss nfs fscache lockd sunrpc ext2 ppdev microcode psmouse serio_raw
parport_pc i2c_piix4 mac_hid lp parport e1000

[ 177.464615] CPU 0

[ 177.464615] Pid: 385, comm: kworker/0:3 Tainted: G D
3.6.9-030609-generic #201212031610 innotek GmbH VirtualBox/VirtualBox

[ 177.464615] RIP: 0010:[<ffffffff8107c8b1>] [<ffffffff8107c8b1>]
kthread_data+0x11/0x20

[ 177.464615] RSP: 0018:ffff88007b497a70 EFLAGS: 00010096

[ 177.464615] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000

[ 177.464615] RDX: ffffffff81e593c0 RSI: 0000000000000000 RDI:
ffff880079735bc0

[ 177.464615] RBP: ffff88007b497a88 R08: 0000000000989680 R09:
0000000000000400

[ 177.464615] R10: 0000000000000000 R11: ffff880078fb09e0 R12:
0000000000000000

[ 177.464615] R13: ffff880079735f90 R14: 0000000000000001 R15:
0000000000000006

[ 177.464615] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000)
knlGS:0000000000000000

[ 177.464615] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

[ 177.464615] CR2: fffffffffffffff8 CR3: 000000007b73e000 CR4:
00000000000006f0

[ 177.464615] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000

[ 177.464615] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400

[ 177.464615] Process kworker/0:3 (pid: 385, threadinfo ffff88007b496000,
task ffff880079735bc0)

[ 177.464615] Stack:

[ 177.464615] ffffffff81077dc5 ffff88007b497a88 ffff88007fc13dc0
ffff88007b497b08

[ 177.464615] ffffffff816ade3f ffff88007b497ab8 0000000000000000
ffff88007b497fd8

[ 177.464615] ffff88007b497fd8 ffff88007b497fd8 0000000000013dc0
ffff880078d8d618

[ 177.464615] Call Trace:

[ 177.464615] [<ffffffff81077dc5>] ? wq_worker_sleeping+0x15/0xc0

[ 177.464615] [<ffffffff816ade3f>] __schedule+0x5cf/0x6f0

[ 177.464615] [<ffffffff816ae279>] schedule+0x29/0x70

[ 177.464615] [<ffffffff8105d793>] do_exit+0x2b3/0x470

[ 177.464615] [<ffffffff816b04a0>] oops_end+0xb0/0xf0

[ 177.464615] [<ffffffff81017c78>] die+0x58/0x90

[ 177.464615] [<ffffffff816afd94>] do_trap+0xc4/0x170

[ 177.464615] [<ffffffff81015385>] do_invalid_op+0x95/0xb0

[ 177.464615] [<ffffffffa0183ff7>] ? ceph_fault+0x267/0x270 [libceph]

[ 177.464615] [<ffffffff81340fd1>] ? vsnprintf+0x461/0x600

[ 177.464615] [<ffffffff816b8a7b>] invalid_op+0x1b/0x20

[ 177.464615] [<ffffffffa0183ff7>] ? ceph_fault+0x267/0x270 [libceph]

[ 177.464615] [<ffffffffa018525c>] con_work+0x14c/0x1c0 [libceph]

[ 177.464615] [<ffffffff810763b6>] process_one_work+0x136/0x550

[ 177.464615] [<ffffffffa0185110>] ? try_read+0x440/0x440 [libceph]

[ 177.464615] [<ffffffff810775b5>] worker_thread+0x165/0x3c0

[ 177.464615] [<ffffffff81077450>] ? manage_workers+0x190/0x190

[ 177.464615] [<ffffffff8107c5e3>] kthread+0x93/0xa0

[ 177.464615] [<ffffffff816b8c04>] kernel_thread_helper+0x4/0x10

[ 177.464615] [<ffffffff8107c550>] ? flush_kthread_worker+0xb0/0xb0

[ 177.464615] [<ffffffff816b8c00>] ? gs_change+0x13/0x13

[ 177.464615] Code: ff ff eb 88 be 57 01 00 00 48 c7 c7 38 3f a2 81 e8 75
a6 fd ff e9 b4 fe ff ff 55 48 89 e5 0f 1f 44 00 00 48 8b 87 78 03 00 00 5d
<48> 8b 40 f8 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f

[ 177.464615] RIP [<ffffffff8107c8b1>] kthread_data+0x11/0x20

[ 177.464615] RSP <ffff88007b497a70>

[ 177.464615] CR2: fffffffffffffff8

[ 177.464615] ---[ end trace f66ddfdda09b9822 ]---

[ 177.464615] Fixing recursive fault but reboot is needed!

Post by Jay Janardhan

Post by Jay Janardhan
Ilya, The SysRq is not doing anything as the kernel is hung. Btw, this

is a

Post by Jay Janardhan
VirtualBox environment so I used the VBoxManage to send the SysRq

commands.

Post by Jay Janardhan
Just to let you know, the system locksup and the only way out is a hard
reset.

Well, that's not much to go on. Was there something in dmesg when it
locked up or in response to SysRqs?
Thanks,
Ilya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140519/1e0700f0/attachment.htm>

[Ilya Dryomov]

Post by Jay Janardhan
Got the stack trace when it crashed. I had to enable serial port to capture
this. Would this help?
[ 172.227318] libceph: mon0 192.168.56.102:6789 feature set mismatch, my
40002 < server's 20042040002, missing 20042000000
[ 172.451109] libceph: mon0 192.168.56.102:6789 socket error on read
[ 172.539837] ------------[ cut here ]------------
[ 172.640704] kernel BUG at /home/apw/COD/linux/net/ceph/messenger.c:2366!
[ 172.740775] invalid opcode: 0000 [#1] SMP
[ 172.805429] Modules linked in: rbd libceph libcrc32c nfsd nfs_acl
auth_rpcgss nfs fscache lockd sunrpc ext2 ppdev microcode psmouse serio_raw
parport_pc i2c_piix4 mac_hid lp parport e1000
[ 173.072985] CPU 0
[ 173.143909] Pid: 385, comm: kworker/0:3 Not tainted 3.6.9-030609-generic
#201212031610 innotek GmbH VirtualBox/VirtualBox
[ 173.358836] RIP: 0010:[<ffffffffa0183ff7>] [<ffffffffa0183ff7>]
ceph_fault+0x267/0x270 [libceph]
[ 173.629918] RSP: 0018:ffff88007b497d90 EFLAGS: 00010286
0000000000000003
0000000000000039
000000000000fffb
ffff88007b9092c8
ffffffffa0195773
[ 174.541288] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000)
knlGS:0000000000000000
[ 174.620856] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
00000000000006f0
0000000000000000
0000000000000400
[ 175.320731] Process kworker/0:3 (pid: 385, threadinfo ffff88007b496000,
task ffff880079735bc0)
[ 175.630655] 0000000000000000 ffff88007b909298 ffff88007b909690
ffff88007b9093d0
[ 175.699571] ffff88007b909418 ffff88007fc0e300 ffff88007b497df0
ffffffffa018525c
[ 175.710012] ffff88007b909690 ffff880078e4d800 ffff88007fc1bf00
ffff88007fc0e340
[ 175.909572] [<ffffffffa018525c>] con_work+0x14c/0x1c0 [libceph]
[ 176.010436] [<ffffffff810763b6>] process_one_work+0x136/0x550
[ 176.131098] [<ffffffffa0185110>] ? try_read+0x440/0x440 [libceph]
[ 176.249904] [<ffffffff810775b5>] worker_thread+0x165/0x3c0
[ 176.368412] [<ffffffff81077450>] ? manage_workers+0x190/0x190
[ 176.512415] [<ffffffff8107c5e3>] kthread+0x93/0xa0
[ 176.623469] [<ffffffff816b8c04>] kernel_thread_helper+0x4/0x10
[ 176.670502] [<ffffffff8107c550>] ? flush_kthread_worker+0xb0/0xb0
[ 176.731089] [<ffffffff816b8c00>] ? gs_change+0x13/0x13
[ 176.901284] Code: 00 00 00 00 48 8b 83 38 01 00 00 a8 02 0f 85 f6 fe ff
ff 3e 80 a3 38 01 00 00 fb 48 c7 83 40 01 00 00 06 00 00 00 e9 37 ff ff ff
<0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8
[ 177.088895] RIP [<ffffffffa0183ff7>] ceph_fault+0x267/0x270 [libceph]
[ 177.251573] RSP <ffff88007b497d90>
[ 177.310320] ---[ end trace f66ddfdda09b9821 ]---

OK, it definitely shouldn't have crashed here and there is a patch in
later kernels that prevents this crash from happening. But, because
3.6 is too old and misses features, which is reported just prior to the
crash splat, you wouldn't be able to use it with firefly userspace even
if it didn't crash.

You are going to need to run at least 3.9 and then disable vary_r
tunable in your crushmap (vary_r will only be supported starting with
3.15) or primary-affinity adjustments - I can't tell which one is it
just from the feature set mismatch message.

Thanks,

Ilya

[Jay Janardhan]

Thanks again Ilya.

I was following this recommendations:
http://ceph.com/docs/master/start/os-recommendations/. Should this page be
updated in that case?

I'm going to upgrade to 3.9. Should I update the Ceph cluster nodes as well
or just the ceph block device client?

Post by Jay Janardhan

Post by Jay Janardhan
Got the stack trace when it crashed. I had to enable serial port to

capture

Post by Jay Janardhan
this. Would this help?
[ 172.227318] libceph: mon0 192.168.56.102:6789 feature set mismatch,

my

Post by Jay Janardhan
40002 < server's 20042040002, missing 20042000000
[ 172.451109] libceph: mon0 192.168.56.102:6789 socket error on read
[ 172.539837] ------------[ cut here ]------------
[ 172.640704] kernel BUG at

/home/apw/COD/linux/net/ceph/messenger.c:2366!

Post by Jay Janardhan
[ 172.740775] invalid opcode: 0000 [#1] SMP
[ 172.805429] Modules linked in: rbd libceph libcrc32c nfsd nfs_acl
auth_rpcgss nfs fscache lockd sunrpc ext2 ppdev microcode psmouse

serio_raw

Post by Jay Janardhan
parport_pc i2c_piix4 mac_hid lp parport e1000
[ 173.072985] CPU 0
[ 173.143909] Pid: 385, comm: kworker/0:3 Not tainted

3.6.9-030609-generic

Post by Jay Janardhan
#201212031610 innotek GmbH VirtualBox/VirtualBox
[ 173.358836] RIP: 0010:[<ffffffffa0183ff7>] [<ffffffffa0183ff7>]
ceph_fault+0x267/0x270 [libceph]
[ 173.629918] RSP: 0018:ffff88007b497d90 EFLAGS: 00010286
0000000000000003
0000000000000039
000000000000fffb
ffff88007b9092c8
ffffffffa0195773
[ 174.541288] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000)
knlGS:0000000000000000
[ 174.620856] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
00000000000006f0
0000000000000000
0000000000000400
[ 175.320731] Process kworker/0:3 (pid: 385, threadinfo

ffff88007b496000,

Post by Jay Janardhan
task ffff880079735bc0)
[ 175.630655] 0000000000000000 ffff88007b909298 ffff88007b909690
ffff88007b9093d0
[ 175.699571] ffff88007b909418 ffff88007fc0e300 ffff88007b497df0
ffffffffa018525c
[ 175.710012] ffff88007b909690 ffff880078e4d800 ffff88007fc1bf00
ffff88007fc0e340
[ 175.909572] [<ffffffffa018525c>] con_work+0x14c/0x1c0 [libceph]
[ 176.010436] [<ffffffff810763b6>] process_one_work+0x136/0x550
[ 176.131098] [<ffffffffa0185110>] ? try_read+0x440/0x440 [libceph]
[ 176.249904] [<ffffffff810775b5>] worker_thread+0x165/0x3c0
[ 176.368412] [<ffffffff81077450>] ? manage_workers+0x190/0x190
[ 176.512415] [<ffffffff8107c5e3>] kthread+0x93/0xa0
[ 176.623469] [<ffffffff816b8c04>] kernel_thread_helper+0x4/0x10
[ 176.670502] [<ffffffff8107c550>] ? flush_kthread_worker+0xb0/0xb0
[ 176.731089] [<ffffffff816b8c00>] ? gs_change+0x13/0x13
[ 176.901284] Code: 00 00 00 00 48 8b 83 38 01 00 00 a8 02 0f 85 f6 fe

ff

Post by Jay Janardhan
ff 3e 80 a3 38 01 00 00 fb 48 c7 83 40 01 00 00 06 00 00 00 e9 37 ff ff

ff

Post by Jay Janardhan
<0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8
[ 177.088895] RIP [<ffffffffa0183ff7>] ceph_fault+0x267/0x270 [libceph]
[ 177.251573] RSP <ffff88007b497d90>
[ 177.310320] ---[ end trace f66ddfdda09b9821 ]---

OK, it definitely shouldn't have crashed here and there is a patch in
later kernels that prevents this crash from happening. But, because
3.6 is too old and misses features, which is reported just prior to the
crash splat, you wouldn't be able to use it with firefly userspace even
if it didn't crash.
You are going to need to run at least 3.9 and then disable vary_r
tunable in your crushmap (vary_r will only be supported starting with
3.15) or primary-affinity adjustments - I can't tell which one is it
just from the feature set mismatch message.
Thanks,
Ilya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/51bf331a/attachment.htm>

[Ilya Dryomov]

Post by Jay Janardhan
Thanks again Ilya.
http://ceph.com/docs/master/start/os-recommendations/. Should this page be
updated in that case?
I'm going to upgrade to 3.9. Should I update the Ceph cluster nodes as well
or just the ceph block device client?

If you are going to be upgrading, you might as well upgrade to a later
kernel. No, only the nodes that you want to do 'rbd map' on.

Thanks,

Ilya

[Jay Janardhan]

Ilya, how exactly do I disable vary_r in the crushmap?

I ran "ceph osd crush tunables firefly" but that is resulting in

feature set mismatch, my 384a042a42 < server's 2384a042a42, missing
20000000000

Post by Ilya Dryomov

Post by Jay Janardhan
Thanks again Ilya.
http://ceph.com/docs/master/start/os-recommendations/. Should this page

be

Post by Jay Janardhan
updated in that case?
I'm going to upgrade to 3.9. Should I update the Ceph cluster nodes as

well

Post by Jay Janardhan
or just the ceph block device client?

If you are going to be upgrading, you might as well upgrade to a later
kernel. No, only the nodes that you want to do 'rbd map' on.
Thanks,
Ilya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/18c0a845/attachment.htm>

[Ilya Dryomov]

Post by Jay Janardhan
Ilya, how exactly do I disable vary_r in the crushmap?
I ran "ceph osd crush tunables firefly" but that is resulting in
feature set mismatch, my 384a042a42 < server's 2384a042a42, missing
20000000000

ceph osd getcrushmap -o /tmp/crush
crushtool -i /tmp/crush --set-chooseleaf_vary_r 0 -o /tmp/crush.new
ceph osd setcrushmap -i /tmp/crush.new

Thanks,

Ilya

[Jay Janardhan]

I'm following quick start guide and have a ceph cluster with three nodes.
When I try to map image to block device my command hangs. This seems like a
kernel hang as the only way I was able to get out is via a hard reset of
the image. The following is my configuration. Any help is greatly
appreciated.

command on the ceph-client node (that hangs):

$ sudo rbd map foo1 --pool rbd --name client.admin

*ceph-client node info:*

$ rbd info foo1

rbd image 'foo1':

size 4096 MB in 1024 objects

order 22 (4096 kB objects)

block_name_prefix: rb.0.1050.74b0dc51

format: 1

Kernel and Ubuntu release:

$ uname -r

3.6.9-030609-generic

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 12.04 LTS

Release: 12.04

Codename: precise


logs from /var/log/syslog:

May 17 14:12:48 ceph-client kernel: [ 128.866445] Key type ceph registered

May 17 14:12:48 ceph-client kernel: [ 128.866453] libceph: loaded (mon/osd
proto 15/24, osdmap 5/6 5/6)

May 17 14:12:48 ceph-client kernel: [ 128.867313] rbd: loaded rbd (rados
block device)



*ceph-node1 node info (node2 and node3 are similar):*

$ uname -r

3.2.0-23-generic

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 12.04 LTS

Release: 12.04

Codename: precise



$ ceph status

cluster df4f503a-04a9-4572-96d3-e31218592cfa

health HEALTH_OK

monmap e1: 1 mons at {ceph-node1=192.168.56.102:6789/0}, election
epoch 2, quorum 0 ceph-node1

osdmap e60: 3 osds: 3 up, 3 in

pgmap v1168: 192 pgs, 3 pools, 1373 bytes data, 4 objects

22739 MB used, 202 GB / 236 GB avail

192 active+clean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140517/5b8ea951/attachment.htm>