[ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

[Jan Kalcic]

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Karan Singh]

Hello Jan

I faced similar kind of errors and these are really annoying. I tried this and worked for me.

1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.


Below is my solution ( you should change it as per your hostname and IP etc)



#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and monitors UP and Running

[root at ceph-mon1 ~]# ceph status
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
health HEALTH_OK
monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB / 3512 GB avail
mdsmap e1: 0/0/1 up
[root at ceph-mon1 ~]#


All the best :-)

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Jan Kalcic]

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Karan Singh]

what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Alfredo Deza]

Post by Karan Singh
what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?
Many Thanks
Karan Singh
----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?

Do you have iptables enabled? If so, either open the ports for the
monitors to be able to talk or disable it and try again

Post by Karan Singh
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Alfredo Deza]

Post by Karan Singh
what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?
Many Thanks
Karan Singh
----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?

Do you have iptables enabled? If so, either open the ports for the
monitors to be able to talk or disable it and try again

Post by Karan Singh
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Alfredo Deza]

Post by Karan Singh
what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?
Many Thanks
Karan Singh
----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?

Do you have iptables enabled? If so, either open the ports for the
monitors to be able to talk or disable it and try again

Post by Karan Singh
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Alfredo Deza]

Post by Karan Singh
what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?
Many Thanks
Karan Singh
----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?

Do you have iptables enabled? If so, either open the ports for the
monitors to be able to talk or disable it and try again

Post by Karan Singh
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Karan Singh]

what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Karan Singh]

what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Karan Singh]

what ceph status and ceph mon_status output says , did you checked logs after this anything interesting there ?

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: "Karan Singh" <ksingh at csc.fi>
Cc: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Jan Kalcic]

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Jan Kalcic]

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Jan Kalcic]

Hi Karan,

Post by Karan Singh
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and worked for me.

Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...

Post by Karan Singh
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554

I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:


ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:

ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...

Anything else?

Thanks,
Jan

[Joao Eduardo Luis]

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same id,
and the monitor you're trying to start is unable to acquire the lock.

Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1' running.
If this error persists, set 'debug mon = 10' on ceph.conf, restart the
monitor and send us the resulting log.

-Joao

Post by Jan Kalcic
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Jan Kalcic]

Hi Joao,

Post by Joao Eduardo Luis

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same
id, and the monitor you're trying to start is unable to acquire the lock.

For example the following (3931, 3933) ??

# ps aux | grep ceph
root 3303 0.0 0.7 82184 3564 ? Ss 18:08 0:00 sshd:
ceph [priv]
ceph 3306 0.0 0.4 82312 2164 ? S 18:08 0:00 sshd:
ceph at pts/0
ceph 3307 0.0 0.5 14004 2952 pts/0 Ss 18:08 0:00 -bash
root 3835 0.0 0.3 11648 1680 ? S 18:30 0:00 /bin/sh
/usr/sbin/rcceph -c /etc/ceph/ceph.conf start mon.ceph-node1
root 3930 0.0 0.2 11512 1312 ? S 18:30 0:00 bash -c
ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3931 0.0 1.0 76612 5328 ? S 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3933 0.0 1.0 101688 5096 ? Sl 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3941 0.0 1.7 69468 8796 ? S 18:30 0:00 python
/usr/bin/ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
ceph 3964 0.0 0.1 4940 968 pts/0 R+ 18:37 0:00 ps aux
ceph 3965 0.0 0.1 5712 820 pts/0 R+ 18:37 0:00 grep ceph

but I am not doing tricky things, just following basic steps in the doc
and running ceph-deploy as indicated.. what's wrong with ceph-deploy then??

Post by Joao Eduardo Luis
Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1'
running. If this error persists, set 'debug mon = 10' on ceph.conf,
restart the monitor and send us the resulting log.

# find /var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1/sysvinit
/var/lib/ceph/mon/ceph-ceph-node1/store.db
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOG
/var/lib/ceph/mon/ceph-ceph-node1/store.db/CURRENT
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000006.log
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000005.ldb
/var/lib/ceph/mon/ceph-ceph-node1/store.db/MANIFEST-000004
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/keyring

# cat ceph.conf
[global]
filestore_xattr_use_omap = true
mon_host = 172.17.43.11
fsid = 18203301-f70b-40f1-b93e-fde4731f3bee
mon_initial_members = ceph-node1
auth_supported = cephx
osd_journal_size = 1024
debug mon = 10

...this is actually on ceph-admin but I have verified it is deployed
afterward to ceph-node1 (by running --overwrite-conf)

# ceph-deploy --overwrite-conf mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy
--overwrite-conf mon create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

# cat /var/log/ceph/ceph-mon.ceph-node1.log
2013-12-05 18:30:36.473193 7f8f0ada3780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 3931

Not a lot of info in there though :( I see it is PID 3931, so 3933 might
be the unwanted one?

Thanks,
Jan

[Jan Kalcic]

Hi Joao,

Post by Joao Eduardo Luis

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same
id, and the monitor you're trying to start is unable to acquire the lock.

For example the following (3931, 3933) ??

# ps aux | grep ceph
root 3303 0.0 0.7 82184 3564 ? Ss 18:08 0:00 sshd:
ceph [priv]
ceph 3306 0.0 0.4 82312 2164 ? S 18:08 0:00 sshd:
ceph at pts/0
ceph 3307 0.0 0.5 14004 2952 pts/0 Ss 18:08 0:00 -bash
root 3835 0.0 0.3 11648 1680 ? S 18:30 0:00 /bin/sh
/usr/sbin/rcceph -c /etc/ceph/ceph.conf start mon.ceph-node1
root 3930 0.0 0.2 11512 1312 ? S 18:30 0:00 bash -c
ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3931 0.0 1.0 76612 5328 ? S 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3933 0.0 1.0 101688 5096 ? Sl 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3941 0.0 1.7 69468 8796 ? S 18:30 0:00 python
/usr/bin/ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
ceph 3964 0.0 0.1 4940 968 pts/0 R+ 18:37 0:00 ps aux
ceph 3965 0.0 0.1 5712 820 pts/0 R+ 18:37 0:00 grep ceph

but I am not doing tricky things, just following basic steps in the doc
and running ceph-deploy as indicated.. what's wrong with ceph-deploy then??

Post by Joao Eduardo Luis
Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1'
running. If this error persists, set 'debug mon = 10' on ceph.conf,
restart the monitor and send us the resulting log.

# find /var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1/sysvinit
/var/lib/ceph/mon/ceph-ceph-node1/store.db
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOG
/var/lib/ceph/mon/ceph-ceph-node1/store.db/CURRENT
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000006.log
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000005.ldb
/var/lib/ceph/mon/ceph-ceph-node1/store.db/MANIFEST-000004
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/keyring

# cat ceph.conf
[global]
filestore_xattr_use_omap = true
mon_host = 172.17.43.11
fsid = 18203301-f70b-40f1-b93e-fde4731f3bee
mon_initial_members = ceph-node1
auth_supported = cephx
osd_journal_size = 1024
debug mon = 10

...this is actually on ceph-admin but I have verified it is deployed
afterward to ceph-node1 (by running --overwrite-conf)

# ceph-deploy --overwrite-conf mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy
--overwrite-conf mon create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

# cat /var/log/ceph/ceph-mon.ceph-node1.log
2013-12-05 18:30:36.473193 7f8f0ada3780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 3931

Not a lot of info in there though :( I see it is PID 3931, so 3933 might
be the unwanted one?

Thanks,
Jan

[Jan Kalcic]

Hi Joao,

Post by Joao Eduardo Luis

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same
id, and the monitor you're trying to start is unable to acquire the lock.

For example the following (3931, 3933) ??

# ps aux | grep ceph
root 3303 0.0 0.7 82184 3564 ? Ss 18:08 0:00 sshd:
ceph [priv]
ceph 3306 0.0 0.4 82312 2164 ? S 18:08 0:00 sshd:
ceph at pts/0
ceph 3307 0.0 0.5 14004 2952 pts/0 Ss 18:08 0:00 -bash
root 3835 0.0 0.3 11648 1680 ? S 18:30 0:00 /bin/sh
/usr/sbin/rcceph -c /etc/ceph/ceph.conf start mon.ceph-node1
root 3930 0.0 0.2 11512 1312 ? S 18:30 0:00 bash -c
ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3931 0.0 1.0 76612 5328 ? S 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3933 0.0 1.0 101688 5096 ? Sl 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3941 0.0 1.7 69468 8796 ? S 18:30 0:00 python
/usr/bin/ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
ceph 3964 0.0 0.1 4940 968 pts/0 R+ 18:37 0:00 ps aux
ceph 3965 0.0 0.1 5712 820 pts/0 R+ 18:37 0:00 grep ceph

but I am not doing tricky things, just following basic steps in the doc
and running ceph-deploy as indicated.. what's wrong with ceph-deploy then??

Post by Joao Eduardo Luis
Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1'
running. If this error persists, set 'debug mon = 10' on ceph.conf,
restart the monitor and send us the resulting log.

# find /var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1/sysvinit
/var/lib/ceph/mon/ceph-ceph-node1/store.db
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOG
/var/lib/ceph/mon/ceph-ceph-node1/store.db/CURRENT
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000006.log
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000005.ldb
/var/lib/ceph/mon/ceph-ceph-node1/store.db/MANIFEST-000004
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/keyring

# cat ceph.conf
[global]
filestore_xattr_use_omap = true
mon_host = 172.17.43.11
fsid = 18203301-f70b-40f1-b93e-fde4731f3bee
mon_initial_members = ceph-node1
auth_supported = cephx
osd_journal_size = 1024
debug mon = 10

...this is actually on ceph-admin but I have verified it is deployed
afterward to ceph-node1 (by running --overwrite-conf)

# ceph-deploy --overwrite-conf mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy
--overwrite-conf mon create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

# cat /var/log/ceph/ceph-mon.ceph-node1.log
2013-12-05 18:30:36.473193 7f8f0ada3780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 3931

Not a lot of info in there though :( I see it is PID 3931, so 3933 might
be the unwanted one?

Thanks,
Jan

[Jan Kalcic]

Hi Joao,

Post by Joao Eduardo Luis

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same
id, and the monitor you're trying to start is unable to acquire the lock.

For example the following (3931, 3933) ??

# ps aux | grep ceph
root 3303 0.0 0.7 82184 3564 ? Ss 18:08 0:00 sshd:
ceph [priv]
ceph 3306 0.0 0.4 82312 2164 ? S 18:08 0:00 sshd:
ceph at pts/0
ceph 3307 0.0 0.5 14004 2952 pts/0 Ss 18:08 0:00 -bash
root 3835 0.0 0.3 11648 1680 ? S 18:30 0:00 /bin/sh
/usr/sbin/rcceph -c /etc/ceph/ceph.conf start mon.ceph-node1
root 3930 0.0 0.2 11512 1312 ? S 18:30 0:00 bash -c
ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3931 0.0 1.0 76612 5328 ? S 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3933 0.0 1.0 101688 5096 ? Sl 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3941 0.0 1.7 69468 8796 ? S 18:30 0:00 python
/usr/bin/ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
ceph 3964 0.0 0.1 4940 968 pts/0 R+ 18:37 0:00 ps aux
ceph 3965 0.0 0.1 5712 820 pts/0 R+ 18:37 0:00 grep ceph

but I am not doing tricky things, just following basic steps in the doc
and running ceph-deploy as indicated.. what's wrong with ceph-deploy then??

Post by Joao Eduardo Luis
Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1'
running. If this error persists, set 'debug mon = 10' on ceph.conf,
restart the monitor and send us the resulting log.

# find /var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1/sysvinit
/var/lib/ceph/mon/ceph-ceph-node1/store.db
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOG
/var/lib/ceph/mon/ceph-ceph-node1/store.db/CURRENT
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000006.log
/var/lib/ceph/mon/ceph-ceph-node1/store.db/000005.ldb
/var/lib/ceph/mon/ceph-ceph-node1/store.db/MANIFEST-000004
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/keyring

# cat ceph.conf
[global]
filestore_xattr_use_omap = true
mon_host = 172.17.43.11
fsid = 18203301-f70b-40f1-b93e-fde4731f3bee
mon_initial_members = ceph-node1
auth_supported = cephx
osd_journal_size = 1024
debug mon = 10

...this is actually on ceph-admin but I have verified it is deployed
afterward to ceph-node1 (by running --overwrite-conf)

# ceph-deploy --overwrite-conf mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy
--overwrite-conf mon create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

# cat /var/log/ceph/ceph-mon.ceph-node1.log
2013-12-05 18:30:36.473193 7f8f0ada3780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 3931

Not a lot of info in there though :( I see it is PID 3931, so 3933 might
be the unwanted one?

Thanks,
Jan

[Jan Kalcic]

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Karan Singh]

Hello Jan

I faced similar kind of errors and these are really annoying. I tried this and worked for me.

1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.


Below is my solution ( you should change it as per your hostname and IP etc)



#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and monitors UP and Running

[root at ceph-mon1 ~]# ceph status
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
health HEALTH_OK
monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB / 3512 GB avail
mdsmap e1: 0/0/1 up
[root at ceph-mon1 ~]#


All the best :-)

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Joao Eduardo Luis]

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same id,
and the monitor you're trying to start is unable to acquire the lock.

Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1' running.
If this error persists, set 'debug mon = 10' on ceph.conf, restart the
monitor and send us the resulting log.

-Joao

Post by Jan Kalcic
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Jan Kalcic]

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Karan Singh]

Hello Jan

I faced similar kind of errors and these are really annoying. I tried this and worked for me.

1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.


Below is my solution ( you should change it as per your hostname and IP etc)



#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and monitors UP and Running

[root at ceph-mon1 ~]# ceph status
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
health HEALTH_OK
monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB / 3512 GB avail
mdsmap e1: 0/0/1 up
[root at ceph-mon1 ~]#


All the best :-)

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Joao Eduardo Luis]

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same id,
and the monitor you're trying to start is unable to acquire the lock.

Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1' running.
If this error persists, set 'debug mon = 10' on ceph.conf, restart the
monitor and send us the resulting log.

-Joao

Post by Jan Kalcic
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Jan Kalcic]

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Karan Singh]

Hello Jan

I faced similar kind of errors and these are really annoying. I tried this and worked for me.

1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will give you more strength.
3. You might need to manually add monitor in cluster.


Below is my solution ( you should change it as per your hostname and IP etc)



#### login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and monitors UP and Running

[root at ceph-mon1 ~]# ceph status
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
health HEALTH_OK
monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB / 3512 GB avail
mdsmap e1: 0/0/1 up
[root at ceph-mon1 ~]#


All the best :-)

Many Thanks
Karan Singh


----- Original Message -----
From: "Jan Kalcic" <jandot at gmail.com>
To: ceph-users at lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

Hi all,

approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)

Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

Thanks,
Jan

[Joao Eduardo Luis]

Post by Jan Kalcic
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph at ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store

This is oftentime a symptom of another monitor running with the same id,
and the monitor you're trying to start is unable to acquire the lock.

Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1' running.
If this error persists, set 'debug mon = 10' on ceph.conf, restart the
monitor and send us the resulting log.

-Joao

Post by Jan Kalcic
Thanks,
Jan
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com