[ceph-users] 'ceph-deploy osd create' and filestore OSDs

[Matthew Pounsett]

I'm in the process of updating some development VMs that use ceph-fs. It
looks like recent updates to ceph have deprecated the 'ceph-deploy osd
prepare' and 'activate' commands in favour of the previously-optional
'create' command.

We're using filestore OSDs on these VMs, but I can't seem to figure out the
syntax that ceph-deploy wants to specify the path.

Where we used to use:
ceph-deploy osd prepare tldhost01:/var/local/osd0 tldhost02:/var/local/osd0
tldhost03:/var/local/osd0 tldhost04:/var/local/osd0
ceph-deploy osd activate tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0
.. a similar path syntax for 'osd create' generates an error.

The help output for 'ceph-deploy osd create --help' seems to suggest the
following could work:
ceph-deploy osd-create --filestore tldhost01:/var/local/osd0 ...

However it does not.

What's the actual process for using ceph-deploy to set up filestore OSDs
now?

[Vasu Kulkarni]

Post by Matthew Pounsett
I'm in the process of updating some development VMs that use ceph-fs. It
looks like recent updates to ceph have deprecated the 'ceph-deploy osd
prepare' and 'activate' commands in favour of the previously-optional
'create' command.
We're using filestore OSDs on these VMs, but I can't seem to figure out
the syntax that ceph-deploy wants to specify the path.
ceph-deploy osd prepare tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0
ceph-deploy osd activate tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0
.. a similar path syntax for 'osd create' generates an error.
The help output for 'ceph-deploy osd create --help' seems to suggest the
ceph-deploy osd-create --filestore tldhost01:/var/local/osd0 ...

what version of ceph-deploy are you using and what is the version of ceph?
If you are using latest ceph-deploy from pypi, it
will default to ceph-volume and you have to specifiy '--journal' option
which is mandatory for filestore, since you are using a VM, you can
partition part
of the disk for data and and another smaller partition for journal.

Post by Matthew Pounsett
However it does not.
What's the actual process for using ceph-deploy to set up filestore OSDs
now?
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Matthew Pounsett]

Going to take another stab at this...

We have a development environment–made up of VMs–for developing and testing
the deployment tools for a particular service that depends on cephfs for
sharing state data between hosts. In production we will be using filestore
OSDs because of the very low volume of data (a few hundred kilobytes) and
the very low rate of change. There's insufficient performance benefit for
it to make sense for us to create an operational exception by configuring
the hardware differently from everything else just to have separate block
devices.

Unfortunately, even though the documentation says that filestore OSDs are
well tested and supported, they don't seem to be well documented.

In a recent test of our deployment tools (using Kraken on Centos/7) the
'ceph-deploy osd' steps failed. Assuming this was simply because Kraken is
now so far past EOL that it just wasn't supported properly on an updated
Centos box I started working on an update to Luminos. However, I've since
discovered that the problem is actually that ceph-deploy's OSD 'prepare'
and 'activate' commands have been deprecated regardless of ceph release. I
now realize that ceph-deploy is maintained independently from the rest of
ceph, but not documented independently, so the ceph documentation that
references ceph-deploy seems to now be frequently incorrect.

Except where mentioned otherwise, the rest of this is using the latest
Luminos from the download.ceph.com Yum archive (12.2.10) with ceph-deploy
2.0.1

Our scripts, written for Kraken, were doing this to create filestore OSDs
on four dev VMs:
ceph-deploy osd prepare tldhost01:/var/local/osd0 tldhost02:/var/local/osd0
tldhost03:/var/local/osd0 tldhost04:/var/local/osd0
ceph-deploy osd activate tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0

Both 'prepare' and 'activate' seem to be completely deprecated now (neither
shows up in the help output generated when the above commands fail) in
Kraken and Luminos. This seems to have changed in the last 60 days or so.
The above commands now fail with this error:
usage: ceph-deploy osd [-h] {list,create} ...
ceph-deploy osd: error: argument subcommand: invalid choice: 'prepare'
(choose from 'list', 'create')

I'm trying to figure out the 'ceph-deploy osd create' syntax to duplicate
the above, but the documentation is no help. The Luminos documentation
still shows the above prepare/activate syntax should be valid, and
continues to show the journal path as being optional for the 'ceph-deploy
osd create' command.
<
http://docs.ceph.com/docs/luminous/rados/deployment/ceph-deploy-osd/#prepare-osds

.

The same documentation for Mimic seems to be updated for the new
ceph-deploy syntax, including the elimination of 'prepare' and 'activate',
but doesn't include specifics for a filestore deployment:
<
http://docs.ceph.com/docs/mimic/rados/deployment/ceph-deploy-osd/#create-osds
The new syntax seems to suggest I can now only do one host at a time, and
must split up the host, data, and journal values. After much trial and
error I've also found it's now required to specify the journal path, but
not knowing for sure what ceph-deploy was doing in the background with the
journal path by default before, I've had a hard time sorting out things to
try with the new syntax. Following the above logic, and skipping over a
few things I've tried to get here, in my latest attempt I've moved the ceph
data down one level in the directory tree and added a journal directory.
Where tldhost01 is localhost:
mkdir -p /var/local/ceph/{osd0,journal}
ceph-deploy osd create --data /var/local/ceph/osd0 --journal
/var/local/ceph/journal --filestore tldhost01

The assumption in this is that --data and --journal accept filesystem paths
the same way the 'prepare' and 'activate' commands used to, but that is
clearly not the case, as the above complains that I have not supplied block
devices. It looks like --filestore is not doing what I hoped.

At this point I'm stuck. I've gone through all the documentation I can
find, and although it frequently mentions that ceph started by storing its
data on the filesystem and that doing so is still well supported, I can't
actually find any documentation that says how to do it. When we started
this project we used information from the quickstart documents to get
filestore OSDs set up, but even the quickstart documents don't seem to
supply that information (anymore).

Thanks for any pointers anyone can supply.
Matt

[Vasu Kulkarni]

Post by Matthew Pounsett
Going to take another stab at this...
We have a development environment–made up of VMs–for developing and
testing the deployment tools for a particular service that depends on
cephfs for sharing state data between hosts. In production we will be
using filestore OSDs because of the very low volume of data (a few hundred
kilobytes) and the very low rate of change. There's insufficient
performance benefit for it to make sense for us to create an operational
exception by configuring the hardware differently from everything else just
to have separate block devices.
Unfortunately, even though the documentation says that filestore OSDs are
well tested and supported, they don't seem to be well documented.

Sadly the documentation on ceph-deploy is bit behind :(

Post by Matthew Pounsett
In a recent test of our deployment tools (using Kraken on Centos/7) the
'ceph-deploy osd' steps failed. Assuming this was simply because Kraken is
now so far past EOL that it just wasn't supported properly on an updated
Centos box I started working on an update to Luminos. However, I've since
discovered that the problem is actually that ceph-deploy's OSD 'prepare'
and 'activate' commands have been deprecated regardless of ceph release. I
now realize that ceph-deploy is maintained independently from the rest of
ceph, but not documented independently, so the ceph documentation that
references ceph-deploy seems to now be frequently incorrect.
Except where mentioned otherwise, the rest of this is using the latest
Luminos from the download.ceph.com Yum archive (12.2.10) with ceph-deploy
2.0.1
Our scripts, written for Kraken, were doing this to create filestore OSDs
ceph-deploy osd prepare tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0
ceph-deploy osd activate tldhost01:/var/local/osd0
tldhost02:/var/local/osd0 tldhost03:/var/local/osd0
tldhost04:/var/local/osd0

you are using HOST:DIR option which is bit old and I think it was supported
till jewel, since you are using 2.0.1 you should be using only 'osd
create' with logical volume or full block device as defined here:
http://docs.ceph.com/docs/mimic/ceph-volume/lvm/prepare/#ceph-volume-lvm-prepare-filestore
, ceph-deploy calls ceph-volume with the same underlying syntax. Since this
is a VM, you can just add addtional smaller raw devices (eg: /dev/sde ) and
use that for journal.

Post by Matthew Pounsett
Both 'prepare' and 'activate' seem to be completely deprecated now
(neither shows up in the help output generated when the above commands
fail) in Kraken and Luminos. This seems to have changed in the last 60
usage: ceph-deploy osd [-h] {list,create} ...
ceph-deploy osd: error: argument subcommand: invalid choice: 'prepare'
(choose from 'list', 'create')
I'm trying to figure out the 'ceph-deploy osd create' syntax to duplicate
the above, but the documentation is no help. The Luminos documentation
still shows the above prepare/activate syntax should be valid, and
continues to show the journal path as being optional for the 'ceph-deploy
osd create' command.
<
http://docs.ceph.com/docs/luminous/rados/deployment/ceph-deploy-osd/#prepare-osds

.

The same documentation for Mimic seems to be updated for the new
ceph-deploy syntax, including the elimination of 'prepare' and 'activate',
<
http://docs.ceph.com/docs/mimic/rados/deployment/ceph-deploy-osd/#create-osds
The new syntax seems to suggest I can now only do one host at a time, and
must split up the host, data, and journal values. After much trial and
error I've also found it's now required to specify the journal path, but
not knowing for sure what ceph-deploy was doing in the background with the
journal path by default before, I've had a hard time sorting out things to
try with the new syntax. Following the above logic, and skipping over a
few things I've tried to get here, in my latest attempt I've moved the ceph
data down one level in the directory tree and added a journal directory.
mkdir -p /var/local/ceph/{osd0,journal}
ceph-deploy osd create --data /var/local/ceph/osd0 --journal
/var/local/ceph/journal --filestore tldhost01
The assumption in this is that --data and --journal accept filesystem
paths the same way the 'prepare' and 'activate' commands used to, but that
is clearly not the case, as the above complains that I have not supplied
block devices. It looks like --filestore is not doing what I hoped.

you cannot use DIR as option to --data and --journal as explained above.
--filestore doesn't actually mean filesystem option here, It needs a block
device and will automatically create fs on the block device.

Post by Matthew Pounsett
At this point I'm stuck. I've gone through all the documentation I can
find, and although it frequently mentions that ceph started by storing its
data on the filesystem and that doing so is still well supported, I can't
actually find any documentation that says how to do it. When we started
this project we used information from the quickstart documents to get
filestore OSDs set up, but even the quickstart documents don't seem to
supply that information (anymore).
Thanks for any pointers anyone can supply.
Matt
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Matthew Pounsett]

Post by Vasu Kulkarni
you are using HOST:DIR option which is bit old and I think it was
supported till jewel, since you are using 2.0.1 you should be using only
http://docs.ceph.com/docs/mimic/ceph-volume/lvm/prepare/#ceph-volume-lvm-prepare-filestore
, ceph-deploy calls ceph-volume with the same underlying syntax. Since this
is a VM, you can just add addtional smaller raw devices (eg: /dev/sde ) and
use that for journal.

As explained above, we can't just create smaller raw devices. Yes, these
are VMs but they're meant to replicate physical servers that will be used
in production, where no such volumes are available.

So.. we're trying to figure out how to replicate the configuration we've
been using where the ceph data is stored on the OS filesystem.

[Vasu Kulkarni]

Post by Matthew Pounsett

Post by Vasu Kulkarni
you are using HOST:DIR option which is bit old and I think it was
supported till jewel, since you are using 2.0.1 you should be using only
http://docs.ceph.com/docs/mimic/ceph-volume/lvm/prepare/#ceph-volume-lvm-prepare-filestore
, ceph-deploy calls ceph-volume with the same underlying syntax. Since this
is a VM, you can just add addtional smaller raw devices (eg: /dev/sde ) and
use that for journal.

As explained above, we can't just create smaller raw devices. Yes, these
are VMs but they're meant to replicate physical servers that will be used
in production, where no such volumes are available.

In that case you will have to use the same version of ceph-deploy you have
used to deploy the original systems. you cannot do this now with newer
version.

Post by Matthew Pounsett
So.. we're trying to figure out how to replicate the configuration we've
been using where the ceph data is stored on the OS filesystem.

[Matthew Pounsett]

Post by Vasu Kulkarni

Post by Matthew Pounsett
As explained above, we can't just create smaller raw devices. Yes, these
are VMs but they're meant to replicate physical servers that will be used
in production, where no such volumes are available.

In that case you will have to use the same version of ceph-deploy you have
used to deploy the original systems. you cannot do this now with newer
version.

Post by Matthew Pounsett
So.. we're trying to figure out how to replicate the configuration we've
been using where the ceph data is stored on the OS filesystem.

That's irritating, but understood. :)

Is there a way we can easily set that up without trying to use outdated
tools? Presumably if ceph still supports this as the docs claim, there's a
way to get it done without using ceph-deploy?

If not.. I guess worst case we can revisit having the hardware group to
repartition the drives with separate, tiny, data and journal partitions...
that is assuming it's not required that those functions have access to
whole disks. We would rather avoid that if at all possible, though.

[Vasu Kulkarni]

Post by Matthew Pounsett

Post by Vasu Kulkarni

Post by Matthew Pounsett
As explained above, we can't just create smaller raw devices. Yes,
these are VMs but they're meant to replicate physical servers that will be
used in production, where no such volumes are available.

In that case you will have to use the same version of ceph-deploy you
have used to deploy the original systems. you cannot do this now with newer
version.

Post by Matthew Pounsett
So.. we're trying to figure out how to replicate the configuration we've
been using where the ceph data is stored on the OS filesystem.

That's irritating, but understood. :)
Is there a way we can easily set that up without trying to use outdated
tools? Presumably if ceph still supports this as the docs claim, there's a
way to get it done without using ceph-deploy?

It might be more involved if you are trying to setup manually, you can give
1.5.38 a try(not that old) and see if it works
https://pypi.org/project/ceph-deploy/1.5.38/

Post by Matthew Pounsett
If not.. I guess worst case we can revisit having the hardware group to
repartition the drives with separate, tiny, data and journal partitions...
that is assuming it's not required that those functions have access to
whole disks. We would rather avoid that if at all possible, though.

[Matthew Pounsett]

Post by Vasu Kulkarni

Post by Matthew Pounsett
Is there a way we can easily set that up without trying to use outdated
tools? Presumably if ceph still supports this as the docs claim, there's a
way to get it done without using ceph-deploy?

It might be more involved if you are trying to setup manually, you can
give 1.5.38 a try(not that old) and see if it works
https://pypi.org/project/ceph-deploy/1.5.38/

Not that old now.. but eventually it will be. :)

The goal was to revisit our deployment tools later (after getting the rest
of the service working) and replace ceph-deploy with direct configuration
of all the machines, so maybe having a deprecated piece of software around
that needs to be replaced will help with that motivation when the time
comes.

Thanks for your help!
Matt

[Alfredo Deza]

Post by Matthew Pounsett

Is there a way we can easily set that up without trying to use outdated tools? Presumably if ceph still supports this as the docs claim, there's a way to get it done without using ceph-deploy?

It might be more involved if you are trying to setup manually, you can give 1.5.38 a try(not that old) and see if it works https://pypi.org/project/ceph-deploy/1.5.38/

Vasu has pointed out pretty much everything correctly. If you don't
want the new syntax, 1.5.38 is what you want. Would like to point out
a couple of things here:

* ceph-deploy has a pretty good changelog that gets updated for every
release. The 2.0.0 release has a backwards incompatibility notice
explaining much of the issues you've raised:
http://docs.ceph.com/ceph-deploy/docs/changelog.html

* Deploying to directories is not supported anymore with ceph-deploy,
and soon to be impossible in Ceph with tooling. If you are trying to
replicate production environments with smaller devices, you can still
do this
with some manual work: ceph-deploy (and ceph-volume on the remote
machine) can consume logical volumes, which could be easily set up to
be on a loop device. That is what we do for some of the functional
testing:
we create sparse files of 10GB and attach them to a loop device, then
create an LV on top.

Post by Matthew Pounsett
Not that old now.. but eventually it will be. :)
The goal was to revisit our deployment tools later (after getting the rest of the service working) and replace ceph-deploy with direct configuration of all the machines, so maybe having a deprecated piece of software around that needs to be replaced will help with that motivation when the time comes.
Thanks for your help!
Matt
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com