This. You can even constrain placement by size or model number.
On Aug 2, 2023, at 6:53 AM, Eugen Block <eblock@nde.ag> wrote:
But that could be done easily like this:
service_type: osd service_id: ssd-db service_name: osd.ssd-db placement: hosts: - storage01 - storage02 ... spec: block_db_size: 64G data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
Anyway, I would expect that fixing the drivegroup config would fix your issue, but I'm not sure either.
Zitat von Kilian Ries <mail@kilian-ries.de>:
Yes i need specific device paths because all HDD / SSD are the same size / same vendor etc. I group multiple HDDs with an exclusive SSD for cacheing, for example:
spec:
data_devices:
paths:
- /dev/sdh
- /dev/sdi
- /dev/sdj
- /dev/sdk
- /dev/sdl
db_devices:
paths:
- /dev/sdf
filter_logic: AND
objectstore: bluestore
________________________________ Von: Eugen Block <eblock@nde.ag> Gesendet: Mittwoch, 2. August 2023 08:13:41 An: ceph-users@ceph.io Betreff: [ceph-users] Re: Disk device path changed - cephadm faild to apply osd service
Do you really need device paths in your configuration? You could use other criteria like disk sizes, vendors, rotational flag etc. If you really want device paths you'll probably need to ensure they're persistent across reboots via udev rules.
Zitat von Kilian Ries <mail@kilian-ries.de>:
Hi,
it seems that after reboot / OS update my disk labels / device paths may have changed. Since then i get an error like this:
CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-12-22_hdd-2
###
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada -e NODE_NAME=ceph-osd07.intern -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd-12-22_hdd-2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/run/ceph:z -v /var/log/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/log/ceph:z -v /var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2cvmr5lf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpb38cuw7q:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9 025b6db2e8be9173186e068a79a0da5a54ada lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq --db-devices /dev/sdg --yes --no-systemd
/bin/docker: stderr Traceback (most recent call last):
/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
/bin/docker: stderr self.main(self.argv)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/bin/docker: stderr return f(*a, **kw)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 192, in dispatch
/bin/docker: stderr instance = mapper.get(arg)(argv[count:])
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 348, in __init__
/bin/docker: stderr self.args = parser.parse_args(argv)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1734, in parse_args
/bin/docker: stderr args, argv = self.parse_known_args(args, namespace)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1766, in parse_known_args
/bin/docker: stderr namespace, args = self._parse_known_args(args, namespace)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1954, in _parse_known_args
/bin/docker: stderr positionals_end_index = consume_positionals(start_index)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1931, in consume_positionals
/bin/docker: stderr take_action(action, args)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1824, in take_action
/bin/docker: stderr argument_values = self._get_values(action, argument_strings)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in _get_values
/bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings]
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in <listcomp>
/bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings]
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2294, in _get_value
/bin/docker: stderr result = type_func(arg_string)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 116, in __call__
/bin/docker: stderr return self._format_device(self._is_valid_device())
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 127, in _is_valid_device
/bin/docker: stderr super()._is_valid_device(raise_sys_exit=False)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 104, in _is_valid_device
/bin/docker: stderr super()._is_valid_device()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 69, in _is_valid_device
/bin/docker: stderr super()._is_valid_device()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 47, in _is_valid_device
/bin/docker: stderr raise RuntimeError("Device {} has partitions.".format(self.dev_path))
/bin/docker: stderr RuntimeError: Device /dev/sdq has partitions.
Traceback (most recent call last):
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9309, in <module>
main()
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9297, in main
r = ctx.func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1941, in _infer_config
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1872, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1969, in _infer_image
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1859, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 5366, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1661, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
###
/dev/sdg is my boot device at the moment wich was formerly /dev/sda.
Is it safe to edit the ceph orch yaml file and change the device pathes to the new format? Like this:
ceph orch ls --service_name=<service-name> --export > myservice.yaml
vi (change device pathes in spec -> data_devices -> path | db_devices -> path)
ceph orch apply -i myservice.yaml [--dry-run]
Is that ok / expected behaviour? Or is there a better way?
However can see that mgr detects new devices in ceph orch log:
mgr.ceph-mon03.lrfomu [INF] Detected new or changed devices on ceph-osd07
Regards,
Kilian
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io