Hi,
Could you tell me whether it's ok to remove device_health_metrics pool
after disabling device monitoring feature?
I don't use device monitoring feature because I capture hardware
information from other way.
However, after disabling this feature, device_health_metrics pool stll
exists.
I don't want to concern HEALTH_WARN caused by problems in PGs of this pool.
As a result of reading the source code of device monitoring module,
it seems to be safe to remove this pool. Is my understanding correct?
Thanks,
Satoru
Hello,
I have a ceph nautilus cluster. The crushmap is organized with 2 rooms,
servers in these rooms and osd in these servers, I have a crush rule to
replicate data over the servers in different rooms.
Now, I want to add a new server in one of the rooms. My point is that I
would like to specify the room of this new server BEFORE creating osd in
this server (so the data added to the osd will be directly at the right
location). My problem is that it seems that servers appears in the
crushmap only when they have osds... and when you create a first osd,
the server is inserted in the crushmap under the default bucket (so not
in a room and then the first data stored in this osd will not be at the
correct location). I could move it after (if I do it rapidly, there will
be no that much data to move after), but I was wondering if there is a
way to either define the position of a server in the crushmap hierarchy
before creating osd or eventually to specify the room when creating the
first osd ?
F.
Ever since we jumped from 14.2.9 to .12 (and beyond) a lot of the ceph commands just hang. The mgr daemon also just stops responding to our Prometheus scrapes occasionally. A daemon restart and it wakes back up. I have nothing pointing to these being related but it feels that way.
I also tried to get device health monitoring with smart up and running around that upgrade time. It never seemed to be able to pull in and report on the health across the drives. I did see the osd process firing off smartctl on occasion though so it was trying to do something. Again, I have nothing pointing to this being related but it feels like it may be.
Some commands that currently hang:
ceph osd pool autoscale-status
ceph balancer *
ceph iostat (oddly, this spit out a line of all 0 stats once and then hung)
ceph fs status
toggling ceph device monitoring on or off and a lot of the device health stuff too
Mgr logs on disk show flavors of this:
2020-11-24 13:05:07.883 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,e,r,f,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch
2020-11-24 13:05:07.895 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,o,o,l, ,s,t,a,t,s,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch
2020-11-24 13:05:08.567 7f19e1c3e700 0 log_channel(cluster) log [DBG] : pgmap v587: 17149 pgs: 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 55 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 17082 active+clean; 2.1 PiB data, 3.5 PiB used, 2.9 PiB / 6.4 PiB avail; 108 MiB/s rd, 53 MiB/s wr, 1.20k op/s; 7525420/9900121381 objects misplaced (0.076%); 99 MiB/s, 40 objects/s recovering
ceph status:
cluster:
id: 971a5242-f00d-421e-9bf4-5a716fcc843a
health: HEALTH_WARN
1 nearfull osd(s)
1 pool(s) nearfull
services:
mon: 3 daemons, quorum ceph-mon-01,ceph-mon-03,ceph-mon-02 (age 4h)
mgr: ceph-mon-01(active, since 97s), standbys: ceph-mon-03, ceph-mon-02
mds: cephfs:1 {0=ceph-mds-02=up:active} 3 up:standby
osd: 843 osds: 843 up (since 13d), 843 in (since 2w); 10 remapped pgs
rgw: 1 daemon active (ceph-rgw-01)
task status:
scrub status:
mds.ceph-mds-02: idle
data:
pools: 16 pools, 17149 pgs
objects: 1.61G objects, 2.1 PiB
usage: 3.5 PiB used, 2.9 PiB / 6.4 PiB avail
pgs: 6482000/9900825469 objects misplaced (0.065%)
17080 active+clean
54 active+clean+scrubbing+deep
9 active+remapped+backfilling
5 active+clean+scrubbing
1 active+remapped+backfill_wait
io:
client: 877 MiB/s rd, 1.8 GiB/s wr, 1.91k op/s rd, 3.33k op/s wr
recovery: 136 MiB/s, 55 objects/s
ceph config dump:
WHO MASK LEVEL OPTION VALUE RO
global advanced cluster_network 192.168.42.0/24 *
global advanced mon_max_pg_per_osd 400
global advanced mon_pg_warn_max_object_skew -1.000000
global dev mon_warn_on_pool_pg_num_not_power_of_two false
global advanced osd_max_backfills 2
global advanced osd_max_scrubs 4
global advanced osd_scrub_during_recovery false
global advanced public_network 1xx.xx.171.0/24 10.16.171.0/24 *
mon advanced mon_allow_pool_delete true
mgr advanced mgr/balancer/mode none
mgr advanced mgr/devicehealth/enable_monitoring false
osd advanced bluestore_compression_mode passive
osd advanced osd_deep_scrub_large_omap_object_key_threshold 2000000
osd advanced osd_op_queue_cut_off high *
osd advanced osd_scrub_load_threshold 5.000000
mds advanced mds_beacon_grace 300.000000
mds basic mds_cache_memory_limit 16384000000
mds advanced mds_log_max_segments 256
client advanced rbd_default_features 5
client.libvirt advanced admin_socket /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok *
client.libvirt basic log_file /var/log/ceph/qemu-guest-$pid.log *
/etc/ceph/ceph.conf is the stub file with fsid and the mons listed.
Yes I have a drive that just started to tickle the full warn limit. That's what pulled me back into the "I should fix this" mode. I'm manually adjusting the weight on that one for the time being along with slowly lowering pg_num on an oversized pool. The cluster still has this issue when in health_ok.
I'm free to do a lot of debugging and poking around even though this is our production cluster. The only service I refuse to play around with is the MDS. That one bites back. Does anyone have more ideas on where to look to try and figure out what's going on?
--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfmeec(a)rit.edu
CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.
------------------------
Hi,
We are wondering why adding an OSD to a healthy cluster results in a
(very small percentage of) "Degraded data redundancy". (0.020%)
We understand a large percentage of misplaced objects (7.622%)
But since we're adding an OSD to a HEALTH_OK cluster, there should
really not be any degraded data redundancy, right..?
MJ
I have same issue. My cluster version is 14.2.1, i never meet it before. I
see some information from this tracker :
https://tracker.ceph.com/issues/48033
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this.
I have tried:
1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that.
Any help is appreciated. Thanks.
Hi,
I did some search about replacing osd, and found some different
steps, probably for different release?
Is there recommended process to replace an osd with Octopus?
Two cases here:
1) replace HDD whose WAL and DB are on a SSD.
1-1) failed disk is replaced by the same model.
1-2) working disk is replaced by bigger one.
2) replace the SSD holding WAL and DB for multiple HDDs.
Thanks!
Tony
Hello.
The topic of Ceph-Ansible hasn't appeared in the list for a few months,
but there's been a lot of talk about Cephadm. So what are the pros and
cons? Is Cephadm good enough to put Ceph-Ansible out of business, or
will it still be viable beyond Pacific? For my part the Ansible
approach seems more straightforward since one can just lay out a cluster
in a flat file and press 'go'.
BTW, I currently have a non-containerized cluster running Nautilus, but
I'll eventually have to move to Octopus. Further, I'm not (yet)
convinced that the container-based approach is better than bare-metal.
I think I saw that Cephadm will only deploy container-based clusters.
Is this a hint that bare-metal is going away in the long run?
Thanks.
-Dave
--
Dave Hall
Binghamton University
kdhall(a)binghamton.edu
I tried upgrading my home cluster to 15.2.7 (from 15.2.5) today and it appears to be entering a loop when trying to match docker images for ceph:v15.2.7:
2020-12-01T16:47:26.761950-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons...
2020-12-01T16:47:26.769581-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date.
2020-12-01T16:47:26.770096-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons...
2020-12-01T16:47:28.800426-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date.
2020-12-01T16:47:28.800878-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons...
2020-12-01T16:47:28.851819-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash...
2020-12-01T16:47:28.855595-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date.
2020-12-01T16:47:28.856283-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons...
2020-12-01T16:47:31.348345-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay
2020-12-01T16:47:35.311065-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting
2020-12-01T16:47:35.534893-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959
2020-12-01T16:47:35.546444-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons...
2020-12-01T16:47:35.547185-0700 mgr.aladdin.liknom [INF] Upgrade: Need to upgrade myself (mgr.aladdin.liknom)
2020-12-01T16:47:37.506337-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on ether
2020-12-01T16:47:40.770290-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on ether got new image 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e (not 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959), restarting
2020-12-01T16:47:41.172402-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e
2020-12-01T16:47:41.226550-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons...
2020-12-01T16:47:41.230932-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date.
2020-12-01T16:47:41.231887-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons...
2020-12-01T16:47:43.179844-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date.
2020-12-01T16:47:43.180305-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons...
2020-12-01T16:47:43.187481-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash...
2020-12-01T16:47:43.191821-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date.
2020-12-01T16:47:43.192290-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons...
2020-12-01T16:47:45.692126-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay
2020-12-01T16:47:50.679789-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting
The machines 'ether' and 'aladdin' are x86_64 machines, but 'mandalaybay' is a raspberry pi 4 (arm64).
Is there a way to bypass this check to allow me to finish upgrading the cluster?
Thanks,
Bryan