- ceph-users - lists.ceph.io

Many ceph commands hang. broken mgr?
by Paul Mezzanini 03 Dec '20

03 Dec '20

Ever since we jumped from 14.2.9 to .12 (and beyond) a lot of the ceph commands just hang. The mgr daemon also just stops responding to our Prometheus scrapes occasionally. A daemon restart and it wakes back up. I have nothing pointing to these being related but it feels that way. I also tried to get device health monitoring with smart up and running around that upgrade time. It never seemed to be able to pull in and report on the health across the drives. I did see the osd process firing off smartctl on occasion though so it was trying to do something. Again, I have nothing pointing to this being related but it feels like it may be. Some commands that currently hang: ceph osd pool autoscale-status ceph balancer * ceph iostat (oddly, this spit out a line of all 0 stats once and then hung) ceph fs status toggling ceph device monitoring on or off and a lot of the device health stuff too Mgr logs on disk show flavors of this: 2020-11-24 13:05:07.883 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,e,r,f,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:07.895 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,o,o,l, ,s,t,a,t,s,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:08.567 7f19e1c3e700 0 log_channel(cluster) log [DBG] : pgmap v587: 17149 pgs: 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 55 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 17082 active+clean; 2.1 PiB data, 3.5 PiB used, 2.9 PiB / 6.4 PiB avail; 108 MiB/s rd, 53 MiB/s wr, 1.20k op/s; 7525420/9900121381 objects misplaced (0.076%); 99 MiB/s, 40 objects/s recovering ceph status: cluster: id: 971a5242-f00d-421e-9bf4-5a716fcc843a health: HEALTH_WARN 1 nearfull osd(s) 1 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-03,ceph-mon-02 (age 4h) mgr: ceph-mon-01(active, since 97s), standbys: ceph-mon-03, ceph-mon-02 mds: cephfs:1 {0=ceph-mds-02=up:active} 3 up:standby osd: 843 osds: 843 up (since 13d), 843 in (since 2w); 10 remapped pgs rgw: 1 daemon active (ceph-rgw-01) task status: scrub status: mds.ceph-mds-02: idle data: pools: 16 pools, 17149 pgs objects: 1.61G objects, 2.1 PiB usage: 3.5 PiB used, 2.9 PiB / 6.4 PiB avail pgs: 6482000/9900825469 objects misplaced (0.065%) 17080 active+clean 54 active+clean+scrubbing+deep 9 active+remapped+backfilling 5 active+clean+scrubbing 1 active+remapped+backfill_wait io: client: 877 MiB/s rd, 1.8 GiB/s wr, 1.91k op/s rd, 3.33k op/s wr recovery: 136 MiB/s, 55 objects/s ceph config dump: WHO MASK LEVEL OPTION VALUE RO global advanced cluster_network 192.168.42.0/24 * global advanced mon_max_pg_per_osd 400 global advanced mon_pg_warn_max_object_skew -1.000000 global dev mon_warn_on_pool_pg_num_not_power_of_two false global advanced osd_max_backfills 2 global advanced osd_max_scrubs 4 global advanced osd_scrub_during_recovery false global advanced public_network 1xx.xx.171.0/24 10.16.171.0/24 * mon advanced mon_allow_pool_delete true mgr advanced mgr/balancer/mode none mgr advanced mgr/devicehealth/enable_monitoring false osd advanced bluestore_compression_mode passive osd advanced osd_deep_scrub_large_omap_object_key_threshold 2000000 osd advanced osd_op_queue_cut_off high * osd advanced osd_scrub_load_threshold 5.000000 mds advanced mds_beacon_grace 300.000000 mds basic mds_cache_memory_limit 16384000000 mds advanced mds_log_max_segments 256 client advanced rbd_default_features 5 client.libvirt advanced admin_socket /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok * client.libvirt basic log_file /var/log/ceph/qemu-guest-$pid.log * /etc/ceph/ceph.conf is the stub file with fsid and the mons listed. Yes I have a drive that just started to tickle the full warn limit. That's what pulled me back into the "I should fix this" mode. I'm manually adjusting the weight on that one for the time being along with slowly lowering pg_num on an oversized pool. The cluster still has this issue when in health_ok. I'm free to do a lot of debugging and poking around even though this is our production cluster. The only service I refuse to play around with is the MDS. That one bites back. Does anyone have more ideas on where to look to try and figure out what's going on? -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

2 7

add OSDs to cluster
by mj 03 Dec '20

03 Dec '20

Hi, We are wondering why adding an OSD to a healthy cluster results in a (very small percentage of) "Degraded data redundancy". (0.020%) We understand a large percentage of misplaced objects (7.622%) But since we're adding an OSD to a HEALTH_OK cluster, there should really not be any degraded data redundancy, right..? MJ

2 1

slow down keys/s in recovery
by Seena Fallah 03 Dec '20

03 Dec '20

Hi all, Is there any configuration to slow down keys/s in recovery mode? Thanks.

4 17

Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
by Hoan Nguyen Van 03 Dec '20

03 Dec '20

I have same issue. My cluster version is 14.2.1, i never meet it before. I see some information from this tracker : https://tracker.ceph.com/issues/48033

2 1

How to create single OSD with SSD db device with cephadm
by 胡玮文 03 Dec '20

03 Dec '20

Hi all, I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this. I have tried: 1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually. 2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based. 3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way. 4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does. 5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that. Any help is appreciated. Thanks.

2 4

replace osd with Octopus
by Tony Liu 02 Dec '20

02 Dec '20

Hi, I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs. Thanks! Tony

4 13

Ceph-ansible vs. Cephadm - Nautilus to Octopus and beyond
by Dave Hall 02 Dec '20

02 Dec '20

Hello. The topic of Ceph-Ansible hasn't appeared in the list for a few months, but there's been a lot of talk about Cephadm. So what are the pros and cons? Is Cephadm good enough to put Ceph-Ansible out of business, or will it still be viable beyond Pacific? For my part the Ansible approach seems more straightforward since one can just lay out a cluster in a flat file and press 'go'. BTW, I currently have a non-containerized cluster running Nautilus, but I'll eventually have to move to Octopus. Further, I'm not (yet) convinced that the container-based approach is better than bare-metal. I think I saw that Cephadm will only deploy container-based clusters. Is this a hint that bare-metal is going away in the long run? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Upgrade to 15.2.7 fails on mixed x86_64/arm64 cluster
by Bryan Stillwell 02 Dec '20

02 Dec '20

I tried upgrading my home cluster to 15.2.7 (from 15.2.5) today and it appears to be entering a loop when trying to match docker images for ceph:v15.2.7: 2020-12-01T16:47:26.761950-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:26.769581-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date. 2020-12-01T16:47:26.770096-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons... 2020-12-01T16:47:28.800426-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date. 2020-12-01T16:47:28.800878-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons... 2020-12-01T16:47:28.851819-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash... 2020-12-01T16:47:28.855595-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date. 2020-12-01T16:47:28.856283-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons... 2020-12-01T16:47:31.348345-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay 2020-12-01T16:47:35.311065-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting 2020-12-01T16:47:35.534893-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 2020-12-01T16:47:35.546444-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:35.547185-0700 mgr.aladdin.liknom [INF] Upgrade: Need to upgrade myself (mgr.aladdin.liknom) 2020-12-01T16:47:37.506337-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on ether 2020-12-01T16:47:40.770290-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on ether got new image 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e (not 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959), restarting 2020-12-01T16:47:41.172402-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e 2020-12-01T16:47:41.226550-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:41.230932-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date. 2020-12-01T16:47:41.231887-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons... 2020-12-01T16:47:43.179844-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date. 2020-12-01T16:47:43.180305-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons... 2020-12-01T16:47:43.187481-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash... 2020-12-01T16:47:43.191821-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date. 2020-12-01T16:47:43.192290-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons... 2020-12-01T16:47:45.692126-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay 2020-12-01T16:47:50.679789-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting The machines 'ether' and 'aladdin' are x86_64 machines, but 'mandalaybay' is a raspberry pi 4 (arm64). Is there a way to bypass this check to allow me to finish upgrading the cluster? Thanks, Bryan

2 1

Determine effective min_alloc_size for a specific OSD
by 胡玮文 02 Dec '20

02 Dec '20

Hi all, I’ve read from this mail list that too high bluestore_min_alloc_size will result in too much space wasted if I have many small objects. But too low bluestore_min_alloc_size will reduce performance. I’ve also read that this config can’t be changed after OSD creation. Now I want to tune this config myself. I change bluestore_min_alloc_size_hdd from default 64k to 32k, then deployed some more OSDs with cephadm. I assume old OSD will continue to use the old setting, and new OSDs will use the new setting. But how can I verify that? To be specific, how can I query the effective min_alloc_size for a specific OSD? I’ve tried to use “ceph daemon osd.X ...” but cannot find the appropriate command. I also searched the OSD logs. Thanks.

4 4

ceph in docker the log_file config is empty
by goodluck 02 Dec '20

02 Dec '20

when I use kolla to deploy ceph in docker. I found there is no ceph logs output with osds and mons. But the mgr have the ceph logs output in the logfile. While I found the ceph log_file is empty even I set the log_file config in ceph.conf. ceph daemon /var/run/ceph/ceph-osd.0.asok config show|grep log_file "log_file": "", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_cluster_log_file_level": "debug", cat /etc/ceph/ceph.conf [global] log file = /var/log/kolla/ceph/$cluster-$name.log log to syslog = false err to syslog = false log to stderr = false err to stderr = false fsid = 3722db7f-9831-473f-a591-a98bd76464d8 mon initial members = mon host = mon addr = auth cluster required = cephx auth service required = cephx auth client required = cephx #setuser match path = /var/lib/ceph/$type/$cluster-$id osd crush update on start = false osd objectstore = filestore bluestore = false [mon] mon compact on start = true mon cluster log file = /var/log/kolla/ceph/$cluster.log [ { "Id": "c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df", "Created": "2020-12-01T06:42:06.300762742Z", "Path": "dumb-init", "Args": [ "--single-child", "--", "kolla_start" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 51905, "ExitCode": 0, "Error": "", "StartedAt": "2020-12-01T06:48:05.787234081Z", "FinishedAt": "2020-12-01T06:48:05.485792802Z" }, "Image": "sha256:973a3e0d341100d5a53f64ef7f9b440aadfce01a9ca52e9637678361aa7fb1ba", "ResolvConfPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/resolv.conf", "HostnamePath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/hostname", "HostsPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/hosts", "LogPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df-json.log", "Name": "/ceph_osd_0", "RestartCount": 0, "Driver": "btrfs", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "unconfined", "ExecIDs": null, "HostConfig": { "Binds": [ "/etc/localtime:/etc/localtime:ro", "/etc/kolla/ceph-osd/:/var/lib/kolla/config_files/:ro", "/var/lib/ceph/osd/d4e7ed3e-f997-463b-9350-ffa9f5c1b429:/var/lib/ceph/osd/ceph-0:rw", "kolla_logs:/var/log/kolla/:rw", "/var/run/:/var/run/:rw", "/dev/:/dev/:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": { "max-file": "5", "max-size": "50m" } }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "unless-stopped", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": true, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "label=disable" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": null, "Name": "btrfs" }, "Mounts": [ { "Type": "volume", "Name": "kolla_logs", "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "/var/run", "Destination": "/var/run", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/dev", "Destination": "/dev", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/localtime", "Destination": "/etc/localtime", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/kolla/ceph-osd", "Destination": "/var/lib/kolla/config_files", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ceph/osd/d4e7ed3e-f997-463b-9350-ffa9f5c1b429", "Destination": "/var/lib/ceph/osd/ceph-0", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "compute-taishan-01", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "OSD_BS_FSUUID=d4e7ed3e-f997-463b-9350-ffa9f5c1b429", "OSD_STORETYPE=filestore", "OSD_ID=0", "KOLLA_SERVICE_NAME=ceph-osd-0", "KOLLA_INSTALL_TYPE=source", "KOLLA_INSTALL_METATYPE=mixed", "KOLLA_BASE_ARCH=aarch64", "DEBIAN_FRONTEND=noninteractive", "TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728", "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS", "JOURNAL_PARTITION=/dev/disk/by-partuuid/80664f67-6196-496c-81f2-02d974edbe52", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG=en_US.UTF-8", "KOLLA_BASE_DISTRO=ubuntu", "KOLLA_DISTRO_PYTHON_VERSION=3.6", "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id un)@$(hostname -s) $(pwd)]$ " ], "Cmd": [ "kolla_start" ], "Image": "10.12.254.254:4020/cszt-v4-anke/ubuntu-source-ceph-osd:4.0.1", "Volumes": { "/dev/": {}, "/etc/localtime": {}, "/var/lib/ceph/osd/ceph-0": {}, "/var/lib/kolla/config_files/": {}, "/var/log/kolla/": {}, "/var/run/": {} }, "WorkingDir": "", "Entrypoint": [ "dumb-init", "--single-child", "-" ], "OnBuild": null, "Labels": { "build-date": "20201019", "kolla_version": "0.1.0", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "ceph-osd" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "5bdcea4f0b2cc2f54e4d6c4abad0ca60ef080c74e7419ac75cde837025aedffe", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/default", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "host": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "b5bfa3839097ea5932eaef6cd2b05ed2190eeec845f765f5ead543247787f860", "EndpointID": "67275cf88430263a8e49e5be9109a9ca2fedb3b99b348840072b54e52d2bdb57", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null }

2 2