November 2020 - ceph-users

Re: cephadm POC deployment with two networks, can't mount cephfs

by Juan Miguel Olmo Martinez

Hi Oliver Review this "step by step" guide to see if you forgot something: BR NFS: 1. chmod +x cephadm 2. ./cephadm bootstrap - Record dashboard user & password printed out at the end 3. ADD OTHER HOSTS (assuming 3+ total after adding) 4. ./cephadm shell 5. ceph orch apply osd --all-available-devices 6. ceph fs volume create test 1 7. ceph orch apply mds test 3 8. ceph nfs cluster create cephfs testnfs 9. ceph nfs cluster info testnfs - (verify hostname, ip and port are listed) - Record ip and port for later 10. ceph nfs export create cephfs test testnfs /cephfs 11. ceph auth ls - (check “client.testnfs1” keyring is present) 12. ceph nfs export get testnfs /cephfs - (should have output) 13. rados -p nfs-ganesha -N testnfs get export-1 - testnfs/cephnfs - (check that export was successfully created) 14. ceph nfs export ls testnfs - (should show pseudo path “/cephfs”) 15. Verify nfs export exists on dashboard - Login to dashboard with credentials from bootstrap - URL will be https://{host-ip}:8443/ - Navigate to NFS page - Table should contain the export you just created 16. Exit shell - Command should just be “exit” 17. systemctl status nfs-server - If service is listed as inactive, run “systemctl start nfs-server” - Run “systemctl status nfs-server”. Should now be active 18. sudo mount -t nfs -o port={nfs-port} {nfs-ip}:/cephfs /mnt - Port and ip should be from “ceph nfs cluster info testnfs” command ran earlier Ex: mount -t nfs -o port=2049 10.8.128.94:/cephfs /mnt/cephfs/ Then give mount command -> to check if its mounted #mount Output: 10.8.128.94:/cephfs on /mnt/cephfs type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.8.128.94,local_lock=none,addr=10.8.128.94) -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

3 years, 5 months

1
0
0 0

RGW pubsub deprecation

by Yuval Lifshitz

Dear Community, Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes in buckets and objects: "bucket notifications" [1] and "pubsub" [2]. In "bucket notifications" (="push mode") the events are sent from the RGW to an external entity (kafka, rabbitmq etc.), while in "pubsub" (="pull mode") the events are synched with a special zone, where they are stored and could be later fetched by an external app. From communications that I've seen so far, users preferred to use "bucket notifications" over "pubsub". Since supporting both modes has maintenance overhead, I was considering deprecating "pubsub". However, before doing that I would like to see what the community has to say! So, if you are currently using pubsub, or plan to use it, as "pull mode" fits your usecase better than "push mode" please chime in. Yuval [1] https://docs.ceph.com/en/latest/radosgw/notifications/ [2] https://docs.ceph.com/en/latest/radosgw/pubsub-module/

3 years, 5 months

1
0
0 0

cephadm POC deployment with two networks, can't mount cephfs

by Oliver Weinmann

Hi, I setup a small 3 node cluster as a POC. I bootstrapped the cluster with separate networks for frontend (public network 192.168.30.0/24) and backend (cluster network 192.168.41.0/24). 1st small question: After the bootstrap, I recognized that I had mixed up cluster and public network. :( Is there a way to fix this on a running cluster? Last resort I would rebuild the cluster. Never the less I can't mount cephfs on a Linux client using any of the two networks. My linux client is CentOS 7 (latest updates) and has 3 nics, two of them in one of the public and cluster networks. I bootstrapped the cluster using the following conf file to have two networks: /root/ceph.conf: [global] public network = 192.168.41.0/24 cluster network = 192.168.30.0/24 cephadm bootstrap -c /root/ceph.conf --mon-ip 192.168.30.11 I have 2 mons, one running on the bootstrap host (192.168.30.11 / 192.168.41.11) and one (gedaopl01 192.168.30.12/ 192.168.41.12) running on one of the 3 osds: [root@gedasvl02 ~]# ceph -s cluster: id: dad3c9fa-1ec7-11eb-94d6-005056b703af health: HEALTH_OK services: mon: 2 daemons, quorum gedasvl02,gedaopl01 (age 5h) mgr: gedasvl02.cspuee(active, since 12h), standbys: gedaopl01.llogef mds: cephfs:1 {0=cephfs.gedaopl03.prrkll=up:active} 1 up:standby osd: 3 osds: 3 up (since 11h), 3 in (since 11h) task status: scrub status: mds.cephfs.gedaopl03.prrkll: idle data: pools: 3 pools, 81 pgs objects: 29 objects, 2.2 KiB usage: 450 GiB used, 407 GiB / 857 GiB avail pgs: 81 active+clean [root@gedasvl02 ~]# ceph osd metadata 2 | grep addr "back_addr": "[v2:192.168.30.12:6800/3112350288,v1:192.168.30.12:6801/3112350288]", "front_addr": "[v2:192.168.41.12:6800/3112350288,v1:192.168.41.12:6801/3112350288]", "hb_back_addr": "[v2:192.168.30.12:6802/3112350288,v1:192.168.30.12:6803/3112350288]", "hb_front_addr": "[v2:192.168.41.12:6802/3112350288,v1:192.168.41.12:6803/3112350288]", Now when I try to mount cephfs from the linux client, the mount command is just stuck and runs into a timeout. I can ping the mon from the client on both IPs, public (192.168.41.12) and cluster (192.168.30.12) and I can also see packets coming in on the mon using tcpdump. What could be wrong here? I'm using fuse-cephfs. One more question regarding rebuilding the cluster using cephadm, is there a simple tear-down command? My bootstrap host is a VM so I can use snapshots, but the nodes I have to clean manually, by removing all pods and ceph directories. Best Regards, Oliver

3 years, 5 months

1
0
0 0

File read are not completing and IO shows in bytes able to not reading from cephfs

by Amudhan P

Hi, In my test ceph octopus cluster I was trying to simulate a failure case of when client mounted cephfs thru kernel client and doing read and write process, shutting down entire cluster with OSD flags like no down, no out, no backfiling and no recovery. Cluster is 4 node composed of 3 mons, 2 mgr, 2 mds, 48 OSD's. Public IP range : 10.0.103.0 and Cluster IP range : 10.0.104.0 Write and Read got stalled after some time cluster was brought live and healthy. But when reading file thru kernel mount read start at above 100MB/s and suddenly drops to byte and continues for long. only error msg I could see in the client machine. [ 167.591095] ceph: loaded (mds proto 32) [ 167.600010] libceph: mon0 10.0.103.1:6789 session established [ 167.601167] libceph: client144519 fsid f8bc7682-0d11-11eb-a332-0cc47a5ec98a [ 272.132787] libceph: osd1 10.0.104.1:6891 socket closed (con state CONNECTING) What went wrong why is this issue.? regards Amudhan P

3 years, 5 months

1
1
0 0

Ceph flash deployment

by Seena Fallah

Hi all, Does this guid is still valid for a bluestore deployment with nautilus or octopus? https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments Thanks.

3 years, 5 months

3
5
0 0

RBD image stuck and no erros on logs

by Salsa

Hi, This same error keeps happening to me: after writing some amount of data to an RBD image it gets stuck and no read or write operation on it works. Every operation hangs. I cannot resize, alter features, read or write data. I can mount it, but using parted or fdisk hangs indefinitely. In the end all I can do is remove the image. Again, I see no errors on the logs and Ceph's status is OK. I tried to alter some log levels, but still no helpful info. Is there anything I should check? Rados? -- Salsa Sent with ProtonMail Secure Email.

3 years, 5 months

1
0
0 0

Re: Cephadm: module not found

by Nadiia Kotelnikova

Hi, i am experience the same problem. Could you please advise something how to resolve this issue? The fix should be shipped with 15.2.6 version of "ceph-common" or ceph version? I have my cluster in docker containers and systemd services. How can I upgrade cluster to 15.2.6 if the command for upgrading fails? sudo ceph orch upgrade start --ceph-version 15.2.5 Error ENOENT: Module not found

3 years, 5 months

2
5
0 0

bluefs_buffered_io

by Marcel Kuiper

Hi list, I see a few changes in the (minor) version changelogs in the default for bluefs_buffered_io setting. Sometimes it is set to true in our version (14.2.11) it is set to false Can someone shed a light on this setting? I fail to find any documentation on it. ceph config help is not entirely clear to me as well - What does it do exactly when true - If false does that mean that the linux buffer cache is always skipped? And caching happens in the osd proces only? - if enabled should we lower the osd_memory_target to leave more space for the linux buffer cache? What would be the percentage of memory that we then assign to osd_memory_targets Marcel

3 years, 5 months

1
0
0 0

Ceph 14.2 - some PGs stuck peering.

by m.sliwinski＠lh.pl

Hi We have a weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hosts assigned to nvme pool restarted triggering recovery in that pool. It wnet fast and cluster went to OK state. During these events or shortly after them ssd pool became unresponsive. It was impossible to either read or write from/to it. We decided to slowly restart fist OSDs assigned to it, thenm as it didn't help - all the mons, wihout breaking quorum of course. At this moment both nvme and hdd polls are working fine, ssd one is stuck in recovery. All OSDs in that ssd pool use large amount of CPU and are exchanging approx 1Mpps per OSD server between each other. PGs seem to be slowly migrating from peering to activating but it's going very slowly - approx 10PGs during last hour. We were using 14.2.2 OSDs when issues happened, upgrade to 14.2.13 didn't help. We increased heartbeat grace, but it didn't change anything. It doesn't seem that there's a network problem as OSDs don't report problems with connecting to MONs or each other. Other OSDs - nvme, connected to that same set of switches work without issues. Can you help? Point me to what should i check or do? I looked on-line and on the group for causes of peering issues and checked most of them, nothing helped. I can't use 'ceph pg 28.1cc query' as it hangs, even for PGs that are marked as active+clean in the results of 'ceph pg dump' I checked status of the one of stuck PGs via ceph-objectstore-tool --data-path [...] --op info --pgid 28.29d for all three copies and got: { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69447, "same_interval_since": 69447, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378496", "reported_epoch": "69447", "state": "peering", "last_fresh": "2020-11-03 20:55:39.247348", "last_change": "2020-11-03 20:55:39.247348", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 20:55:39.247348", "last_undegraded": "2020-11-03 20:55:39.247348", "last_fullsized": "2020-11-03 20:55:39.247348", "mapping_epoch": 69447, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271, 284 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69422, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69630, "same_interval_since": 69630, "same_primary_since": 69628, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378445", "reported_epoch": "69627", "state": "peering", "last_fresh": "2020-11-03 21:15:08.819278", "last_change": "2020-11-03 21:14:18.360957", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 21:15:08.819278", "last_undegraded": "2020-11-03 21:15:08.819278", "last_fullsized": "2020-11-03 21:15:08.819278", "mapping_epoch": 69630, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284 ], "acting": [ 261, 284 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69392, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69411, "same_interval_since": 69411, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68070'205093", "reported_seq": "378344", "reported_epoch": "68160", "state": "active+clean", "last_fresh": "2020-11-03 14:45:02.988293", "last_change": "2020-11-03 03:32:46.896044", "last_active": "2020-11-03 14:45:02.988293", "last_peered": "2020-11-03 14:45:02.988293", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-09-01 13:52:40.091759", "last_unstale": "2020-11-03 14:45:02.988293", "last_undegraded": "2020-11-03 14:45:02.988293", "last_fullsized": "2020-11-03 14:45:02.988293", "mapping_epoch": 69411, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3093, "ondisk_log_size": 3093, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 67746, "hit_set_history": { "current_last_update": "0'0", "history": [] } } Current status of the cluster: Reduced data availability: 1021 pgs inactive, 999 pgs peering Degraded data redundancy: 18357/94939584 objects degraded (0.019%), 3 pgs degraded, 5 pgs undersized services: mon: 3 daemons, quorum monb01,monb02,monb03 mgr: monb03(active), standbys: monb01, monb02 osd: 285 osds: 284 up, 284 in data: pools: 9 pools, 9546 pgs objects: 31.65 M objects, 120 TiB usage: 363 TiB used, 127 TiB / 490 TiB avail pgs: 10.696% pgs not active 18357/94939584 objects degraded (0.019%) 8520 active+clean 999 peering 18 activating 3 active+clean+scrubbing+deep 2 activating+undersized+degraded 2 activating+undersized 1 active+clean+scrubbing 1 active+undersized+degraded io: client: 367 MiB/s rd, 195 MiB/s wr, 24.51 kop/s rd, 5.95 kop/s wr cache: 24 MiB/s flush, 90 MiB/s evict, 23 op/s promote

3 years, 5 months

2
2
0 0

Re: Seriously degraded performance after update to Octopus

by Martin Rasmus Lundquist Hansen

Thank you for the suggestion. It does indeed seem to explain why the OSD nodes are no longer using the Buffers for caching. Unfortunately, changing the value bluefs_buffered_io does not seem to make any difference in performance. I will keep looking for clues.

3 years, 5 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2020