- ceph-users - lists.ceph.io

Configuration of Ceph-ISCSI gateway
by Gesiel Galvão Bernardes 24 Sep '19

24 Sep '19

Hi everyone, I'm configurating ISCSI gateway in Ceph Mimic (13.2.6) using ceph manual: https://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/ But i stopped in this problem: In manual says: "Set the client’s CHAP username to myiscsiusername and password to myiscsipassword: > /iscsi-target...at:rh7-client> auth chap=myiscsiusername/myiscsipassword" But I receive this response: /iscsi-target...at:rh7-client> auth chap=myiscsitest/myiscsitestpasswd Unexpected keyword parameter 'chap'. The options disponibles are: /iscsi-target...at:rh7-client> auth ? To set authentication, specify username=<user> password=<password> [mutual_username]=<user> [mutual_password]=<password> But if configure as asks: auth username=myiscsitest password=myiscsitestpasswd Failed to update the client's auth: Invalid password I tried with high password complexibility, but the problem persists. My questions: - How is the correct mode for configure authentication? - How contribute for update of documentation? A bug report has opened* for broken information of instalation of ceph-iscsi-gw, but was closed without update of documentation:https://github.com/ceph/ceph-ansible/issues/2707 Regards Gesiel Bernardeds

1 0

rados bench performance in nautilus
by 徐蕴 24 Sep '19

24 Sep '19

Hi ceph experts, I deployed Nautilus (v14.2.4) and Luminous (v12.2.11) on the same hardware, and made a rough performance comparison. The result seems Luminous is much better, which is unexpected. My setup: 3 servers, each has 3 HDD OSDs, 1 SSD as DB, two separated 1G network for cluster and public. Pool test has 32 pg and pop numbers, replicated size is 3. Using "rados -p bench 80 write” to measure write performance. The result: Luminous: Average IOPS 36 Nautilus: Average IOPS 28 Is the difference considered valid for Nautilus? Br, Xu Yun

3 6

Re: unsubscribe
by Wesley Peng 24 Sep '19

24 Sep '19

As the signature shows, please send an email to ceph-users-leave(a)ceph.io for unsubscribing. hou guanghua wrote: > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Creating a monmap with V1 & V2 using monmaptool
by Ricardo Dias 24 Sep '19

24 Sep '19

Hi Alberto, Did you tried with the "--addv" option? Here's an example: monmaptool --addv <mon.id> [v2:<ip_address>:<v2_port>,v1:<ip_address>:<v1_port>] Cheers, Ricardo Dias On 23/09/19 16:07, Corona, Alberto wrote: > Hi folks, > > While practicing some disaster recovery I noticed that it currently seems impossible to add both a v1 and v2 monitor to a monmap using monmaptool. Is there any way to create a monmap manually to include both protocol versions to a monmap? Currently on Ceph version 14.2.4 > > > - Alberto Corona > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0

unsubscribe
by hou guanghua 24 Sep '19

24 Sep '19

1 0

unsubscribe
by hou guanghua 24 Sep '19

24 Sep '19

1 0

cache tiering or bluestore partitions
by Shawn A Kwang 23 Sep '19

23 Sep '19

We are planning our ceph architecture and I have a question: How should NVMe drives be used when our spinning storage devices use bluestore: 1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-re…) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else? Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total) I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware. Use Case: The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes. Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering? If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance. Sincerely, Shawn Kwang

2 3

Re: Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Sage Weil 23 Sep '19

23 Sep '19

On Mon, 23 Sep 2019, Koebbe, Brian wrote: > Thanks Sage! > ceph osd dump: https://pastebin.com/raw/zLPz9DQg > > > ceph-monstore-tool /var/lib/ceph/mon/ceph-ufm03 dump-keys |grep osd_snap| cut -c-29 |uniq -c > 2 osd_snap / purged_snap_10_000 > 1 osd_snap / purged_snap_12_000 > 75 osd_snap / purged_snap_13_000 > 4 osd_snap / purged_snap_14_000 > 778106 osd_snap / purged_snap_15_000 > 861 osd_snap / purged_snap_1_0000 > 323 osd_snap / purged_snap_4_0000 > 88 osd_snap / purged_snap_5_0000 > 2 osd_snap / purged_snap_7_0000 > 2 osd_snap / purged_snap_8_0000 > 2 osd_snap / removed_epoch_10_0 > 1 osd_snap / removed_epoch_12_0 > 75 osd_snap / removed_epoch_13_0 > 4 osd_snap / removed_epoch_14_0 > 2316417 osd_snap / removed_epoch_15_0 > 970 osd_snap / removed_epoch_1_00 > 324 osd_snap / removed_epoch_4_00 > 89 osd_snap / removed_epoch_5_00 > 2 osd_snap / removed_epoch_7_00 > 2 osd_snap / removed_epoch_8_00 > 2 osd_snap / removed_snap_10_00 > 1 osd_snap / removed_snap_12_00 > 75 osd_snap / removed_snap_13_00 > 4 osd_snap / removed_snap_14_00 > 2720849 osd_snap / removed_snap_15_00 > 1161 osd_snap / removed_snap_1_000 > 379 osd_snap / removed_snap_4_000 > 89 osd_snap / removed_snap_5_000 > 2 osd_snap / removed_snap_7_000 > 2 osd_snap / removed_snap_8_000 Thanks! I opened a ticket at https://tracker.ceph.com/issues/42012. Can you do the aboe dump with the 'grep osd_snap' only and attach that to the bug? This code has reworekd/improved in master, but I'm not sure the behavior of keeping a full record of past span deletions was changed, so we may need to make further improvements for octopus. Thanks! sage > ________________________________ > From: Sage Weil <sage(a)newdream.net> > Sent: Monday, September 23, 2019 9:41 AM > To: Koebbe, Brian <koebbe(a)wustl.edu> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io>; dev(a)ceph.io <dev(a)ceph.io> > Subject: Re: [ceph-users] Seemingly unbounded osd_snap keys in monstore. Normal? Expected? > > Hi, > > On Mon, 23 Sep 2019, Koebbe, Brian wrote: > > Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. > > A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. > > While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: > > > > ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c > > 153 auth > > 2 config > > 10 health > > 1441 logm > > 3 mdsmap > > 313 mgr > > 1 mgr_command_descs > > 3 mgr_metadata > > 163 mgrstat > > 1 mkfs > > 323 mon_config_key > > 1 mon_sync > > 6 monitor > > 1 monitor_store > > 32 monmap > > 120 osd_metadata > > 1 osd_pg_creating > > 5818618 osd_snap > > 41338 osdmap > > 754 paxos > > > > A few questions: > > > > Could this be the cause of the slow addition/synchronization? > > Probably! > > > Is what looks like an unbounded number of osd_snaps expected? > > Maybe. Can you send me the output of 'ceph osd dump'? Also, if you don't > mind doing the dump above and grepping out just the osd_snap keys, so I > can see what they look like and if they match the osd map contents? > > Thanks! > sage > > > > If trimming/compacting them would help, how would one do that? > > > > Thanks, > > Brian > > > > ________________________________ > > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. >

2 1

Re: Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Sage Weil 23 Sep '19

23 Sep '19

Hi, On Mon, 23 Sep 2019, Koebbe, Brian wrote: > Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. > A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. > While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: > > ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c > 153 auth > 2 config > 10 health > 1441 logm > 3 mdsmap > 313 mgr > 1 mgr_command_descs > 3 mgr_metadata > 163 mgrstat > 1 mkfs > 323 mon_config_key > 1 mon_sync > 6 monitor > 1 monitor_store > 32 monmap > 120 osd_metadata > 1 osd_pg_creating > 5818618 osd_snap > 41338 osdmap > 754 paxos > > A few questions: > > Could this be the cause of the slow addition/synchronization? Probably! > Is what looks like an unbounded number of osd_snaps expected? Maybe. Can you send me the output of 'ceph osd dump'? Also, if you don't mind doing the dump above and grepping out just the osd_snap keys, so I can see what they look like and if they match the osd map contents? Thanks! sage > If trimming/compacting them would help, how would one do that? > > Thanks, > Brian > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. >

2 1

Activate Cache Tier on Running Pools
by Eikermann, Robert 23 Sep '19

23 Sep '19

Hi, I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert

6 9