- ceph-users - lists.ceph.io

issue with monitors
by techno10＠verizon.net 28 Aug '20

28 Aug '20

i'm running the following [root@node1 ~]# ceph --versionceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on fedora 32.. installed from the built-in repos. I'm running into a simple issue that's rather frustrating. Here is a set of commands i'm running and output: [root@node1 ~]# ceph orch daemon add mon node3:[v2:172.16.0.47:3000,v1:172.16.0.47:6789]Deployed mon.node3 on host 'node3' [root@node1 ~]# ceph mon dumpdumped monmap epoch 4epoch 4fsid c43406b4-e8c2-11ea-b934-001b21d6d88clast_changed 2020-08-28T00:13:27.040659+0000created 2020-08-28T00:10:04.265004+0000min_mon_release 15 (octopus)0: [v2:172.16.0.45:3300/0,v1:172.16.0.45:6789/0] mon.node11: [v2:172.16.0.46:3300/0,v1:172.16.0.46:6789/0] mon.node2 it said it added it but it fails to show up? I attempt to re-add: [root@node1 ~]# ceph orch daemon add mon node3:[v2:172.16.0.47:3000,v1:172.16.0.47:6789]Error ENOENT: ('name %s already in use', 'node3') but again, it's not there, checking with a different command: [root@node1 ~]# ceph mon state4: 2 mons at {node1=[v2:172.16.0.45:3300/0,v1:172.16.0.45:6789/0],node2=[v2:172.16.0.46:3300/0,v1:172.16.0.46:6789/0]}, election epoch 16, leader 0 node1, quorum 0,1 node1,node2 this is a very basic function that I think I'm doing correctly but it seems like it's not working properly?? thanks,Mike

1 0

Re: [cephadm] Deploy Ceph in a closed environment
by Tony Liu 28 Aug '20

28 Aug '20

Please discard this question, I figure it out. Tony > -----Original Message----- > From: Tony Liu <tonyliu0592(a)hotmail.com> > Sent: Thursday, August 27, 2020 1:55 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] [cephadm] Deploy Ceph in a closed environment > > Hi, > > I'd like to deploy Ceph in a closed environment (no connectivity to > public). I will build repository and registry to hold required packages > and container images. How do I specify the private registry when running > "cephadm bootstrap"? The same question for adding OSD. > > Thanks! > Tony > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

[cephadm] Deploy Ceph in a closed environment
by Tony Liu 28 Aug '20

28 Aug '20

Hi, I'd like to deploy Ceph in a closed environment (no connectivity to public). I will build repository and registry to hold required packages and container images. How do I specify the private registry when running "cephadm bootstrap"? The same question for adding OSD. Thanks! Tony

1 0

ceph auth ls
by Marc Roos 28 Aug '20

28 Aug '20

Am I the only one that thinks it is not necessary to dump these keys with every command (ls and get)? Either remove these keys from auth ls and auth get. Or remove the commands "auth print_key" "auth print-key" and "auth get-key"

1 1

Is it possible to mount a cephfs within a container?
by Marc Roos 28 Aug '20

28 Aug '20

I am getting this, on a osd node I am able to mount the path. adding ceph secret key to kernel failed: Operation not permitted

1 0

Fwd: Upgrade Path Advice Nautilus (CentOS 7) -> Octopus (new OS)
by Cloud Guy 27 Aug '20

27 Aug '20

Hello, Looking for a bit of guidance / approach to upgrading from Nautilus to Octopus considering CentOS and Ceph-Ansible. We're presently running a Nautilus cluster (all nodes / daemons 14.2.11 as of this post). - There are 4 monitor-hosts with mon, mgr, and dashboard functions consolidated; - 4 RGW hosts - 4 ODS costs, with 10 OSDs each. This is planned to scale to 7 nodes with additional OSDs and capacity (considering to do this as part of upgrade process) - Currently using ceph-ansible (however it's a process to maintain scripts / configs between playbook versions - although a great framework, not ideal in our case; - All hosts run CentOS 7.x; - dm-crypt in use on LVM OSDs (via ceph-ansible); - Deployment IS NOT containerized. Octopus support on CentOS 7 is limited due to python dependencies, as a result we want to move to CentOS 8 or Ubuntu 20.04. The other outlier is CentOS native Kernel support for LSI2008 (eg. 9211) HBAs which some of our OSD nodes use. Irrespective of OS considerations above, the upgrade will be to an OS that fully supports Octopus. We'd like to make use of ceph orchestrator for on-going cluster management. Here's an upgrade path scenario that is being considered. At a high-level: 1. Deploy a new monitor on CentOS 8. May be Nautilus via established ceph-ansible playbook. 2. Upgrade new monitor to Octopus using dnf / ceph package upgrade. 3. Decommission individual monitor hosts (existing on CentOS 7) and redeploy on CentOS 8 via ceph orchestrator from new monitor node; 4. Repeat until all monitors are on new OS + Octopus (all deployed via Ceph Orchestrator. 5. Add additional OSD nodes / drives / capacity via orchestrator on Octopus; 6. Upgrade existing OSD hosts by keeping OSDs intact, reinstalling new OS (CentOS 8 or Ubuntu 20.04); 7. Deploy ceph octopus on new nodes via orchestrator; 8. Reactivate / rescan in-tact OSDs on newly redeployed node. (i.e. ceph-volume lvm activate --all) 9. Rinse / repeat for remaining Nautilus nodes. 10. Manually upgrade RGW packages on gateway nodes. Thank you.

2 2

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Zhenshi Zhou 27 Aug '20

27 Aug '20

Official document says that you should allocate 4% of the slow device space for block.db. But the main problem is that Bluestore uses RocksDB and RocksDB puts a file on the fast device only if it thinks that the whole layer will fit there. As for RocksDB, L1 is about 300M, L2 is about 3G, L3 is near 30G, and L4 is about 300G. For instance, RocksDB puts L2 files to block.db only if it’s at least 3G there. As a result, 30G is a acceptable value. Tony Liu <tonyliu0592(a)hotmail.com> 于2020年8月25日周二上午10:49写道： > > -----Original Message----- > > From: Anthony D'Atri <anthony.datri(a)gmail.com> > > Sent: Monday, August 24, 2020 7:30 PM > > To: Tony Liu <tonyliu0592(a)hotmail.com> > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL and DB on > > SSD > > > > Why such small HDDs? Kinda not worth the drive bays and power, instead > > of the complexity of putting WAL+DB on a shared SSD, might you have been > > able to just buy SSDs and not split? ymmv. > > 2TB is for testing, it will bump up to 10TB for production. > > > The limit is a function of the way the DB levels work, it’s not > > intentional. > > > > WAL by default takes a fixed size, like 512 MB or something. > > > > 64 GB is a reasonable size, it accomodates the WAL and allows space for > > DB compaction without overflowing. > > For each 10TB HDD, what's the recommended DB device size for both > DB and WAL? The doc recommends 1% - 4%, meaning 100GB - 400GB for > each 10TB HDD. But given the WAL data size and DB data size, I am > not sure if that 100GB - 400GB will be used efficiently. > > > With this commit the situation should be improved, though you don’t > > mention what release you’re running > > > > https://github.com/ceph/ceph/pull/29687 > > I am using ceph version 15.2.4 octopus (stable). > > Thanks! > Tony > > > >>> I don't need to create > > >>> WAL device, just primary on HDD and DB on SSD, and WAL will be using > > >>> DB device cause it's faster. Is that correct? > > >> > > >> Yes. > > >> > > >> > > >> But be aware that the DB sizes are limited to 3GB, 30GB and 300GB. > > >> Anything less than those sizes will have a lot of untilised space, > > >> e.g a 20GB device will only utilise 3GB. > > > > > > I have 1 480GB SSD and 7 2TB HDDs. 7 LVs are created on SSD, each is > > > about 64GB, for 7 OSDs. > > > > > > Since it's shared by DB and WAL, DB will take 30GB and WAL will take > > > the rest 34GB. Is that correct? > > > > > > Is that size of DB and WAL good for 2TB HDD (block store and object > > > store cases)? > > > > > > Could you share a bit more about the intention of such limit? > > > > > > > > > Thanks! > > > Tony > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Ceph Tech Talk: Secure Token Service in the Rados Gateway
by Mike Perez 27 Aug '20

27 Aug '20

Hi everyone, Join us August 27th at 17:00 UTC to hear Pritha Srivastava present on this month's Ceph Tech Talk: Secure Token Service in the Rados Gateway. Calendar invite and archive can be found here: https://ceph.io/ceph-tech-talks/ If you're interested or know someone who can present September 24th, or October 22nd please let me know! -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 1

[Ceph Octopus 15.2.3 ] MDS crashed suddenly
by carlimeunier＠gmail.com 27 Aug '20

27 Aug '20

Hi, I made a fresh install of Ceph Octopus 15.2.3 recently. And after a few days, the 2 standby MDS suddenly crashed with segmentation fault error. I try to restart it but it does not start. Here is the error : -20> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _renew_subs -19> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _send_mon_message to mon.2 at v1:172.31.36.98:6789/0 -18> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcf9530c0 version 269 -17> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa87520 version 269 -16> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa875c0 version 269 -15> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa871c0 version 269 -14> 2020-07-17T13:50:27.888+0000 7fc8c8c55700 10 monclient: get_auth_request con 0x559dcfada000 auth_method 0 -13> 2020-07-17T13:50:27.888+0000 7fc8c9456700 10 monclient: get_auth_request con 0x559dcfada800 auth_method 0 -12> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) recover start -11> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) read_head -10> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 4 mds.0.log Waiting for journal 0x200 to recover... -9> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_read_head loghead(trim 4194304, expire 4231216, write 4329405, stream_format 1). probing for end of log (from 4329405)... -8> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) probing for end of the log -7> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_probe_end write_pos = 4329949 (header had 4329405). recovered. -6> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Journal 0x200 recovered. -5> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Recovered journal 0x200 in format 1 -4> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 2 mds.0.0 Booting: 1: loading/discovering base inodes -3> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x100 -2> 2020-07-17T13:50:27.894+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x1 -1> 2020-07-17T13:50:27.894+0000 7fc8c0444700 2 mds.0.0 Booting: 2: replaying mds log 0> 2020-07-17T13:50:27.896+0000 7fc8bec41700 -1 *** Caught signal (Segmentation fault) ** in thread 7fc8bec41700 thread_name:md_log_replay Here is the cluster information : # ceph status cluster: id: dd024fe1-4996-4fed-ba57-03090e53724d health: HEALTH_WARN 20 daemons have recently crashed services: mon: 3 daemons, quorum 2,0,1 (age 2d) mgr: mgr.0(active, since 9d), standbys: mgr.2, mgr.1 mds: cephfs:1 {0=node0=up:active} 1 up:standby-replay 1 up:standby osd: 3 osds: 3 up (since 28h), 3 in (since 9d) task status: scrub status: mds.node0: idle mds.node2: idle data: pools: 3 pools, 49 pgs objects: 29 objects, 170 KiB usage: 3.0 GiB used, 41 TiB / 41 TiB avail pgs: 49 active+clean io: client: 853 B/s rd, 1 op/s rd, 0 op/s wr There is only 1 client connected to the cluster. Please, does anyone have any idea? Thanks

3 4

rados df with nautilus / bluestore
by Manuel Lausch 27 Aug '20

27 Aug '20

Hi we found a very ugly issue in rados df I have several clusters, all running ceph nautilus (14.2.11), We have there replicated pools with replica size 4. On the older clusters "rados df" shows in the used column the net used space. On our new cluster, rados df shows in the used column the gross used space. The older clusters was upgraded from luminous (and before) and uses filestore and the new cluster is initally deployed with nautilus and bluestore. why are the outputs different? Is this related with nautilus or with bluestore? For our reporting this values have a significant relevance and now I am running in such discrepancies. What commands/metrics can I use to get more reliable values. Maybe "ceph df detail"? Manuel

2 1