- ceph-users - lists.ceph.io

consistency of import-diff
by Stefan Priebe - Profihost AG 05 Mar '20

05 Mar '20

Hello, does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff? While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days. Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards. Greets, Stefan

5 10

Re: High memory ceph mgr 14.2.7
by hoannv46＠gmail.com 05 Mar '20

05 Mar '20

I disabled some module in mgr : influx, dashboard, prometheus. After i restart mgr, ram of mgr increase to 20GB in some seconds.

1 0

MDS getting stuck on 'resolve' and 'rejoin'
by Anastasia Belyaeva 04 Mar '20

04 Mar '20

Hello! Our CephFS mds cluster consists of 3 ranks. We had a minor issue with the network the ceph runs on. And after that cephfs became unavaialble: rank 1 and 2 stuck in rejoin rank 0 can't get pass 'resolve' state and keeps getting blacklisted I checked the logs (with debug_mds 5/5) on the rank 0 mds server and found out the following: goes through 'replay' fine then 'resolve' starts and the log gets flooded with messages like -18> 2020-03-04 16:59:56.934 7f77445f0700 5 mds.0.log _submit_thread 442443596224~41 : EImportFinish 0x30000412462 failed -17> 2020-03-04 16:59:56.950 7f77445f0700 5 mds.0.log _submit_thread 442443596285~41 : EImportFinish 0x3000041246c failed -16> 2020-03-04 16:59:56.966 7f77445f0700 5 mds.0.log _submit_thread 442443596346~41 : EImportFinish 0x3000041247b failed -15> 2020-03-04 16:59:56.983 7f77445f0700 5 mds.0.log _submit_thread 442443596407~41 : EImportFinish 0x30000412485 failed then messages about heartbeat errors start showing up (in between) -3210> 2020-03-04 16:59:04.079 7f77485f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -3209> 2020-03-04 16:59:04.079 7f77485f8700 0 mds.beacon.ceph-server11.ibnet Skipping beacon heartbeat to monitors (last acked 8.00204s ago); MDS internal heartbeat is not healthy! the 'flood' ends with messages -14> 2020-03-04 16:59:57.001 7f77455f2700 -1 mds.0.journaler.mdlog(rw) _finish_write_head got (108) Cannot send after transport endpoint shutdown -13> 2020-03-04 16:59:57.001 7f77455f2700 -1 mds.0.journaler.mdlog(rw) handle_write_error (108) Cannot send after transport endpoint shutdown after wich the mds-server gets blacklisted and becomes a 'standby'. And then the same scenario happens to the standby. Also in my attempt to recover the fs I managed to make it worse. I executed cephfs-table-tool 0 reset session and now the mds daemon crashes at 'replay' with the following error -2> 2020-03-04 22:00:54.228 7f4ca6e44700 -1 log_channel(cluster) log [ERR] : error replaying open sessions(1) sessionmap v 7348424 table 0 -1> 2020-03-04 22:00:54.229 7f4ca6e44700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/src/mds/SessionMap.cc: In function 'void SessionMap::replay_open_sessions(version_t, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&)' thread 7f4ca6e44700 time 2020-03-04 22:00:54.229427 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/src/mds/SessionMap.cc: 750: FAILED ceph_assert(g_conf()->mds_wipe_sessions) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f4cb7913ac2] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4cb7913c90] 3: (()+0x3b618b) [0x55b43860518b] 4: (EImportStart::replay(MDSRank*)+0x4a8) [0x55b4386805f8] 5: (MDLog::_replay_thread()+0x8ee) [0x55b43861b3ae] 6: (MDLog::ReplayThread::entry()+0xd) [0x55b43838fecd] 7: (()+0x7e25) [0x7f4cb57efe25] 8: (clone()+0x6d) [0x7f4cb46b234d] 0> 2020-03-04 22:00:54.230 7f4ca6e44700 -1 *** Caught signal (Aborted) ** in thread 7f4ca6e44700 thread_name:md_log_replay About the setup ceph version 14.2.4 OS Centos 7.4 3.10.0-693.5.2.el7.x86_64 Any help would be greatly appreciated! Best regards, Anastasia Belyaeva С уважением, Анастасия Беляева Best regards, Anastasia Belyaeva С уважением, Анастасия Беляева

1 0

Need clarification on CephFS, EC Pools, and File Layouts
by Dave Hall 04 Mar '20

04 Mar '20

Hello, This is for a cluster currently running at 14.2.7. Since our cluster is still relatively small we feel a strong need to run our CephFS on an EC Pool (8 + 2) and Crush Failure Domain = OSD to maximize capacity. I have read and re-read https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools and https://docs.ceph.com/docs/nautilus/cephfs/file-layouts/#file-layouts, but it still isn't quite clear to me. Since this topic is mentioned in the release notes for 14.2.8 I thought I should probably ask so I can configure this correctly. If using a small replicated pool as the default data pool, how does one use a file layout induce the bulk of the data to be stored in the secondary EC data pool? From the links referenced I infer that a file layout is required. Is it possible to have a file layout based solely on size? BTW, we want to do this in a way that we don't have to think about which directory goes with which file size or anything like that. This needs to be an internal detail that is completely hidden from the client. Also, is it possible to insert a replicated data pool as the default on an already deployed CephFS, or will I need to create a new FS and copy the data over? Thanks. -Dave -- Dave Hall Binghamton University

3 3

How can I fix "object unfound" error?
by Chad William Seys 04 Mar '20

04 Mar '20

Hi Simone, Maybe you've hit this bug: https://tracker.ceph.com/issues/44286 ? Chad.

1 0

Re: High memory ceph mgr 14.2.7
by Mark Lopez 04 Mar '20

04 Mar '20

I noticed the same - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WS6OWKZ5NQX…. Out of curiosity, do you use the dashboard, and if so, does keeping the dashboard open cause the mgr RAM usage to increase overtime? Regards, Mark

1 0

Re: v14.2.8 Nautilus released
by kefu chai 04 Mar '20

04 Mar '20

On Wed, Mar 4, 2020 at 3:46 AM Kaleb Keithley <kkeithle(a)redhat.com> wrote: > > > Just FYI, 14.2.8 build fails on Fedora-32 on S390x. Other architectures build fine. > Kaleb, see https://github.com/ceph/ceph/pull/33716. hopefully we can get in the next release or probably you could include this patch in the rpm packaging? > Build log at https://kojipkgs.fedoraproject.org//work/tasks/6999/42146999/build.log > > On Tue, Mar 3, 2020 at 7:38 AM Abhishek Lekshmanan <abhishek(a)suse.com> wrote: >> >> >> This is the eighth update to the Ceph Nautilus release series. This release >> fixes issues across a range of subsystems. We recommend that all users upgrade >> to this release. Please note the following important changes in this >> release; as always the full changelog is posted at: >> https://ceph.io/releases/v14-2-8-nautilus-released >> >> Notable Changes >> --------------- >> >> * The default value of `bluestore_min_alloc_size_ssd` has been changed >> to 4K to improve performance across all workloads. >> >> * The following OSD memory config options related to bluestore cache autotuning can now >> be configured during runtime: >> >> - osd_memory_base (default: 768 MB) >> - osd_memory_cache_min (default: 128 MB) >> - osd_memory_expected_fragmentation (default: 0.15) >> - osd_memory_target (default: 4 GB) >> >> The above options can be set with:: >> >> ceph config set osd <option> <value> >> >> * The MGR now accepts `profile rbd` and `profile rbd-read-only` user caps. >> These caps can be used to provide users access to MGR-based RBD functionality >> such as `rbd perf image iostat` an `rbd perf image iotop`. >> >> * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap >> balancing has been removed. Instead use the mgr balancer config >> `upmap_max_deviation` which now is an integer number of PGs of deviation >> from the target PGs per OSD. This can be set with a command like >> `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default >> `upmap_max_deviation` is 1. There are situations where crush rules >> would not allow a pool to ever have completely balanced PGs. For example, if >> crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of >> the racks. In those cases, the configuration value can be increased. >> >> * RGW: a mismatch between the bucket notification documentation and the actual >> message format was fixed. This means that any endpoints receiving bucket >> notification, will now receive the same notifications inside a JSON array >> named 'Records'. Note that this does not affect pulling bucket notification >> from a subscription in a 'pubsub' zone, as these are already wrapped inside >> that array. >> >> * CephFS: multiple active MDS forward scrub is now rejected. Scrub currently >> only is permitted on a file system with a single rank. Reduce the ranks to one >> via `ceph fs set <fs_name> max_mds 1`. >> >> * Ceph now refuses to create a file system with a default EC data pool. For >> further explanation, see: >> https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools >> >> * Ceph will now issue a health warning if a RADOS pool has a `pg_num` >> value that is not a power of two. This can be fixed by adjusting >> the pool to a nearby power of two:: >> >> ceph osd pool set <pool-name> pg_num <new-pg-num> >> >> Alternatively, the warning can be silenced with:: >> >> ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false >> >> Getting Ceph >> ------------ >> >> * Git at git://github.com/ceph/ceph.git >> * Tarball at http://download.ceph.com/tarballs/ceph-14.2.8.tar.gz >> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >> * Release git sha1: 2d095e947a02261ce61424021bb43bd3022d35cb >> >> -- >> Abhishek Lekshmanan >> SUSE Software Solutions Germany GmbH >> GF: Felix Imendörffer HRB 21284 (AG Nürnberg) >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io -- Regards Kefu Chai

1 0

MIgration from weight compat to pg_upmap
by Stefan Priebe - Profihost AG 04 Mar '20

04 Mar '20

Hello, is there any way to switch to pg_upmap without triggering heavy rebalancing two times? 1.) happens at: ceph osd crush weight-set rm-compat 2.) happens after running the balancer in pg_upmap mode Greets, Stefan

2 2

Deleting Multiparts stuck directly from rgw.data pool
by EDH - Manuel Rios 04 Mar '20

04 Mar '20

Hi, We're at 14.2.8, still got problems with abort multiparts. This night we created a full list of objects with the string multipart like 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.8__multipart_MBS-a3d91785-9579-4b18-9635-f4694bd0db1c/CBB_SRVALFA/F$/Correo/Gmolins/Outlook/Outlook.pst$/20190702110014/Outlook.pst.2~nu6skEciSGZVSz3SlXZ-GMURki6wPB3.10 Total 5-6Millions parts Its secure delete rgw object directly from data pool? We tried with LC, it return list_multiparts -2, with AWS cli an error already reported in the tracker. I'm thinking about do a full bucket rewrite with radosgw-admin bucket rewrite but im not sure if this will solve something or generate more trash. Best Regards Manuel

1 0

14.2.8 Multipart delete still not working
by EDH - Manuel Rios 03 Mar '20

03 Mar '20

Hi, We have updated our cluster to 14.2.8 since we suffered the bug https://tracker.ceph.com/issues/43583, now life cycle policies give more information than before. In 14.2.7 they ended instantly so something we have advanced. But they are not yet able to eliminate multipart. Just a line of the error that return -2 1. 2020-03-03 18:13:19.361 7fb58bcfb6c0 5 lifecycle: ERROR: abort_multipart_upload failed, ret=-2009, meta:_multipart_MBS-0fc78b70-efa6-49ef-bdd2-fd3a4b4f2c84/CBB_BIM-AUTOLOG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20190315230135/160.cbrevision.2~5nOv_6K_GZVwAJNqmEZ RrmE4lMs_-91.meta 2. 2020-03-03 18:13:19.361 7fb58bcfb6c0 20 obj_has_expired(): mtime=2019-03-16 00:53:58.0.940346s days=1 base_time=2020-03-03 00:00:00.000000 timediff=3.0496e+07 cmp=86400 3. 2020-03-03 18:13:19.362 7fb58bcfb6c0 20 abort_multipart_upload: list_multipart_parts returned -2 LC Applied <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Incomplete Multipart Uploads</ID> <Prefix/> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> Suggestion ? Regards Manuel

1 0