- ceph-users - lists.ceph.io

ceph osd pool ls detail 'removed_snaps' on empty pool?
by Marc Roos 16 Dec '19

16 Dec '19

I have removed_snaps listed on pools that I am not using. They are mostly for doing some performance testing, so I cannot imagine ever creating snapshots in them. pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs removed_snaps [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, [@ ]# rados df| egrep '^POOL|fs_data.ssd' POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR fs_data.ssd 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

1 1

v13.2.7 mimic released
by Sage Weil 16 Dec '19

16 Dec '19

This is the seventh bugfix release of the Mimic v13.2.x long term stable release series. We recommend all Mimic users upgrade. For the full release notes, see https://ceph.io/releases/v13-2-7-mimic-released/ Notable Changes MDS: - Cache trimming is now throttled. Dropping the MDS cache via the “ceph tell mds.<foo> cache drop” command or large reductions in the cache size will no longer cause service unavailability. - Behavior with recalling caps has been significantly improved to not attempt recalling too many caps at once, leading to instability. MDS with a large cache (64GB+) should be more stable. - MDS now provides a config option “mds_max_caps_per_client” (default: 1M) to limit the number of caps a client session may hold. Long running client sessions with a large number of caps have been a source of instability in the MDS when all of these caps need to be processed during certain session events. It is recommended to not unnecessarily increase this value. - The “mds_recall_state_timeout” config parameter has been removed. Late client recall warnings are now generated based on the number of caps the MDS has recalled which have not been released. The new config parameters “mds_recall_warning_threshold” (default: 32K) and “mds_recall_warning_decay_rate” (default: 60s) set the threshold for this warning. - The “cache drop” admin socket command has been removed. The “ceph tell mds.X cache drop” remains. OSD: - A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option “mon_warn_on_slow_ping_ratio” specifies a percentage of “osd_heartbeat_grace” to determine the threshold. A value of zero disables the warning. A new configuration option “mon_warn_on_slow_ping_time”, specified in milliseconds, overrides the computed value, causing a warning when OSD heartbeat pings take longer than the specified amount. A new admin command “ceph daemon mgr.# dump_osd_network [threshold]” lists all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. A new admin command ceph daemon osd.# dump_osd_network [threshold]” does the same but only including heartbeats initiated by the specified OSD. - The default value of the “osd_deep_scrub_large_omap_object_key_threshold” parameter has been lowered to detect an object with large number of omap keys more easily. RGW: - radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details.

4 7

any way to read magic number like #1018a1b3c14?
by Sang, Oliver 16 Dec '19

16 Dec '19

we are using cephfs. in /sys/kernel/debug/ceph/*/mdsc sometimes we saw something like - 2015909 mds0 setfilelock #1018a1b3c14 I know this #1018a1b3c14 is a file or a folder, but how to know what it really is in cephfs tree? Thanks!

1 0

Re: ceph-volume sizing osds
by Jan Fajerski 14 Dec '19

14 Dec '19

On Fri, Dec 13, 2019 at 11:49:50PM +0100, Oscar Segarra wrote: > Hi, > I have recently started working with Ceph Nautilus release and I have > realized that you have to start working with LVM to create OSD instead > of the "old fashioned" ceph-disk. > In terms of performance and best practices, as far as I must use LVM I > can create volume groups that joins or extends two or more physical > disks. In this scenario (many disks for server) where ceph-volume is > manatory, It still remains the rule of one OSD for each physical > device? or I can reduce the number of OSDs? That is hard to answer generically. ceph-volume will not create multi-disk volumes. One disk, one OSD is still the recommendation afaik. However there are certainly scenarios where multi-disk OSDs are sensible. For example if you have a machine that has _a lot_ of disks attached but lacks CPU and RAM to run as many OSDs as disks, it can make sense to deploy one OSD on multiple disks. > Thanks in advance, > Óscar >_______________________________________________ >ceph-users mailing list >ceph-users(a)lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Re: Can't create new OSD
by Gregory Farnum 13 Dec '19

13 Dec '19

[ Re-adding users list ] On Thu, Dec 12, 2019 at 10:12 AM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > > Em qui., 12 de dez. de 2019 às 12:55, Gregory Farnum > <gfarnum(a)redhat.com> escreveu: > > > > On Wed, Dec 11, 2019 at 12:04 PM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > >> > >> Hi, > >> > >> > >> Trying to create a new OSD following the instructions available at > >> https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/ > >> > >> On step 3 I'm instructed to run "ceph-osd -i {osd-num} --mkfs > >> --mkkey". Unfortunately it doesn't work: > >> > >> # ceph-osd -i 3 --mkfs --mkkey > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 AuthRegistry(0x55ad976ea140) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 AuthRegistry(0x7fffac4075e8) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> failed to fetch mon config (--no-mon-config to skip) > > I'm full of questions. > > > This is the important bit. Since you’re running the command without a way to access the Ceph monitors, > > Should I provide a way do access the Ceph monitors? I was just > following the docs and there is no mention to it in the above page. > > Anyway I tried to do it but the results were exactly the same: > > # ceph-osd -m 192.168.109.233:3300 -i 3 --mkfs --mkkey > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x564545484140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x7ffccdef0958) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > I also tried in the 6789 port and without providing port at all. > Always the same results. Tried also on the other monitors. Always the > same results. > > > it can’t use the cluster default configuration. You need to move the client.admin keyring onto the node (or another one, if you configure the ceph.conf or pass in the name on each command invocation). I haven't been through the docs in a while so maybe it's missing or lost but it should describe that somewhere. This is necessary whenever you are making changes to the cluster, such as adding an OSD — without a keyring, the cluster has no idea if the user making the change is authorized! > > Shouldn't it automatically read /etc/ceph/ceph.conf and figure everything out? > > I also tried explicitly setting the conf path but, again, same results: It's reading the ceph.conf but still needs cluster access, which you haven't given it. > > # ceph-osd -i 3 --mkfs --mkkey -c /etc/ceph/ceph.conf > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x56457c784140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x7ffc6eccb0b8) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > > You can pass the given option to not worry about that, > > Tried that. Apparently it worked, despite several disturbing messages: > > # ceph-osd -i 3 --mkfs --mkkey --no-mon-config > 2019-12-12 14:57:50.256 7f31d77b2c00 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-3/keyring: can't open > /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 created new key in keyring > /var/lib/ceph/osd/ceph-3/keyring > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.629 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid > # ll /var/lib/ceph/osd/ceph-3 > total 172 > drwxr-xr-x 2 root root 4096 Dec 12 14:57 ./ > drwxr-xr-x 4 ceph ceph 4096 Dec 11 16:19 ../ > -rw-r----- 1 root root 10737418240 Dec 12 14:57 block > -rw------- 1 root root 2 Dec 12 14:57 bluefs > -rw------- 1 root root 37 Dec 12 14:57 ceph_fsid > -rw-r----- 1 root root 37 Dec 12 14:57 fsid > -rw------- 1 root root 56 Dec 12 14:57 keyring > -rw------- 1 root root 8 Dec 12 14:57 kv_backend > -rw------- 1 root root 21 Dec 12 14:57 magic > -rw------- 1 root root 4 Dec 12 14:57 mkfs_done > -rw------- 1 root root 6 Dec 12 14:57 ready > -rw------- 1 root root 10 Dec 12 14:57 type > -rw------- 1 root root 2 Dec 12 14:57 whoami > > Is it how it should be? With all these "failed to open" messages? Yeah, those are BlueStore et al attempting various opens that are allowed to fail. I agree they're not great to have show up as standard. > > > or you can run the command with a user key ring that’s allowed to read the monitor configs. (You’ll need that key access next to add the OSD’s key to the cluster, anyway!) > > Haven't tried that. How is it done? I can't find any option to provide > a user key ring in ceph-osd's man page. It's not ceph-osd specific, just anywhere in ceph. -n client.[name], -k [keyring_file] etc > > > Many thanks for your help. > > Rodrigo >

1 0

v13.2.8 Mimic released
by Abhishek Lekshmanan 13 Dec '19

13 Dec '19

This is the eighth backport release in the Ceph Mimic stable release series. Its sole purpose is to fix a regression that found its way into the previous release. Notable Changes --------------- * Due to a missed backport, clusters in the process of being upgraded from 13.2.6 to 13.2.7 might suffer an OSD crash in build_incremental_map_msg. This regression was reported in https://tracker.ceph.com/issues/43106 and is fixed in 13.2.8 (this release). Users of 13.2.6 can upgrade to 13.2.8 directly - i.e., skip 13.2.7 - to avoid this. Changelog --------- * osd: fix sending incremental map messages (issue#43106 pr#32000, Sage Weil) * tests: added missing point release versions (pr#32087, Yuri Weinstein) * tests: rgw: add missing force-branch: ceph-mimic for swift tasks (pr#32033, Casey Bodley) For a blog with links to PRs and issues please check out https://ceph.io/releases/v13-2-8-mimic-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.8.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 0

Re: RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 12 Dec '19

12 Dec '19

Phillippe, Maybe you can try the "crush-compat" balancer mode instead of "upmap" until the new code is released. David On 12/11/19 9:36 PM, Philippe D'Anjou wrote: > > Hi, > I see your code balanced my ssdpool to about 146 each. > I can confirm this did NOT happen. > The state it ended up in is: > > 0 ssd 3.49219 1.00000 3.5 TiB 797 GiB 757 GiB 36 GiB 3.9 GiB 2.7 > TiB 22.30 0.31 147 up > 1 ssd 3.49219 1.00000 3.5 TiB 803 GiB 751 GiB 49 GiB 3.7 GiB 2.7 > TiB 22.47 0.31 146 up > 2 ssd 3.49219 1.00000 3.5 TiB 818 GiB 764 GiB 51 GiB 3.8 GiB 2.7 > TiB 22.89 0.32 150 up > 3 ssd 3.49219 1.00000 3.5 TiB 794 GiB 757 GiB 34 GiB 3.2 GiB 2.7 > TiB 22.21 0.31 146 up > 4 ssd 3.49219 1.00000 3.5 TiB 837 GiB 798 GiB 34 GiB 4.4 GiB 2.7 > TiB 23.39 0.32 156 up > 6 ssd 3.49219 1.00000 3.5 TiB 790 GiB 751 GiB 35 GiB 3.6 GiB 2.7 > TiB 22.09 0.31 146 up > 8 ssd 3.49219 1.00000 3.5 TiB 874 GiB 831 GiB 40 GiB 3.5 GiB 2.6 > TiB 24.44 0.34 156 up > 10 ssd 3.49219 1.00000 3.5 TiB 807 GiB 761 GiB 43 GiB 3.4 GiB 2.7 > TiB 22.58 0.31 146 up > 5 ssd 3.49219 1.00000 3.5 TiB 744 GiB 708 GiB 32 GiB 4.2 GiB 2.8 > TiB 20.81 0.29 141 up > 7 ssd 3.49219 1.00000 3.5 TiB 732 GiB 690 GiB 39 GiB 3.2 GiB 2.8 > TiB 20.48 0.28 136 up > 9 ssd 3.49219 1.00000 3.5 TiB 702 GiB 657 GiB 42 GiB 3.9 GiB 2.8 > TiB 19.64 0.27 131 up > 11 ssd 3.49219 1.00000 3.5 TiB 805 GiB 781 GiB 22 GiB 2.3 GiB 2.7 > TiB 22.50 0.31 138 up > 101 ssd 3.49219 1.00000 3.5 TiB 835 GiB 793 GiB 38 GiB 3.7 GiB 2.7 > TiB 23.36 0.32 146 up > 103 ssd 3.49219 1.00000 3.5 TiB 846 GiB 803 GiB 40 GiB 3.3 GiB 2.7 > TiB 23.67 0.33 150 up > 105 ssd 3.49219 1.00000 3.5 TiB 800 GiB 762 GiB 36 GiB 2.5 GiB 2.7 > TiB 22.38 0.31 148 up > 107 ssd 3.49219 1.00000 3.5 TiB 843 GiB 790 GiB 49 GiB 3.4 GiB 2.7 > TiB 23.58 0.33 147 up > 100 ssd 3.49219 1.00000 3.5 TiB 804 GiB 753 GiB 48 GiB 2.6 GiB 2.7 > TiB 22.47 0.31 144 up > 102 ssd 3.49219 1.00000 3.5 TiB 752 GiB 737 GiB 13 GiB 2.4 GiB 2.8 > TiB 21.02 0.29 141 up > 104 ssd 3.49219 1.00000 3.5 TiB 805 GiB 771 GiB 31 GiB 2.8 GiB 2.7 > TiB 22.50 0.31 144 up > 106 ssd 3.49219 1.00000 3.5 TiB 793 GiB 724 GiB 66 GiB 2.9 GiB 2.7 > TiB 22.17 0.31 143 up > 108 ssd 3.49219 1.00000 3.5 TiB 816 GiB 778 GiB 36 GiB 2.7 GiB 2.7 > TiB 22.83 0.32 156 up > 109 ssd 3.49219 1.00000 3.5 TiB 811 GiB 763 GiB 45 GiB 2.8 GiB 2.7 > TiB 22.68 0.31 146 up > 110 ssd 3.49219 1.00000 3.5 TiB 863 GiB 832 GiB 28 GiB 2.5 GiB 2.6 > TiB 24.13 0.33 154 up > 111 ssd 3.49219 1.00000 3.5 TiB 784 GiB 737 GiB 45 GiB 2.7 GiB 2.7 > TiB 21.92 0.30 146 up > > > It did not try to balance any further. Someone said he had the same issue. > I am pretty sure it will also not balance out the HDDs as neatly as > you got it there. There is definitely an issue somewhere, so far 3 > people telling the same story. I never had this issue under Luminous > but im fighting with it since 4 months on 2 clusters. One got upgraded > to Nautilus and the other one (the one the pastes are from) is a fresh > 14.2.4 one. > > Any ideas on that? > > Thanks > Am Donnerstag, 12. Dezember 2019, 02:09:33 OEZ hat David Zafman > <dzafman(a)redhat.com> Folgendes geschrieben: > > > > Philippe, > > I have a master branch version of the code to test. The nautilus > backport https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >should be the same. > > Using your OSDMap, the code in master branch and some additional changes > to osdmaptool I was able to balance your cluster. The osdmaptool > changes simulate the mgr active balancer behavior. It never took no > more than 0.13991 seconds to calculate more upmaps per round. And that's > on a virtual machine used for development. It took 35 rounds with 10 > maximum upmaps per crush rule set of pools per round. With the default > 1 minute sleeps inside the mgr it would take 35 minutes. Obviously, > recovery/backfill has to finish before the cluster settles into the new > configuration. It needed 397 additional upmaps and removed 8. > > Because all pools for a given crush rule are balanced together you can > see that this is more balanced than Rich's configuration uising Luminous. > > This balancer code is subject to change before final release of the next > Nautilus point release. > > > Final layout: > > osd.0 pgs 146 > osd.1 pgs 146 > osd.2 pgs 146 > osd.3 pgs 146 > osd.4 pgs 146 > osd.5 pgs 146 > osd.6 pgs 146 > osd.7 pgs 146 > osd.8 pgs 146 > osd.9 pgs 146 > osd.10 pgs 146 > osd.11 pgs 146 > osd.12 pgs 74 > osd.13 pgs 74 > osd.14 pgs 73 > osd.15 pgs 74 > osd.16 pgs 74 > osd.17 pgs 74 > osd.18 pgs 73 > osd.19 pgs 74 > osd.20 pgs 73 > osd.21 pgs 73 > osd.22 pgs 74 > osd.23 pgs 73 > osd.24 pgs 73 > osd.25 pgs 75 > osd.26 pgs 74 > osd.27 pgs 74 > osd.28 pgs 73 > osd.29 pgs 73 > osd.30 pgs 73 > osd.31 pgs 73 > osd.32 pgs 74 > osd.33 pgs 73 > osd.34 pgs 73 > osd.35 pgs 74 > osd.36 pgs 74 > osd.37 pgs 74 > osd.38 pgs 74 > osd.39 pgs 74 > osd.40 pgs 73 > osd.41 pgs 73 > osd.42 pgs 73 > osd.43 pgs 73 > osd.44 pgs 74 > osd.45 pgs 73 > osd.46 pgs 73 > osd.47 pgs 73 > osd.48 pgs 73 > osd.49 pgs 73 > osd.50 pgs 73 > osd.51 pgs 73 > osd.52 pgs 75 > osd.53 pgs 59 > osd.54 pgs 74 > osd.55 pgs 74 > osd.56 pgs 74 > osd.57 pgs 73 > osd.58 pgs 74 > osd.59 pgs 74 > osd.60 pgs 74 > osd.61 pgs 74 > osd.62 pgs 73 > osd.63 pgs 74 > osd.64 pgs 73 > osd.65 pgs 74 > osd.66 pgs 74 > osd.67 pgs 74 > osd.68 pgs 73 > osd.69 pgs 74 > osd.70 pgs 73 > osd.71 pgs 73 > osd.72 pgs 73 > osd.73 pgs 73 > osd.74 pgs 73 > osd.75 pgs 73 > osd.76 pgs 73 > osd.77 pgs 73 > osd.78 pgs 73 > > osd.79 pgs 73 > osd.80 pgs 73 > osd.81 pgs 73 > osd.82 pgs 73 > osd.83 pgs 73 > osd.84 pgs 73 > osd.85 pgs 73 > osd.86 pgs 73 > osd.87 pgs 73 > osd.88 pgs 73 > osd.89 pgs 73 > osd.90 pgs 73 > osd.91 pgs 73 > osd.92 pgs 73 > osd.93 pgs 73 > osd.94 pgs 73 > osd.95 pgs 73 > osd.96 pgs 73 > osd.97 pgs 73 > osd.98 pgs 73 > osd.99 pgs 73 > osd.100 pgs 146 > osd.101 pgs 146 > osd.102 pgs 146 > osd.103 pgs 146 > osd.104 pgs 146 > osd.105 pgs 146 > osd.106 pgs 146 > osd.107 pgs 146 > osd.108 pgs 146 > osd.109 pgs 146 > osd.110 pgs 146 > osd.111 pgs 146 > osd.112 pgs 73 > osd.113 pgs 73 > osd.114 pgs 73 > osd.115 pgs 73 > osd.116 pgs 73 > osd.117 pgs 73 > osd.118 pgs 73 > osd.119 pgs 73 > osd.120 pgs 73 > osd.121 pgs 73 > osd.122 pgs 73 > osd.123 pgs 73 > osd.124 pgs 73 > osd.125 pgs 73 > osd.126 pgs 73 > osd.127 pgs 74 > osd.128 pgs 73 > osd.129 pgs 73 > osd.130 pgs 73 > osd.131 pgs 73 > osd.132 pgs 73 > osd.133 pgs 73 > osd.134 pgs 73 > osd.135 pgs 73 > > > David > > On 12/10/19 9:59 PM, Philippe D'Anjou wrote: > > Given I was told its an issue of too low PGs I am raising and testing > > this, although my SSDs which have about 150 each also are not well > > distributed. > > I attached my OSDmap, I'd appreciate if you could run your test on it > > like you did with the other guy, so I know if this will ever > > distribute equally or not.. > > > > If you're busy I understand that too, then ignore this. > > > > Thanks in either case. Just have been dealing with this since months > > now and it gets frustrating. > > > > best regards > > > > Am Dienstag, 10. Dezember 2019, 03:53:17 OEZ hat David Zafman > > <dzafman(a)redhat.com <mailto:dzafman@redhat.com>> Folgendes geschrieben: > > > > > > > > Please file a tracker with the symptom and examples. Please attach your > > OSDMap (ceph osd getmap > osdmap.bin). > > > > Note that https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 > > > <https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >>has the Nautilus > > version of improved upmap code. It also changes osdmaptool to match the > > mgr behavior, so that one can observe the behavior of the upmap balancer > > offline. > > > > Thanks > > > > David > > > > On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > > > It's only getting worse after raising PGs now. > > > > > > Anything between: > > > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > > > TiB 53.62 0.76 54 up > > > > > > and > > > > > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > > > GiB 89.25 1.27 87 up > > > > > > How is that possible? I dont know how much more proof I need to > > > present that there's a bug. > > > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > <mailto:ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >

1 0

Can't create new OSD
by Rodrigo Severo - Fábrica 12 Dec '19

12 Dec '19

Hi, Trying to create a new OSD following the instructions available at https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/ On step 3 I'm instructed to run "ceph-osd -i {osd-num} --mkfs --mkkey". Unfortunately it doesn't work: # ceph-osd -i 3 --mkfs --mkkey 2019-12-11 16:59:58.257 7fac4597fc00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-11 16:59:58.257 7fac4597fc00 -1 AuthRegistry(0x55ad976ea140) no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx 2019-12-11 16:59:58.261 7fac4597fc00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-11 16:59:58.261 7fac4597fc00 -1 AuthRegistry(0x7fffac4075e8) no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) Shouldn't it create the keyring? Why is it complaining about not being able to find a keyring? Regards, Rodrigo

2 2

Ceph User Survey 2019
by Mike Perez 12 Dec '19

12 Dec '19

Hi Cephers, To better understand how our current users utilize Ceph, we conducted a public community survey. This information is a guide to the community of how we spend our contribution efforts for future development. The survey results will remain anonymous and aggregated in future Ceph Foundation publications to the community. I'm pleased to announce after much discussion on the Ceph dev mailing list [0] that the community has formed the Ceph Survey for 2019. The deadline for this survey due to it being out later than we'd like will be January 31st, 2020 at 11:59 PT. https://ceph.io/user-survey/ We have discussed in the future to use the Ceph telemetry module to collect the data to save time for our users. Please let me know of any mistakes that need to be corrected on the survey. Thanks! [0] - https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/WU374ZJP5N3NKY22X2… -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

4 3

Re: RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 12 Dec '19

12 Dec '19

Philippe, I have a master branch version of the code to test. The nautilus backport https://github.com/ceph/ceph/pull/31956 should be the same. Using your OSDMap, the code in master branch and some additional changes to osdmaptool I was able to balance your cluster. The osdmaptool changes simulate the mgr active balancer behavior. It never took no more than 0.13991 seconds to calculate more upmaps per round. And that's on a virtual machine used for development. It took 35 rounds with 10 maximum upmaps per crush rule set of pools per round. With the default 1 minute sleeps inside the mgr it would take 35 minutes. Obviously, recovery/backfill has to finish before the cluster settles into the new configuration. It needed 397 additional upmaps and removed 8. Because all pools for a given crush rule are balanced together you can see that this is more balanced than Rich's configuration uising Luminous. This balancer code is subject to change before final release of the next Nautilus point release. Final layout: osd.0 pgs 146 osd.1 pgs 146 osd.2 pgs 146 osd.3 pgs 146 osd.4 pgs 146 osd.5 pgs 146 osd.6 pgs 146 osd.7 pgs 146 osd.8 pgs 146 osd.9 pgs 146 osd.10 pgs 146 osd.11 pgs 146 osd.12 pgs 74 osd.13 pgs 74 osd.14 pgs 73 osd.15 pgs 74 osd.16 pgs 74 osd.17 pgs 74 osd.18 pgs 73 osd.19 pgs 74 osd.20 pgs 73 osd.21 pgs 73 osd.22 pgs 74 osd.23 pgs 73 osd.24 pgs 73 osd.25 pgs 75 osd.26 pgs 74 osd.27 pgs 74 osd.28 pgs 73 osd.29 pgs 73 osd.30 pgs 73 osd.31 pgs 73 osd.32 pgs 74 osd.33 pgs 73 osd.34 pgs 73 osd.35 pgs 74 osd.36 pgs 74 osd.37 pgs 74 osd.38 pgs 74 osd.39 pgs 74 osd.40 pgs 73 osd.41 pgs 73 osd.42 pgs 73 osd.43 pgs 73 osd.44 pgs 74 osd.45 pgs 73 osd.46 pgs 73 osd.47 pgs 73 osd.48 pgs 73 osd.49 pgs 73 osd.50 pgs 73 osd.51 pgs 73 osd.52 pgs 75 osd.53 pgs 59 osd.54 pgs 74 osd.55 pgs 74 osd.56 pgs 74 osd.57 pgs 73 osd.58 pgs 74 osd.59 pgs 74 osd.60 pgs 74 osd.61 pgs 74 osd.62 pgs 73 osd.63 pgs 74 osd.64 pgs 73 osd.65 pgs 74 osd.66 pgs 74 osd.67 pgs 74 osd.68 pgs 73 osd.69 pgs 74 osd.70 pgs 73 osd.71 pgs 73 osd.72 pgs 73 osd.73 pgs 73 osd.74 pgs 73 osd.75 pgs 73 osd.76 pgs 73 osd.77 pgs 73 osd.78 pgs 73 osd.79 pgs 73 osd.80 pgs 73 osd.81 pgs 73 osd.82 pgs 73 osd.83 pgs 73 osd.84 pgs 73 osd.85 pgs 73 osd.86 pgs 73 osd.87 pgs 73 osd.88 pgs 73 osd.89 pgs 73 osd.90 pgs 73 osd.91 pgs 73 osd.92 pgs 73 osd.93 pgs 73 osd.94 pgs 73 osd.95 pgs 73 osd.96 pgs 73 osd.97 pgs 73 osd.98 pgs 73 osd.99 pgs 73 osd.100 pgs 146 osd.101 pgs 146 osd.102 pgs 146 osd.103 pgs 146 osd.104 pgs 146 osd.105 pgs 146 osd.106 pgs 146 osd.107 pgs 146 osd.108 pgs 146 osd.109 pgs 146 osd.110 pgs 146 osd.111 pgs 146 osd.112 pgs 73 osd.113 pgs 73 osd.114 pgs 73 osd.115 pgs 73 osd.116 pgs 73 osd.117 pgs 73 osd.118 pgs 73 osd.119 pgs 73 osd.120 pgs 73 osd.121 pgs 73 osd.122 pgs 73 osd.123 pgs 73 osd.124 pgs 73 osd.125 pgs 73 osd.126 pgs 73 osd.127 pgs 74 osd.128 pgs 73 osd.129 pgs 73 osd.130 pgs 73 osd.131 pgs 73 osd.132 pgs 73 osd.133 pgs 73 osd.134 pgs 73 osd.135 pgs 73 David On 12/10/19 9:59 PM, Philippe D'Anjou wrote: > Given I was told its an issue of too low PGs I am raising and testing > this, although my SSDs which have about 150 each also are not well > distributed. > I attached my OSDmap, I'd appreciate if you could run your test on it > like you did with the other guy, so I know if this will ever > distribute equally or not.. > > If you're busy I understand that too, then ignore this. > > Thanks in either case. Just have been dealing with this since months > now and it gets frustrating. > > best regards > > Am Dienstag, 10. Dezember 2019, 03:53:17 OEZ hat David Zafman > <dzafman(a)redhat.com> Folgendes geschrieben: > > > > Please file a tracker with the symptom and examples. Please attach your > OSDMap (ceph osd getmap > osdmap.bin). > > Note that https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >has the Nautilus > version of improved upmap code. It also changes osdmaptool to match the > mgr behavior, so that one can observe the behavior of the upmap balancer > offline. > > Thanks > > David > > On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > > It's only getting worse after raising PGs now. > > > > Anything between: > > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > > TiB 53.62 0.76 54 up > > > > and > > > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > > GiB 89.25 1.27 87 up > > > > How is that possible? I dont know how much more proof I need to > > present that there's a bug. > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0