Hi all!
Reaching out again about this issue since I haven't had much luck. We've
been seeing some strange behavior with our object storage cluster. While
bucket stats (radosgw-admin bucket stats) normally return in a matter of
seconds, we frequently observe it taking almost ten minutes, which is not
convenient since we use those bucket stats for billing/accounting.
Restarting the radosgw process on the RGWs fixes this issue until it crops
up again in maybe a few days.
Someone mentioned that they think this might have to do with bucket
deletions, or more specifically, lifecycle policies to abort incomplete
multipart uploads. He mentioned there was an item in the bug tracker for
this, but I have not been able to find said bug in the tracker. I have no
clue if this is the case or not, but I figured I'd throw it out there to
see if anyone else has run into this problem. I have seen many of these
messages in my RGW logs:
2019-12-02 13:12:52.882 7faa7018f700 0 abort_bucket_multiparts WARNING :
aborted 8553000 incomplete multipart uploads
So maybe there is some truth to the aborted multipart uploads causing
problems?
My cluster has over 200 OSDs, 10 RGWs, about 2200 buckets. Running Nautilus
14.2.5.
If anyone has run into this or has any information I'd appreciate it.
Merry Christmas & Happy Holidays,
- Dave
Hi,
if "MAX AVAIL" displays the wrong data, the bug is just made more
visible through the dashboard, as the calculation is correct.
To get the right percentage you have to divide the used space through
the total, and the total can only consist of two states used and not
used space, so both states will be added together to get the total.
Or in short:
used / (avail + used)
Just looked into the C++ code - Max avail will be calculated the
following way:
avail_res = avail / raw_used_rate (
https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L905)
raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) /
sum.num_object_copies
(https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L892)
Am Dienstag, den 17.12.2019, 07:07 +0100 schrieb ceph(a)elchaka.de:
> I have observed this in the ceph nautilus dashboard too - and Think
> it is a Display Bug... but sometimes it Shows tue right values
>
>
> Which nautilus u use?
>
>
> Am 10. Dezember 2019 14:31:05 MEZ schrieb "David Majchrzak, ODERLAND
> Webbhotell AB" <david(a)oderland.se>:
> > Hi!
> >
> > While browsing /#/pool in nautilus ceph dashboard I noticed it said
> > 93%
> > used on the single pool we have (3x replica).
> >
> > ceph df detal however shows 81% used on the pool and 67% raw
> > useage.
> >
> > # ceph df detail
> > RAW STORAGE:
> > CLASS SIZE AVAIL USED RAW USED %RAW
> > USED
> > ssd 478 TiB 153 TiB 324 TiB 325
> > TiB 67.96
> > TOTAL 478 TiB 153 TiB 324 TiB 325
> > TiB 67.96
> >
> > POOLS:
> > POOL ID STORED OBJECTS USED %USED
> >
> > MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED
> > COMPR UNDER COMPR
> > echo 3 108 TiB 29.49M 324
> > TiB 81.61 24
> > TiB N/A N/A 29.49M 0
> > B 0 B
I manually calculated the used percentage to get "avail" in your case
it seems to be 73 TiB. That means the the total space available for
your pool would be 397 TiB.
I'm not sure why that is, but it's what the math behind those
calculations say.
(Found a thread regarding that on the new mailing list (ceph-
users(a)ceph.io) ->
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NH2LMMX5KVR…
)
0.8161 = used (324) / total => total = 397
Than I looked at the remaining calculations:
raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) /
sum.num_object_copies
and
avail_res = avail / raw_used_rate
First I looked up the init value for "raw_used_rate" for replicated
pools. It's their size so we can put in 3 here and for "avail_res" is
24.
So I first calculated the final "raw_used_rate" which is 3.042. That
means that you have around 4.2% degraded pg's in your pool.
> >
> >
> > I know we're looking at the most full OSD (210PGs, 79% used, 1.17
> > VAR)
> > and count max avail from that. But where's the 93% full from in
> > dashboard?
As said above the calculation is right but the data is wrong... As it
uses the real data that can be put in the selected pool, but it uses
everywhere else sizes that consider all pool replicas.
I created an issue to fix this https://tracker.ceph.com/issues/43384
> >
> > My guess is that is comes from calculating:
> >
> > 1 - Max Avail / (Used + Max avail) = 0.93
> >
> >
> > Kind Regards,
> >
> > David Majchrzak
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users(a)lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users(a)lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hope I could clarify some things and thanks for your feedback :)
BTW this problem currently still exists as there wasn't any change to
these mentioned lines after the nautilus release.
Stephan
So I wanted to report a crush rule/ec profile strange behaviour regarding radosgw items which i am not sure if it's a bug or it's supposed to work that way.
I am trying to implement the below scenario in my home lab:
By default there is a "default" erasure-code-profile with the below settings:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=2
m=1
plugin=jerasure
technique=reed_sol_van
w=8
From the above we see that it uses the root bucket. Now ofcourse you would want to create your own ec-profile with custom algorithm/crush buckets etc
Let's say for example we create two new ec profiles. One with specific crush-root = ssd-performance2 and one with the crush-root=default (there are no disks there according to ceph osd tree-> end of page)
ceph osd erasure-code-profile set test-ec crush-device-class= crush-failure-domain=host crush-root=ssd-performance2 jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8
ceph osd erasure-code-profile set test-ec2 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8
Now let's create the associated crush rules to use these profiles:
ceph osd crush rule create-erasure erasure-test-rule1 test-ec
ceph osd crush rule create-erasure erasure-test-rule2 test-ec2
Now let's say you have a radosgw server that has started and by default it creates the 5 default radosgwpools(supposed you have uploaded some data as well):
default.rgw.buckets.data
default.rgw.buckets.index
default.rgw.control
default.rgw.log
default.rgw.meta
Now if you grep these pools with ceph osd dump you will see that all of them are using replicated rules but we want to use erasure for the radosgw data pool. So let's migrate the default.rgw.buckets.data pool to a erasure-coded one.
1) We shutdown the radosgw-server so that we don't allow any requests coming in.
2) ceph osd pool rename default.rgw.buckets.data default.rgw.buckets.data-old
3) ceph osd pool create default.rgw.buckets.data 8 8 erasure test-ec erasure-test-rule - > We use the newly created erasure crush rule with the profile we created and use the ssd-performance2 root bucket
4) rados cppool default.rgw.buckets.data-old default.rgw.buckets.data
5) Start radosgw server again
At this point i can see the old objects and i can upload new objects in radosgw and everything is working fine.
Now i see this strange behavior after i do the below:
We set the default.rgw.buckets.data to use the other erasure crush rule (This is using the root bucket=default which doesn't have any disks):
ceph osd pool set default.rgw.buckets.data crush_rule erasure-test-rule2
Bug1? You could still browse the data but any attempt to upload/download hangs there with the below log messages:
2019-12-18 17:07:07.037 7f05a1ece700 0 ERROR: client_io->complete_request() returned Input/output error
2019-12-18 17:07:07.037 7f05a1ece700 2 req 712 0.004s s3:list_buckets op status=
Monitor nodes don't display anything and seems that new items cannot be saved (which is correct as it doesn't know where to save them) but at least Monitor nodes should display something as a warning or there must be crush check before to see if the rule can be applied?
Reverting back the rule to erasure-test-rule works fine again
=================================
Bug 2? If you modify the erasure-test-rule profile to use a null crush bucket (like erasure-test-rule2) then this is not being parsed and identified by the crush rule. Seems crush rules skips that part
Example:
ceph osd erasure-code-profile set test-ec crush-root=default --force
At this point nothing happens and radosgw is working fine. Which it shouldn't as it should see that the data cannot be saved anywhere. Unless it keeps the crush root bucket from the crush rules and not from the erasure coded profiles...even if you force apply/change it to the erasure profile like above.
=================================
Bug 3? You don't know which rule is using which erasure-code-profile from ceph osd dump. You only see that this pool is using crush rule number 1 but if you dump this crush rule it doesn't mention which erasure-code profile is using, other than which item_name eg = root bucket
Even with the telemetry on with latest release and if you do "ceph telemetry show basic" with below you see there is no crush-root being mentioned.
So is the crush rule > erasure_code_profile regarding parsing of the crush_root buckets?
{
"min_size": 2,
"erasure_code_profile": {
"crush-failure-domain": "host",
"k": "2",
"technique": "reed_sol_van",
"m": "1",
"plugin": "jerasure"
},
"pg_autoscale_mode": "warn",
"pool": 860,
"size": 3,
"cache_mode": "none",
"target_max_objects": 0,
"pg_num": 8,
"pgp_num": 8,
"target_max_bytes": 0,
"type": "erasure"
}
root@ceph-mon01:~# ceph osd crush rule dump erasure-test-rule
{
"rule_id": 2,
"rule_name": "erasure-test-rule",
"ruleset": 2,
"type": 3,
"min_size": 3,
"max_size": 3,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "ssd-performance2"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
root@ceph-mon01:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-37 0.18398 root really-low
-40 0.09799 host ceph-osd01-really-low
11 hdd 0.09799 osd.11 up 1.00000 1.00000
-41 0.04799 host ceph-osd02-really-low
1 hdd 0.01900 osd.1 up 1.00000 1.00000
9 hdd 0.02899 osd.9 up 1.00000 1.00000
-42 0.03799 host ceph-osd03-really-low
6 hdd 0.01900 osd.6 up 1.00000 1.00000
7 hdd 0.01900 osd.7 up 1.00000 1.00000
-23 10.67598 root spinning-rust
-20 2.04900 rack rack1
-3 2.04900 host ceph-osd01
3 hdd 0.04900 osd.3 up 0.95001 1.00000
22 hdd 1.00000 osd.22 up 0.90002 1.00000
17 ssd 1.00000 osd.17 up 1.00000 1.00000
-25 3.07799 rack rack2
-5 3.07799 host ceph-osd02
4 hdd 0.04900 osd.4 up 1.00000 1.00000
8 hdd 0.02899 osd.8 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
12 ssd 1.00000 osd.12 up 1.00000 1.00000
-28 3.54900 rack rack3
-7 3.54900 host ceph-osd03
0 hdd 1.00000 osd.0 up 0.90002 1.00000
5 hdd 0.04900 osd.5 up 1.00000 1.00000
30 hdd 0.50000 osd.30 up 1.00000 1.00000
21 ssd 1.00000 osd.21 up 0.95001 1.00000
24 ssd 1.00000 osd.24 up 1.00000 1.00000
-55 2.00000 rack rack4
-49 2.00000 host ceph-osd04
26 hdd 1.00000 osd.26 up 1.00000 1.00000
27 hdd 1.00000 osd.27 up 1.00000 1.00000
-2 9.10799 root ssd-performance2
-32 2.09799 host ceph-osd01-ssd
2 ssd 0.09799 osd.2 up 1.00000 1.00000
13 ssd 1.00000 osd.13 up 1.00000 1.00000
16 ssd 1.00000 osd.16 up 1.00000 1.00000
-31 3.00000 host ceph-osd02-ssd
14 ssd 1.00000 osd.14 up 1.00000 1.00000
18 ssd 1.00000 osd.18 up 1.00000 1.00000
19 ssd 1.00000 osd.19 up 1.00000 1.00000
-9 2.00999 host ceph-osd03-ssd
10 ssd 0.00999 osd.10 up 0.90002 1.00000
15 ssd 1.00000 osd.15 up 1.00000 1.00000
20 ssd 1.00000 osd.20 up 1.00000 1.00000
-52 2.00000 host ceph-osd04-ssd
28 ssd 1.00000 osd.28 up 1.00000 1.00000
29 ssd 1.00000 osd.29 up 1.00000 1.00000
-1 0 root default
root@ceph-mon01:~#
Thanks,
Anastasios
Hi!
We've found ourselves in state with our ceph cluster that we haven't seen before, and are looking for a bit of expertise to chime in. We're running a (potentially unusually laid out) moderately large luminous-based ceph cluster in a public cloud, with 234*8TB OSDs, with a single osd per cloud instance. Here's a snippet of our ceph status:
services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon4
mgr: ceph-mon2(active), standbys: ceph-mon1, ceph-mon4
osd: 234 osds: 234 up, 231 in
data:
pools: 5 pools, 7968 pgs
objects: 136.35M objects, 382TiB
usage: 1.09PiB used, 713TiB / 1.79PiB avail
pgs: 7924 active+clean
44 active+clean+scrubbing+deep
Our layout is spread across 6 availability zones (with the majority of osds in three, us-east-1a,us-east-1b and us-east-1e). We've recently decided that a spread across six azs is unnecessary and potentially a detriment to performance, so we are working towards shifting our workload out of 3 of the 6 azs, so that we are evenly placed in 3 azs.
Relevant Recent events:
1. We expanded by 24 osds to handle additional capacity needs as well as to provide the capacity necessary to remove all osds in us-east-1f.
2. After the re-balance related to #1 finished we expanded by an additional 12 osds as a follow-on for the #1 change.
3. On the same day as #2, we also issued `ceph crush move` commands to move the location of oldest 20 osds which had previously not been configured with a "datacenter" bucket denoting their availability zone.
The re-balancing related to #3 caused quite a change in our cluster, resulting in hours with degraded pgs, and waves of "slow requests" from many of the osds as data shifted. Three of the osds which had their location moved are also wildly more full than the other osds (being in a nearfull, >85% utilized state where the other 231 osds are in the range of 58-63% utilized). A `ceph balancer optimize` plan shows no changes to be made by the balancer to rectify this. Because of their nearfull status, we have marked those three osds as "out". During the resulting re-balancing we experienced more "slow requests" piling up, with some pgs dipping into a peering or activating state (which obviously causes some user-visible trouble).
Where we are headed:
* We have yet to set the crush map to use a replicated_datacenter rule now that our osds conform to these buckets.
* We need to take action to remove/replace the three osds which were over-utilized.
* We need to remove the us-east-1f osds that provoked the event in #1 (of which there are 16).
What I'm hoping for a bit of direction on:
* A recommendation on the order of these operations that would result in the least user-visible time.
* Some hints or thoughts on why we are seeing flapping pg availability since the change in #3.
* General thoughts on our layout. It has been rather useful for unbounded rapid growth but it seems very likely the sheer count of osds/instances is optimal at this point.
Obviously I'm happy to provide any additional information that might be helpful that I've overlooked. Thanks so much for looking!
Steve
Hi guys, I want to use tcpdump and Wireshark to capture and analysis
packages between clients and ceph cluster. But the protocol only shows tcp,
no ceph, so I can not read the data between client and cluster. The
wireshark version is 3.07.
Hope your help.
Thank you.
Hi!
Is there a mean to list all snapshots existing in a (subdir of) Cephfs?
I can't use the find dommand to look for the ".snap" dirs.
I'd like to remove certain (or all) snapshots within a CephFS. But how do I find them?
Thanks,
Lars
hi all,
seek your help for this.
we are using luminous 12.2.12 and we enabled 3 active mds.
when I running "ceph daemon mds.<x> dump loads" on any active mds, I always saw such like below
"mds_load": {
"mds.0": {
"request_rate": 526.045993,
"cache_hit_rate": 0.000000,
...
"mds.1": {
"request_rate": 169.845956,
"cache_hit_rate": 0.000000,
...
"mds.2": {
"request_rate": 300.511478,
"cache_hit_rate": 0.000000,
don't understand what's this 'cache_hit_rate' and why it seems always 0?
we actually set a bigger mds_cache_memory_limit than default such like 16G in /etc/ceph/ceph.conf. are they related?
I checked https://docs.ceph.com/docs/luminous/cephfs/mds-config-ref/ , but didn't get further clue from the description of "mds cache *" settings.