- ceph-users - lists.ceph.io

Re: EC profile datastore usage - question
by Mateusz Skała 03 Aug '20

03 Aug '20

Hello, I’m sorry for late response, here is output: ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.0 PiB 797 TiB 267 TiB 272 TiB 25.44 TOTAL 1.0 PiB 797 TiB 267 TiB 272 TiB 25.44 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 7.3 KiB 52 4.5 MiB 0 222 TiB default.rgw.control 2 0 B 25 0 B 0 222 TiB default.rgw.meta 3 15 KiB 152 10 MiB 0 222 TiB default.rgw.log 4 1.8 KiB 631 386 KiB 0 222 TiB default.rgw.buckets.index 6 71 MiB 37 71 MiB 0 222 TiB default.rgw.buckets.data 7 3.3 TiB 1.59M 9.9 TiB 1.46 222 TiB default.rgw.buckets.non-ec 9 962 B 10 385 KiB 0 222 TiB CephFS 10 0 B 0 0 B 0 445 TiB production-repo-old.rgw.buckets.non-ec 15 0 B 0 0 B 0 222 TiB production-repo-old.rgw.buckets.index 21 54 GiB 2 54 GiB 0 222 TiB production-repo-old.rgw.buckets.data 22 75 TiB 266.03M 257 TiB 27.82 400 TiB ceph osd pool ls detail pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 19535 lfor 0/19535/19533 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 19427 lfor 0/19427/19423 flags hashpspool stripe_width 0 pg_num_min 32 application rgw pool 3 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 19647 lfor 0/19647/19645 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15480 lfor 0/15480/15449 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 6 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15508 lfor 0/15508/15449 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 7 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 20549 lfor 0/19147/19152 flags hashpspool max_bytes 10995116277760 stripe_width 0 pg_num_min 256 application rgw pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 16099 lfor 0/16099/16097 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 10 'CephFS' erasure size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15718 lfor 0/15718/15716 flags hashpspool stripe_width 8192 pg_num_min 8 application cephfs,rbd,rgw pool 15 'production-repo-old.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 20404 flags hashpspool stripe_width 0 application rgw pool 21 'production-repo-old.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode on last_change 20547 flags hashpspool stripe_width 0 application rgw pool 22 'production-repo-old.rgw.buckets.data' erasure size 10 min_size 7 crush_rule 2 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 20546 lfor 0/20540/20542 flags hashpspool stripe_width 24576 pg_num_min 512 application rgw Best regards Mateusz Skała > Wiadomość napisana przez Eric Smith <Eric.Smith(a)vecima.com> w dniu 18.07.2020, o godz. 11:46: > > EXTERNAL EMAIL - Do not click any links or open any attachments unless you trust the sender and know the content is safe. > > Can you post the output of a couple of commands: > > ceph df > ceph osd pool ls detail > > Then we can probably explain the utilization you're seeing. > > -----Original Message----- > From: Mateusz Skała <mateusz.skala(a)gmail.com> > Sent: Saturday, July 18, 2020 1:35 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] EC profile datastore usage - question > > Hello Community, > I would like to ask about help in explanation situation. > There is Rados gateway with EC pool profile k=6 m=4. So it shoud take something about 1.4 - 2.0 data usage more from raw data if I’m correct. > Rados df shows me: > 116 TiB used and WR 26 TiB > Can You explain this? It is about 4.5*WR used data. Why? > Regards > Mateusz Skała > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [ANN] A framework for deploying Octopus using cephadm in the cloud
by Patrick Donnelly 02 Aug '20

02 Aug '20

On Sat, Aug 1, 2020 at 5:50 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > I can understand the benefits of having a CO, I am still testing with > mesos. However what is the benefit of having ceph daemons running in CO > environment? As I said in my original post, I use this for testing CephFS. There's no reason why you can't also use Ceph as a storage system in the cloud though. It just may not be as cost effective as the native cloud storage offering (like S3). >Except for your mds, mrg and radosgw, your osd daemons are > bound to the hardware / disks they are running on. It is not like if > osd.121 goes down, you can start it on some random node. Why not? -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Why newly added OSD need to get all historical OSDMAPs in pre-boot
by Xiaoxi Chen 02 Aug '20

02 Aug '20

Hi List, We see newly added OSD take long time to become active, it is fetching old osdmaps from monitors. Curious why it need that much historical ? The cluster is stable , no osd down/out except the ongoing cap add (one osd at a time). { "cluster_fsid": "e959f744-64be-4f4e-9606-103ccaa9d2d6", "osd_fsid": "0a2d4b9d-f6e5-4e40-be7f-16daee131a20", "whoami": 38, "state": "preboot", "oldest_map": 193222, "newest_map": 298982, "num_pgs": 0 } root@host:~# ceph osd getmap | head -5 got osdmap epoch 324825 -Xiaoxi

1 0

which exact decimal value is meant here for S64_MIN in CRUSH Mapper
by Bobby 01 Aug '20

01 Aug '20

Hi, In *mapper.c* file of Ceph CRUSH, I am trying to understand the definition of a linux macro ```*S64_MIN*``` used in the following ```*else*``` condition i.e. ```*draw = S64_MIN*```. Which exact decimal value is meant here for ```*S64_MIN*```? ``` if (weights[i]) { u = hash(bucket->h.hash, x, ids[i], r); u &= 0xffff; ln = crush_ln(u) - 0x1000000000000ll; __s64 draw = div64_s64(ln, weights[i]); } else { __s64 draw = S64_MIN; // #define S64_MAX ((s64)(U64_MAX >> 1)) // #define S64_MIN ((s64)(-S64_MAX -1)) } if (i == 0 || draw > high_draw) { high = i; high_draw = draw; } } return bucket->h.items[high]; } Thanks Bobby !

1 0

[ANN] A framework for deploying Octopus using cephadm in the cloud
by Patrick Donnelly 01 Aug '20

01 Aug '20

Hello all, There's a framework [0] I've been working on for a while to deploy Ceph in the cloud. For now, this is done through the Linode, LLC cloud provider [1]. Primarily I use this for testing CephFS performance/behavior. For example, it's fairly simple to create a decent sized test cluster with 150GB MDSs and 16x 16GB OSDs for a humble 1.7TB usable storage. Then provision 100+ client machine nodes for executing workflows against the Ceph cluster. This has been very useful for isolating aberrant behaviors that are only uncovered at scale. All of the code/playbooks I use for this purpose are also in the ceph-linode repository. Early versions of this project used ceph-ansible but I have updated the code to use the new cephadm deployment technology [2] in Ceph Octopus release. The ansible playbook to deploy the cluster [3] is delightfully simple. Anyway, I thought it may be useful to the broader community to have an option to try out cephadm on a throwaway cluster for pennies per hour through a VPS provider. Getting a small cluster started should take less than 10 minutes by following the README. I hope some folks out there find this useful for their own testing. Feedback is welcome. [0] https://github.com/batrick/ceph-linode [1] https://www.linode.com/ [2] https://docs.ceph.com/docs/master/cephadm/ [3] https://github.com/batrick/ceph-linode/blob/master/cephadm.yml Full disclosure: I have no relationship with Linode except as a customer. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 2

Boost your grades using assignment help online services
by kevin wick 01 Aug '20

01 Aug '20

Are you feeling irritated because of poor grades in assignment submission? Do you need assistance to boost your grades in assignment writing for Hong Kong universities? Then, divert your mind for assignment writing services and connect with a reliable service provider like us. We are the leading online assignment writing service provider in Hong Kong and help every scholar to enhance his or her marks. Our writers help students to expand their knowledge and boost their scores. You will receive the valuable Assignment Help services through online mode of communication in Hong Kong. By choosing our academic writing services, you can save your time and effort to perform great in your all academic tasks. Leave your irritation of getting lower marks using the assistance of subject matter experts in Hong Kong. https://www.greatassignmenthelp.com/hk/

1 0

What are smart ways to fix HP printer offline?
by masonlava77＠gmail.com 31 Jul '20

31 Jul '20

HP printer offline is one of the most common errors experienced by countless users. This offline problem is showing a warning message that printing machine does not print any kind of document. I am passing through technical issue from the last week. I don’t have solutions for this offline error, hence my all printing tasks are pending. Why does my HP printer keep going offline? I look for expert’s assistance. So anyone can guide me properly for solving this issue. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

HP Officejet pro 802 setup
by cfaray3536＠gmail.com 31 Jul '20

31 Jul '20

Get the quick solution to establish the hp Officejet pro 8028 setups without any interruptions. However, if you still have issues or errors and you are stuck in between the process then all you need to do is to visit the HP website and get assistance from the customer support team https://hp-printer-support.us/blog/hp-officejet-pro-8028-setup/

1 0

What is Mcafee Login?
by sophia.smith0321＠gmail.com 31 Jul '20

31 Jul '20

Mcafee is a globally acclaimed brand that gives unbeatable cybersecurity solutions for all of your connected device. It provides the simplest of the breed software applications and tools that reach holistic and robust protection. These innovative products are an ideal blend of avant-garde technology, intuitive features and unsurpassable performance. This user-friendly feature-packed software guarantees unmatchable user experience. It further elevates its overall experience by granting complete control to users through an extremely intuitive <a href="https://mcafeecomactivates.com/mcafee-login/">McAfee Login</a> account. Mcafee Login Account <a href="https://mcafeecomactivates.com/mcafee-login/">McAfee Account Login</a> is your one-stop key for accessing and utilising all the advantages and features of the amazing Mcafee security applications. This online platform allows you to purchase and manage your subscriptions with utmost ease. Besides, it allows you to manage your account and billing information. You can purchase and add new products; register, activate and renew subscriptions; update billing and profile details; check purchase history, add devices and a lot more. In short, it grants you complete control of your service and its features.

1 0

unbalanced pg/osd allocation
by Budai Laszlo 31 Jul '20

31 Jul '20

Dear all, We have a ceph cluster where we are have configured two SSD only pools in order to use them as cache tier for the spinning discs. Altogether there are 27 SSDs organized on 9 hosts distributed in 3 chassis. The hierarchy looks like this: $ ceph osd df tree | grep -E 'ssd|ID' ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -40 8.26199 - 8.26TiB 5.78TiB 2.48TiB 70.02 5.77 - root ssd-root -50 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c1-ssd -41 0.91800 - 940GiB 651GiB 289GiB 69.23 5.71 - host c1-h01-ssd 110 ssd 0.30600 1.00000 313GiB 199GiB 115GiB 63.37 5.22 77 osd.110 116 ssd 0.30600 1.00000 313GiB 219GiB 94.3GiB 69.91 5.76 89 osd.116 119 ssd 0.30600 1.00000 313GiB 233GiB 80.2GiB 74.41 6.13 87 osd.119 -42 0.91800 - 940GiB 701GiB 239GiB 74.61 6.15 - host c1-h02-ssd 112 ssd 0.30600 1.00000 313GiB 228GiB 84.9GiB 72.91 6.01 85 osd.112 117 ssd 0.30600 1.00000 313GiB 245GiB 67.9GiB 78.32 6.46 97 osd.117 122 ssd 0.30600 1.00000 313GiB 227GiB 85.8GiB 72.61 5.99 87 osd.122 -43 0.91800 - 940GiB 622GiB 318GiB 66.21 5.46 - host c1-h03-ssd 109 ssd 0.30600 1.00000 313GiB 192GiB 122GiB 61.15 5.04 77 osd.109 115 ssd 0.30600 1.00000 313GiB 206GiB 107GiB 65.79 5.42 79 osd.115 120 ssd 0.30600 1.00000 313GiB 225GiB 88.7GiB 71.70 5.91 90 osd.120 -51 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c2-ssd -46 0.91800 - 940GiB 651GiB 288GiB 69.31 5.71 - host c2-h01-ssd 125 ssd 0.30600 1.00000 313GiB 211GiB 103GiB 67.22 5.54 81 osd.125 130 ssd 0.30600 1.00000 313GiB 233GiB 80.4GiB 74.33 6.13 89 osd.130 132 ssd 0.30600 1.00000 313GiB 208GiB 105GiB 66.38 5.47 79 osd.132 -45 0.91800 - 940GiB 672GiB 267GiB 71.54 5.90 - host c2-h02-ssd 126 ssd 0.30600 1.00000 313GiB 216GiB 97.4GiB 68.90 5.68 87 osd.126 129 ssd 0.30600 1.00000 313GiB 207GiB 106GiB 66.12 5.45 80 osd.129 134 ssd 0.30600 1.00000 313GiB 249GiB 63.9GiB 79.61 6.56 99 osd.134 -44 0.91800 - 940GiB 650GiB 289GiB 69.20 5.70 - host c2-h03-ssd 123 ssd 0.30600 1.00000 313GiB 201GiB 112GiB 64.23 5.29 76 osd.123 127 ssd 0.30600 1.00000 313GiB 217GiB 96.1GiB 69.31 5.71 85 osd.127 131 ssd 0.30600 1.00000 313GiB 232GiB 81.2GiB 74.06 6.11 92 osd.131 -52 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c3-ssd -47 0.91800 - 940GiB 628GiB 311GiB 66.86 5.51 - host c3-h01-ssd 124 ssd 0.30600 1.00000 313GiB 204GiB 109GiB 65.13 5.37 78 osd.124 128 ssd 0.30600 1.00000 313GiB 202GiB 111GiB 64.59 5.32 76 osd.128 133 ssd 0.30600 1.00000 313GiB 222GiB 91.3GiB 70.86 5.84 86 osd.133 -48 0.91800 - 940GiB 628GiB 312GiB 66.80 5.51 - host c3-h02-ssd 108 ssd 0.30600 1.00000 313GiB 220GiB 92.9GiB 70.35 5.80 86 osd.108 114 ssd 0.30600 1.00000 313GiB 209GiB 105GiB 66.58 5.49 82 osd.114 121 ssd 0.30600 1.00000 313GiB 199GiB 114GiB 63.46 5.23 79 osd.121 -49 0.91800 - 940GiB 718GiB 222GiB 76.40 6.30 - host c3-h03-ssd 111 ssd 0.30600 1.00000 313GiB 219GiB 94.4GiB 69.87 5.76 84 osd.111 113 ssd 0.30600 1.00000 313GiB 241GiB 72.2GiB 76.95 6.34 96 osd.113 118 ssd 0.30600 1.00000 313GiB 258GiB 55.2GiB 82.39 6.79 101 osd.118 The rule used for the two pools is the following: { "rule_id": 1, "rule_name": "ssd", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -40, "item_name": "ssd-root" }, { "op": "chooseleaf_firstn", "num": 0, "type": "chassis" }, { "op": "emit" } ] } both pools have the size 3, and the total number of PGs is 768 (256+512). As you can see from the previous table (the PG column) there is a significant difference between the OSD with the largest number of PGs (101PGs on osd.118) and the ones with the smallest number (76 PGs on osd.123). The ratio between the two is 1.32. So OSD 118 has more chances to receive data then OSD 123, and we can see that indeed osd.118 is the one storing the most data (82.39% full in the above table). I would like to re balance the PG/OSD allocation. I know that I can play around with the OSD weights (currently .306 for all the OSDs), but I wonder if there is any drawback for this on the long run? Are you aware of any reason why I should NOT modify the weights (and leave those modifications permanent)? Any ideas are welcome :) Kind regards, Laszlo

2 2