- ceph-users - lists.ceph.io

Questions on Ceph on ARM
by norman 09 Jul '20

09 Jul '20

Hi all, I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster?

5 10

bluestore: osd bluestore_allocated is much larger than bluestore_stored
by Jerry Pu 09 Jul '20

09 Jul '20

Hi: We have a cluster (v13.2.4), and we do some tests on a EC k=2, m=1 pool "VMPool0". We deploy some VMs (Windows, CentOS7) on the pool and then use IOMeter to write data to these VMs. After a period of time, we observe a strange thing that pool actual usage is much larger than stored data * 1.5 (stored_raw). [root@Sim-str-R6-4 ~]# ceph df GLOBAL: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB 48.74 TOTAL 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB 48.74 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 1 29 GiB 100.00 0 B 2597 cephfs_md 2 831 MiB 100.00 0 B 133 erasure_meta_hdd 3 22 MiB 100.00 0 B 170 VMPool0 4 1.2 TiB 56.77 644 GiB 116011 stresspool 5 2.6 MiB 100.00 0 B 32 [root@Sim-str-R6-4 ~]# ceph df detail -f json-pretty -----snippet----- { "name": "VMPool0", "id": 4, "stats": { "kb_used": 1328888832, "bytes_used": 1360782163968, <---------------- "percent_used": 0.567110, "max_avail": 692481687552, "objects": 116011, "quota_objects": 0, "quota_bytes": 0, "dirty": 116011, "rd": 27449034, "rd_bytes": 126572760064, "wr": 20675381, "wr_bytes": 1006460652544, "comp_ratio": 1.000000, "stored": 497657610240, "stored_raw": 746486431744, <---------------- } }, The perf counters of all osds (all hdd) used by VMPool0 also show that bluestore_allocated is much larger than bluestore_stored. [root@Sim-str-R6-4 ~]# for i in {0..3}; do echo $i; ceph daemon osd.$i perf dump | grep bluestore | head -6; done 0 "bluestore": { "bluestore_allocated": 175032369152, <---------------- "bluestore_stored": 83557936482, <---------------- "bluestore_compressed": 958795770, "bluestore_compressed_allocated": 6431965184, "bluestore_compressed_original": 18576584704, 1 "bluestore": { "bluestore_allocated": 119943593984, <---------------- "bluestore_stored": 53325238866, <---------------- "bluestore_compressed": 670158436, "bluestore_compressed_allocated": 4751818752, "bluestore_compressed_original": 13752328192, 2 "bluestore": { "bluestore_allocated": 155444707328, <---------------- "bluestore_stored": 69067116553, <---------------- "bluestore_compressed": 565170876, "bluestore_compressed_allocated": 4614324224, "bluestore_compressed_original": 13469696000, 3 "bluestore": { "bluestore_allocated": 128179240960, <---------------- "bluestore_stored": 60884752114, <---------------- "bluestore_compressed": 1653455847, "bluestore_compressed_allocated": 9741795328, "bluestore_compressed_original": 27878768640, [root@Sim-str-R6-5 osd]# for i in {4..7}; do echo $i; sh -c "ceph daemon osd.$i perf dump | grep bluestore | head -6"; done 4 "bluestore": { "bluestore_allocated": 165950652416, <---------------- "bluestore_stored": 80255191687, <---------------- "bluestore_compressed": 1526871060, "bluestore_compressed_allocated": 8900378624, "bluestore_compressed_original": 25324142592, 5 admin_socket: exception getting command descriptions: [Errno 111] Connection refused 6 "bluestore": { "bluestore_allocated": 166022152192, <---------------- "bluestore_stored": 84645390708, <---------------- "bluestore_compressed": 1169055606, "bluestore_compressed_allocated": 8647278592, "bluestore_compressed_original": 25135091712, 7 "bluestore": { "bluestore_allocated": 204633604096, <---------------- "bluestore_stored": 100116382041, <---------------- "bluestore_compressed": 1081260422, "bluestore_compressed_allocated": 6510018560, "bluestore_compressed_original": 18654052352, [root@Sim-str-R6-6 osd]# for i in {8..12}; do echo $i; ceph daemon osd.$i perf dump | grep bluestore | head -6; done 8 "bluestore": { "bluestore_allocated": 106330193920, <---------------- "bluestore_stored": 45282848089, <---------------- "bluestore_compressed": 1136610231, "bluestore_compressed_allocated": 7248609280, "bluestore_compressed_original": 20882960384, 9 "bluestore": { "bluestore_allocated": 120657412096, <---------------- "bluestore_stored": 52550745942, <---------------- "bluestore_compressed": 1321632665, "bluestore_compressed_allocated": 7401504768, "bluestore_compressed_original": 21073027072, 10 "bluestore": { "bluestore_allocated": 155985772544, <---------------- "bluestore_stored": 73236910054, <---------------- "bluestore_compressed": 98351920, "bluestore_compressed_allocated": 772210688, "bluestore_compressed_original": 2242043904, 11 "bluestore": { "bluestore_allocated": 106040524800, <---------------- "bluestore_stored": 45353612134, <---------------- "bluestore_compressed": 874216443, "bluestore_compressed_allocated": 4962844672, "bluestore_compressed_original": 14160310272, 12 "bluestore": { "bluestore_allocated": 118751363072, <---------------- "bluestore_stored": 52194408691, <---------------- "bluestore_compressed": 782919969, "bluestore_compressed_allocated": 5546311680, "bluestore_compressed_original": 16043233280, The config min_alloc_size_hdd of all osds are 64K and rados objects of VMPool0 are all 4M rbd_data.x.xxxxxxx.xxxxxxx objects. It's kind of strange that allocation space is much larger than stored data. Can anyone explains this?

2 7

bucket index nvme
by Szabo, Istvan (Agoda) 09 Jul '20

09 Jul '20

Hello, Can someone explain me a bit about the objectstore indexing? It's not really clear, when redhat says one of the important tuning for objectstore is to put the indexes on a fast drive, when I check our current ceph cluster and I see petabytes of read operations but the size of the index pool is 0, so how I can size 1 nvme drive in each server with 4 osd to host the index pool? Also thinking to share the nvme with different partition for journaling and bucket index, but no problem to order 1 more for it. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Activate Xfinity Channel Via xfinity.com authorize
by xfinityauthorize26＠gmail.com 09 Jul '20

09 Jul '20

Xfinity or Xfinity Stream is a streaming service which provides live broadcast channels, DVR recordings, linear cable channels and on-demand videos. You can watch live TV channels from nearly 200+ favourite networks. Additionally, it is possible to get on-demand videos for offline access. In the next guide, we will illustrate how to prepare and trigger the Xfinity Stream app on Roku apparatus. Currently, Xfinity is available in a beta version for Roku platforms, and plus It Doesn't incorporate the Entire set of features and functionality included in the very first variants . https://www.xfinitycomauthorize.com/

1 0

Ceph multisite secondary zone not sync new changes
by Amit Ghadge 09 Jul '20

09 Jul '20

Hello All, In our test environment we set up ceph multisite in Active/Passive. Cluster A migrated to the master zone without deleting any data and set up a fresh secondary zone. First we stop pushing data to master zone and secondary zone sync all buckets and objects but later 1 hour started uploads 1million objects with newly created buckets this new changes not sync by secondary. It still showing, # radosgw-admin sync status realm 2a7b2a08-3404-40ea-81ed-2d036ee6e54f (masifd) zonegroup 039c426b-4032-4621-9034-d3ee724967a0 (us) zone 213b25f9-e726-43c8-a82e-4f9f0d4ed55c (us-east) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 0c6643f3-0289-4811-ab3e-dff2952e31c6 (us-west) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source

1 0

ceph nautilus repository index is incomplet
by Francois Legrand 09 Jul '20

09 Jul '20

Hello, It seems that the index of https://download.ceph.com/rpm-nautilus/el7/x86_64/ repository is wrong. Only the 14.2.10-0.el7 version is available (all previous versions are missing despite the fact that the rpms are present in the repository). It thus seems that the index needs to be corrected. Who can I contact for that ? Thanks. F.

1 0

Install And Activate Nbc Channel On Roku
by nbcactivate26＠gmail.com 09 Jul '20

09 Jul '20

An American-English commercial real radio and tv system are a forerunner land of the NC Universal. Now, you can view the displays from round the NBC Universal Universal networks such as E!, Oxygen, SYFY, USA, Bravo. The NBC This station is currently available on particular streaming apparatus like Roku, Amazon Fire Stick. All of that should be done would be to bring the station on into the streaming device then need to trigger the station by seeing https://www.nbccomactivatetv.com/

1 0

Octopus upgrade breaks Ubuntu 18.04 libvirt
by Andrei Mikhailovsky 08 Jul '20

08 Jul '20

Hello, I've upgraded ceph to Octopus (15.2.3 from repo) on one of the Ubuntu 18.04 host servers. The update caused problem with libvirtd which hangs when it tries to access the storage pools. The problem doesn't exist on Nautilus. The libvirtd process simply hangs. Nothing seem to happen. The log file for the libvirtd shows: 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. Running strace on the libvirtd process shows: root@ais-cloudhost1:/home/andrei# strace -p 12040 strace: Process 12040 attached restart_syscall(<... resuming interrupted poll ...> Nothing happens after that point. The same host server can get access to the ceph cluster and the pools by running ceph -s or rbd -p <pool> ls -l commands for example. Need some help to get the host servers working again with Octopus. Cheers

3 9

problem with ceph osd blacklist
by 380562518＠qq.com 08 Jul '20

08 Jul '20

Hi all. If cephfs client is in a slow or unreliable network environment, the client will be added to OSD blacklist and OSD map, and the default duration is 1 hour. During this time, the client will be forbidden to access CEPH. If I want to solve this problem and ensure client's normal I/O operation is not interrupted, are the following two options feasible? which one is better? 1. Set "mds_session_blacklist_on_timeout" to false, and forbid to add slow clients to blacklist; 2.Just reduce the duration of slow clients joining the blacklist, and change the default 1 hour to 5 minutes. (set the value of "mon_osd_blacklist_default_expire" to 5 minutes） Are the two schemes feasible? Will it have a great impact on the data security and integrity of CEPH? Can you give me some suggestions? Thanks.

2 1

RGW: rgw_qactive perf is constantly growing
by Jacek Suchenia 08 Jul '20

08 Jul '20

Hi On two of our clusters (all v14.2.8) we observe a very strange behavior: Over a time rgw_qactive perf is constantly growing, within 12h to 6k entries. [image: image.png] We observe this situation only on two of our clusters where the common thing is an app uploading a lot of files as multipart uploads via ssl. How can we debug this situation? How can we check what operations are in a queue or why a perf counter has not been decreased? Jacek -- Jacek Suchenia jacek.suchenia(a)gmail.com

2 2