October 2019 - Dev - lists.ceph.io

by Sage Weil

Hi everyone, CDM is next week, 0200 UTC Nov 6 in the APAC-friendly time slot (which is Wednesday for most of us). The current agenda: https://tracker.ceph.com/projects/ceph/wiki/CDM_06-NOV-2019 RADOS topics: - Josh: small objects. What can/should we do to make them more efficient, either just in BlueStore or in the OSD - Sage: update on the status/plans for deduplication. Myoungwon has been working on this for a while and has mostly been blocked by my slow reviews. Review current status and see if anyone is interested in helping out. - Crimson update. This will be the day after a face to face meeting next week in San Francisco after Seastar Summit. - Sage, Brad: telemetry and crash reports. Status, improvements. CephFS: - Patrick: MDS ephemeral pinning. See you next week! sage

3 years, 11 months

2
1
0 0

help on make-debs.sh error

by Rui Chang (Arm Technology China)

Hi, all These days I met errors when running make-debs.sh on Ubuntu Firstly I run build of ceph, and after this, I see the following folder and files are there: /root/ceph_auto/ceph/src/pybind/mgr/dashboard/frontend/node_modules Then I run make-debs.sh which use git clean to clean lots of file and folder which include Removing src/pybind/mgr/dashboard/frontend/node_modules/ And then make-debs.sh itself fails complaining it cannot find the folder and files that it deleted Error: Cannot find module '/root/ceph_auto/ceph/src/pybind/mgr/dashboard/frontend/node_modules/@angular/compiler-cli/node_modules/fsevents/install' Why it first delete some files and then try to use it, is this a bug? Or did I do something wrong? Regards, Rui IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

3 years, 11 months

1
0
0 0

is ceph master branch broken today ?

by Milind Changire

after running vstart cluster on my laptop, all ./bin/ceph commands are hung; I can also see that ceph-osd processes are still attached to the tty; is this expected ? -- Milind

3 years, 11 months

3
5
0 0

container tags

by Sage Weil

I noticed that ceph/ceph has tags like v14 v14.2 v14.2.2 v14.2.2-20190830 whereas ceph/daemon-base has tags like latest-master latest-nautilus latest-mimic but no version tags. Is there a reason for the difference? Should we be using ceph/ceph or ceph/daemon-base? What's the difference? That tags for ceph/ceph seem more intuitive/useful for users, although it seems like we really want both latest-* and v* tags. Thanks! sage

3 years, 11 months

3
6
0 0

RMDA Bug?

by Mason-Williams, Gabryel (DLSLtd,RAL,LSCI)

I am currently trying to run Ceph on RDMA, either RoCE 1 or 2. However, I am experiencing issues with this. When using Ceph on RDMA I experience issues where OSD’s will randomly become unreachable even if the cluster is left alone alone, it also is not properly talking over RDMA and using Ethernet when the config states it should as shown by the same results in the bench marking of the two setups. After reloading the cluster [cid:36020940-0085-40fc-bb5b-d91de6ace453] After 5m 9s the cluster went from being healthy to down. [cid:ed084bcc-0b97-44bd-9648-ce2e06859cd5] This problem even happens when running a bench mark test on the cluster, OSD’s will just fall over. Another curious issue is that it is not properly talking over RDMA as well and instead using the Ethernet. [cid:05e9dc68-075e-425d-b76b-ce7fa1d2f7a8] Next test: [cid:4183557e-b1da-41f3-afc3-f081b9fb4034] The config used for the RDMA is a so: [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+rdma ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using RDMA sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+rdma", sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT { "AsyncMessenger::RDMAWorker-1": { "tx_no_mem": 0, "tx_parital_mem": 0, "tx_failed_post": 0, "rx_no_registered_mem": 0, "tx_chunks": 9, "tx_bytes": 2529, "rx_chunks": 0, "rx_bytes": 0, "pending_sent_conns": 0 } } When running over Ethernet I have a completely stable system with the current benchmarks as so [cid:544ecbbc-10d9-43e6-ab2f-aa7c2bcd88c0] Config setup when using Ethernet is The Config setup when using Ethernet is [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+posix ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using async+posix sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+posix" sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT {} This clearly a issue with RDMA and not with the OSD's shown by the fact the system is completely fine over Ethernet and not with RDMA. Any guidance or ideas on how to approach this problem to make Ceph work with RDMA would be greatly appreciated. Regards Gabryel Mason-Williams, Placement Student Address: Diamond Light Source Ltd., Diamond House, Harwell Science & Innovation Campus, Didcot, Oxfordshire OX11 0DE Email: gabryel.mason-williams(a)diamond.ac.uk -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 years, 11 months

3
2
0 0

gorados - pure go client to connect RADOS cluster

by oshynsong＠163.com

There is requirement connecting to RADOS cluster by Golang, and the sub-project go-ceph(https://github.com/ceph/go-ceph) is the first solution. It uses cgo linking the librados library and works OK. I used go-ceph in one project, there are some problems annoying me: The connection created to the RADOS cluster must be closed explicitly, because golang runtime GC does not care about it. If your application holds a global connection and does not close, it will be a waste of resource of the RADOS cluster. Otherwise, you need to implement a connection pool for management, which increases complexity and needs a strong testing. librados is implemented with C/C++ and pthread thread model, and it heavily depends on the low-level messenger implementation. What I used in my project is the luminous version, and it will create 14 new threads when creating a connection to the RADOS cluster(use pstack on centos7). So the mixed threads used by the golang runtime and librados causes the problem here(https://stackoverflow.com/questions/47466139/there-are-many-threads-re… ). The golang program must depend on the librados when compiling and running, it decreases the maintainability compared with common static-built golang program. There are also some problems if you use go-ceph in your golang project, such as the messenger layer is not efficient as golang. There is every reason to implement the gorados - pure go client to connect RADOS cluster - to make ceph as a backend storage in production-level golang project directly. I make this idea as a fact, and provide the draft startup on github: https://github.com/oshynsong/gorados. It implements the cephx protocol to connect the monitor, and needs more development. Welcome to join me and I hope gorados as a sub-project of ceph!

3 years, 11 months

1
0
0 0

gorados - pure go client to connect RADOS cluster

by Oshyn Song

There is requirement connecting to RADOS cluster by Golang, and the sub-project go-ceph(https://github.com/ceph/go-ceph) is the first solution. It uses cgo linking the librados library and works OK. I used go-ceph in one project, there are some problems annoying me: 1. The connection created to the RADOS cluster must be closed explicitly, because golang runtime GC does not care about it. If your application holds a global connection and does not close, it will be a waste of resource of the RADOS cluster. Otherwise, you need to implement a connection pool for management, which increases complexity and needs a strong testing. 2. librados is implemented with C/C++ and pthread thread model, and it heavily depends on the low-level messenger implementation. What I used in my project is the luminous version, and it will create 14 new threads when creating a connection to the RADOS cluster(use pstack on centos7). So the mixed threads used by the golang runtime and librados causes the problem here <https://tracker.ceph.com/projects/ceph/wiki/Here>( https://stackoverflow.com/questions/47466139/there-are-many-threads-reserve… ). 3. The golang program must depend on the librados when compiling and running, it decreases the maintainability compared with common static-built golang program. There are also some problems if you use go-ceph in your golang project, such as the messenger layer is not efficient as golang. There is every reason to implement the gorados - pure go client to connect RADOS cluster - to make ceph as a backend storage in production-level golang project directly. I make this idea as a fact, and provide the draft startup on github: https://github.com/oshynsong/gorados. It implements the cephx protocol to connect the monitor, and needs more development. Welcome to join me and I hope gorados as a sub-project of ceph! -- Oshyn Song Software Engineer of Distributed Storage , Baidu.Inc, Beijing, China Sincerely!

3 years, 11 months

1
1
0 0

mimic and luminous baselines

by Yuri Weinstein

In preparation for QE validation I ran limited (10 jobs) baselines for luminous and mimic: http://pulpito.ceph.com/?branch=wip-yuri-luminous_10.22.19 http://pulpito.front.sepia.ceph.com/?branch=wip-yuri-mimic_10.22.19 Dev lead, pls scan thru those to address issues before we start more testing pls. There are issue possible py3 issues (https://tracker.ceph.com/issues/42292) and possible msd crashes (https://tracker.ceph.com/issues/42491) Thx YuriW

3 years, 11 months

1
0
0 0

Authentication failure at radosgw for presigned urls

by Biswajeet Patra

Hi, We recently faced an issue with radosgw authentication of presigned urls. The presigned url generated by client to download object fails at radosgw with http error 403 i.e SignatureDoesNotMatch. The radosgw computes the signature (*v2 signature in this case*) using the s3 specification https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html. In this process, a string_to_sign is created by concatenating selected elements from requests which is then authenticated using hmac to create the final signature. If this signature matches with the client signature, the authentication is successful or else it fails with SignatureDoesNotMatch error. As part of the StringToSign, the CanonicalizedAmzHeaders should only include headers that start with "x-amz-" and ignore other headers. But in the radosgw code, there are other meta prefixes that are checked against the http request headers and if matched are included in the CanonicalizedAmzHeaders to compute the final signature. For e.g, if a request header contains "HTTP_X_ACCOUNT" its selected by radosgw to include in amz_headers but the same will be ignored by AWS SDK as it does not start with "x-amz-". This will result in different signature computed by client and radosgw. Code Snippet: rgw_common.cc struct str_len meta_prefixes[] = { STR_LEN_ENTRY("HTTP_X_AMZ"), STR_LEN_ENTRY("HTTP_X_GOOG"), STR_LEN_ENTRY("HTTP_X_DHO"), STR_LEN_ENTRY("HTTP_X_RGW"), STR_LEN_ENTRY("HTTP_X_OBJECT"), STR_LEN_ENTRY("HTTP_X_CONTAINER"), STR_LEN_ENTRY("HTTP_X_ACCOUNT"), {NULL, 0} }; The method init_meta_info() which matches the above prefixes is called from RGWREST::preprocess() which is invoked for all s3 requests. It will be helpful to know as to why these prefixes that are not specified by AWS S3 comes in the path of s3 authentication. Was it added for swift use-cases only? If so, then why its included in rgw_common.cc? As a proposed fix, we can remove the highlighted meta prefixes that are not specified by aws from the s3 authentication path signature calculation. Let me know if you have any queries or solutions. Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

3 years, 11 months

1
2
0 0

PG undersized+peered and inactive

by Liu, Changcheng

Hi all, I'm using ceph-deploy tool to deploy single node cluster based on Ceph master branch(head id: bf09a04d2). The PGs are inactive and undersized. -bash-4.2$ ceph -s cluster: id: 93e5ff9c-1dee-4bd8-9b0a-318b917dfd8c health: HEALTH_WARN Reduced data availability: 4 pgs inactive Degraded data redundancy: 4 pgs undersized services: mon: 1 daemons, quorum rdmarhel0 (age 30m) mgr: rdmarhel0(active, since 29m) osd: 3 osds: 3 up (since 25m), 3 in (since 25m) data: pools: 1 pools, 4 pgs objects: 0 objects, 0 B usage: 3.0 GiB used, 834 GiB / 837 GiB avail pgs: 100.000% pgs not active 4 undersized+peered -bash-4.2$ Other info is attached. I apply below patch based on head-id:bf09a04d2. It's for Ceph/RDMA deployment. The above problem is hit whatever for RDMA-messenger deployment or Posix-TCP-messenger deployment. diff --git a/systemd/ceph-fuse@.service.in b/systemd/ceph-fuse@.service.in index d603042..ff2e907 100644 --- a/systemd/ceph-fuse@.service.in +++ b/systemd/ceph-fuse@.service.in @@ -12,6 +12,7 @@ ExecStart=/usr/bin/ceph-fuse -f --cluster ${CLUSTER} %I LockPersonality=true MemoryDenyWriteExecute=true NoNewPrivileges=true +LimitMEMLOCK=infinity # ceph-fuse requires access to /dev fuse device PrivateDevices=no ProtectControlGroups=true diff --git a/systemd/ceph-mds@.service.in b/systemd/ceph-mds@.service.in index 39a2e63..0e58dfe 100644 --- a/systemd/ceph-mds@.service.in +++ b/systemd/ceph-mds@.service.in @@ -14,7 +14,8 @@ ExecReload=/bin/kill -HUP $MAINPID LockPersonality=true MemoryDenyWriteExecute=true NoNewPrivileges=true -PrivateDevices=yes +LimitMEMLOCK=infinity +PrivateDevices=no ProtectControlGroups=true ProtectHome=true ProtectKernelModules=true diff --git a/systemd/ceph-mgr@.service.in b/systemd/ceph-mgr@.service.in index c98f637..682c7ec 100644 --- a/systemd/ceph-mgr@.service.in +++ b/systemd/ceph-mgr@.service.in @@ -18,7 +18,8 @@ LockPersonality=true MemoryDenyWriteExecute=false NoNewPrivileges=true -PrivateDevices=yes +LimitMEMLOCK=infinity +PrivateDevices=no ProtectControlGroups=true ProtectHome=true ProtectKernelModules=true diff --git a/systemd/ceph-mon@.service.in b/systemd/ceph-mon@.service.in index c95fcab..51854fa 100644 --- a/systemd/ceph-mon@.service.in +++ b/systemd/ceph-mon@.service.in @@ -21,7 +21,8 @@ LockPersonality=true MemoryDenyWriteExecute=true # Need NewPrivileges via `sudo smartctl` NoNewPrivileges=false -PrivateDevices=yes +LimitMEMLOCK=infinity +PrivateDevices=no ProtectControlGroups=true ProtectHome=true ProtectKernelModules=true diff --git a/systemd/ceph-osd@.service.in b/systemd/ceph-osd@.service.in index 1b5c9c8..06c20d7 100644 --- a/systemd/ceph-osd@.service.in +++ b/systemd/ceph-osd@.service.in @@ -16,6 +16,8 @@ LockPersonality=true MemoryDenyWriteExecute=true # Need NewPrivileges via `sudo smartctl` NoNewPrivileges=false +LimitMEMLOCK=infinity +PrivateDevices=no ProtectControlGroups=true ProtectHome=true ProtectKernelModules=true diff --git a/systemd/ceph-radosgw@.service.in b/systemd/ceph-radosgw@.service.in index 7e3ddf6..fe1a6b9 100644 --- a/systemd/ceph-radosgw@.service.in +++ b/systemd/ceph-radosgw@.service.in @@ -13,7 +13,8 @@ ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i --setuser ce LockPersonality=true MemoryDenyWriteExecute=true NoNewPrivileges=true -PrivateDevices=yes +LimitMEMLOCK=infinity +PrivateDevices=no ProtectControlGroups=true ProtectHome=true ProtectKernelModules=true diff --git a/systemd/ceph-volume@.service b/systemd/ceph-volume@.service index c21002c..e2d1f67 100644 --- a/systemd/ceph-volume@.service +++ b/systemd/ceph-volume@.service @@ -9,6 +9,7 @@ KillMode=none Environment=CEPH_VOLUME_TIMEOUT=10000 ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT /usr/sbin/ceph-volume-systemd %i' TimeoutSec=0 +LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target -- 1.8.3.1

3 years, 11 months

2
4
0 0

2023

2022

2021

2020

2019

Dev October 2019