- ceph-users - lists.ceph.io

Why can’t I get money off Cash app without card?
by david william 29 Aug '20

29 Aug '20

Cash app comes with the Visa debit card to make your transaction reliable as well as safe. Here we describe you to resolve your query get money off Cash app without card. Actually, if you lost your card or you didn’t get a Cash card yet, then you have to make a request for getting a Cash card.https://www.cashapp-customerservice.com/blog/get-money-cash-app-withou…

1 0

Re: Is it possible to mount a cephfs within a container?
by steven prothero 29 Aug '20

29 Aug '20

Hello, octopus 15.2.4 just as a test, I put my OSDs each inside of a LXD container. Set up cephFS and mounted it inside a LXD container and it works.

2 1

The most trusted Quickbooks technical support number
by david beckam 29 Aug '20

29 Aug '20

Quickbooks has changed the lives of thousands of its users by taking up the responsibility of bookkeeping for them. Keeping accounts becomes a breeze with business owners easily meeting tax compliance requirements. If you are new to Quickbooks, it is natural for you to face some difficulties during the learning curve. However, there is no need to worry as you can use our popular and trusted Quickbooks technical support number. Dial 1-877-916-7666, the toll free number of our tech support agency, and talk to our qualified professional. He will help you to overcome your Quickbooks related issues without any delay. https://www.assistanceforall.com/services/quickbook-support/

1 0

Saints redeemed secondary certain vs. superior-run Rams
by montgomeryqian＠yhwebmail.com 29 Aug '20

29 Aug '20

METAIRIE, La. Marshon Lattimore and the Saints secondary no for a longer time appear or strong haunted by way of the Minnesota Wonder touchdown go that stunningly finished their playoffs 1 12 months back.We use t anxiety. We wear t do none of that, Lattimore mentioned. We transform very well.Lattimore sponsored that up with his late-recreation interception towards Philadelphia ultimate weekend his instant of the activity securing the Saints holiday vacation in the direction of Sunday NFC name activity. At the moment the speculate is no matter whether a Saints defensive backfield that rated 29th in opposition to the go this time is penned towards do what it normally takes in direction of gradual down the strong Los Angeles Rams with the Tremendous Bowl bid upon the line.We re geared up, stated Saints cornerback Eli Apple, who was introduced within by means of exchange towards assistance shore up a Refreshing Orleans move safety that was shaky early this time.We ve accurately received in the direction of carry on toward be serious inside of such conferences with our thoughts, anyone choose upon the identical webpage and studying, simply working with it upon the industry and transferring difficult.Fresh new Orleans gave up an ordinary of 268.9 yards for every sport this year, still Lattimore reported element of their unflattering position as a go safety stems against a couple of of undesirable performances early inside of the period, starting up with a 48-40 decline towards Tampa Bay in just 7 days 1 and a 43-37 7 days 3 additional time victory at Atlanta <a href="https://www.ramsgearfanstore.com/Sam_Shields_Jersey-454">https://www.ramsgearfanstore.com/Sam_Shields_Jersey-454</a>.We experienced a substantial gap in the direction of climb out of at the starting off of the time. We use t incredibly pressure in excess of that whilst, Lattimore mentioned. We re successful. As a result, we re Terrific.Lattimore explained he additionally includes better separately because having difficulties some at the starting off of the time and may possibly be enjoying his excellent already.I m only manufacturing performs that occur my direction <a href="https://www.ramsgearfanstore.com/John_Kelly_Jersey-314">John Kelly Jersey</a>, Lattimore stated. I comprise in direction of be the one particular in the direction of create individuals performs. It a greater place immediately by now. Therefore, I comprise in the direction of appear up and do the activity for my workers.Saints nickel back again P.J. Williams explained the Saints terrible rating versus the go is deceptive mainly because Fresh Orleans rated initially utmost of the year towards the work just before completing moment. In the meantime, Drew Brees and Fresh new Orleans significant-driven offense include staked the Saints towards a good deal of prospects.And Although groups drop at the rear of, they are inclined in direction of operate significantly less since it requires much too considerably season off the clock.They can t work the ball. Your self acquired toward move the ball. On your own re participating in towards guiding, Williams claimed. Consequently a employees could possibly receive 300 or consequently passing yards, nevertheless at the finish of the working day, they re attempting towards arrive back again.Williams reported the Saints are further nervous with opposing QBs completion value than how plenty of yards they let <a href="https://www.ramsgearfanstore.com/Earnest_Edwards_Jersey-2286">https://www.ramsgearfanstore.com/Earnest_Edwards_Jersey-2286</a>. Upon that entrance, Contemporary Orleans rated rather superior, at 23rd, letting a 66.5 per cent completion value.Rams quarterback Jared Goff finished 28 of 40 passes for 391 yards and a few TDs the very last year he performed in just the Superdome.Irrespective of whether he ll be as effective inside the rematch relies upon upon how Los Angeles establishes toward assault the Saints, and how Clean Orleans responds.The Rams constantly want the working recreation, and rushed 48 days for 273 yards in just a playoff victory above Dallas previous 7 days <a href="https://www.ramsgearfanstore.com/Eric_Banks_Jersey-2331">https://www.ramsgearfanstore.com/Eric_Banks_Jersey-2331</a>.Yet Los Angeles mainly deserted the work throughout its midseason convention with the Saints immediately after Fresh new Orleans raced in direction of a 35-14 contribute within just the instant quarter. The Rams wound up dashing specifically 19 periods in just a 45-35 decline.Lattimore reported if the Rams deserted the work once again, Fresh Orleans secondary will comprise toward be written for large doses of receivers Brandin Chefs, whose pace results in him an elite deep hazard, and Robert Woods, who strains up all previously mentioned the business.And if the video game is conclusion, it could possibly be up toward the Saints secondary in the direction of conclusion out a further significant-stakes affair.Very last 12 months, that situation finished with the Vikings Stefon Diggs manufacturing a jumping capture close to the sideline and working absolutely free toward the stop zone as season expired when protection Marcus Williams disregarded a deal with and took out teammate Ken Crawley inside of the course of action <a href="https://www.ramsgearfanstore.com/Greg_Gaines_Jersey-1122">https://www.ramsgearfanstore.com/Greg_Gaines_Jersey-1122</a>.This calendar year, the Saints secondary consists of shut out a person playoff victory and seems guaranteed it will increase toward that situation once more, if necessary.We ve finished a best endeavor completing video games, creating modifications, taking part in quick, bodily, developing ideal performs upon the ball, exactly received toward continue to keep it transferring, Apple explained. One thing we normally communicate pertaining to is remaining online games. We require that upon our shoulders and that worry that arrives with it.

1 0

In opposition to large atop Kilimanjaro, Haloti Ngata retires
by Montgomeryqian＠yhwebmail.com 29 Aug '20

29 Aug '20

Haloti Ngata found out a one of a kind course towards announce his retirement towards the NFL versus atop Mount Kilimanjaro.The 35-12 months-aged Ngata, who performed with the Eagles past year immediately after a few yrs with the Lions, was at his perfect with Baltimore. He produced 2 All-Professional groups and 5 Qualified Bowls with the Ravens inside of 9 seasons as soon as getting a to start with-spherical draft decide on inside of 2006 [url=https://www.ravensgearfanstore.com/Broderick_Washington_Jr_Jersey-1586]… Washington Jr. Jersey[/url]. He gained a Tremendous Bowl with them within just February 2013.The defensive take care of printed his retirement concept Monday upon Instagram. Accurately a male track record upon greatest of the environment with a centre complete of gratitude, Ngata wrote. Thank on your own Lord for making it possible for me engage in the sport I appreciate for 13 memorable many years. I m retiring upon greatest. I may possibly be completed taking part in soccer, yet I m keeping restricted in direction of the friendships, recollections and knowledge I ve obtained together the course. No one who at any time gave his most straightforward regretted it. Going for walks absent with no regrets, specifically rest inside of being familiar with I gave it my all and experienced a helluva whole lot of enjoyable undertaking it.Stats have been not the evaluate for Ngata, who tied up blockers upon the inner and freed avid gamers these as Ray Lewis and Terrell Suggs in the direction of do their harm for the Ravens [url=https://www.ravensgearfanstore.com/Alvin_Jones_Jersey-979]https://www.r…] He way too includes been praised in the course of the a long time for his management abilities.

1 0

rados client connection to cluster timeout and debugging.
by Daniel Mezentsev 28 Aug '20

28 Aug '20

Hi all, I continue to work on my rados client, implementing error handling and trying to reconnect to cluster if it's not available. Here is my tests. Client is not connected to network, so cluster is not available, i configured client_mount_timeout = 0.5 (which is 5 seconds) and getting reasonable error - 110, Connection timed out. Then client is doing second and third attempt and in these cases error is a bit strange - 115, Operation now in progress. So failed rados_connect didn't restore state of connection - bug ? I was trying to enable debug: "log file" "./client.log" "log to file" "true" "debug ms" "10/10" "debug rados" "10/10" "debug objecter" "10/10" "debug monc" "10/10" "debug client" "10/10" But no luck to get anything in the log file. Running ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) under archlinux. Help is more then welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

1 1

Re: OSDs get full with bluestore logs
by Janne Johansson 28 Aug '20

28 Aug '20

Den fre 28 aug. 2020 kl 11:47 skrev Khodayar Doustar <khodayard(a)gmail.com>: > I've actually destroyed the cluster and a new one installed. > I've just changed the installation method and version. I've used > ceph-ansible this time and installed Nautilus. > The cluster worked fine with the same hardware. > Yes Janne, you are right that it had very small disks (9X20GB disks, 3 for > each node) but there was no problem with Nautilus. > > I just don't think anyone finds it useful to spend time figuring out the lowest possible limits for each release, for each type of OSD storage. So 10G worked at some point, 20 at another but if you are serious about ceph, make the disks thin provisioned and larger by a huge margin (say 100G) and let it grow to the size it needs for your tests. If the limit is 22 or 23.7 won't matter then and your test will run through. -- May the most significant bit of your life be positive.

1 0

How to change the pg numbers
by norman 28 Aug '20

28 Aug '20

Hi guys, I have a rbd pool pg_num 2048, I want to change it to 4096, how can I do this? If I change it directly to 4096, it may cause the client slow requests, What the better step size should be? Thanks, Kern

6 12

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Zhenshi Zhou 28 Aug '20

28 Aug '20

In my deployment I part the disk for wal and db seperately as I can assign the size manually. For example, I assign every two partitions of nvme with 30G and 2G, for each osd's wal and db. Tony Liu <tonyliu0592(a)hotmail.com> 于2020年8月28日周五上午1:53写道： > How's WAL utilize disk when it shares the same device with DB? > Say device size 50G, 100G, 200G, they are no difference to DB > because DB will take 30G anyways. Does it make any difference > to WAL? > > Thanks! > Tony > > -----Original Message----- > > From: Zhenshi Zhou <deaderzzs(a)gmail.com> > > Sent: Wednesday, August 26, 2020 11:16 PM > > To: Tony Liu <tonyliu0592(a)hotmail.com> > > Cc: Anthony D'Atri <anthony.datri(a)gmail.com>; ceph-users(a)ceph.io > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL and DB on > > SSD > > > > Official document says that you should allocate 4% of the slow device > > space for block.db. > > > > But the main problem is that Bluestore uses RocksDB and RocksDB puts a > > file on the fast device only if it thinks that the whole layer will fit > > there. > > > > As for RocksDB, L1 is about 300M, L2 is about 3G, L3 is near 30G, and L4 > > is about 300G. > > For instance, RocksDB puts L2 files to block.db only if it’s at least 3G > > there. > > As a result, 30G is a acceptable value. > > > > Tony Liu <tonyliu0592(a)hotmail.com <mailto:tonyliu0592@hotmail.com> > > > 于2020年8月25日周二上午10:49写道： > > > > > > > -----Original Message----- > > > From: Anthony D'Atri <anthony.datri(a)gmail.com > > <mailto:anthony.datri@gmail.com> > > > > Sent: Monday, August 24, 2020 7:30 PM > > > To: Tony Liu <tonyliu0592(a)hotmail.com > > <mailto:tonyliu0592@hotmail.com> > > > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL > > and DB on > > > SSD > > > > > > Why such small HDDs? Kinda not worth the drive bays and power, > > instead > > > of the complexity of putting WAL+DB on a shared SSD, might you > > have been > > > able to just buy SSDs and not split? ymmv. > > > > 2TB is for testing, it will bump up to 10TB for production. > > > > > The limit is a function of the way the DB levels work, it’s not > > > intentional. > > > > > > WAL by default takes a fixed size, like 512 MB or something. > > > > > > 64 GB is a reasonable size, it accomodates the WAL and allows > > space for > > > DB compaction without overflowing. > > > > For each 10TB HDD, what's the recommended DB device size for both > > DB and WAL? The doc recommends 1% - 4%, meaning 100GB - 400GB for > > each 10TB HDD. But given the WAL data size and DB data size, I am > > not sure if that 100GB - 400GB will be used efficiently. > > > > > With this commit the situation should be improved, though you > > don’t > > > mention what release you’re running > > > > > > https://github.com/ceph/ceph/pull/29687 > > > > I am using ceph version 15.2.4 octopus (stable). > > > > Thanks! > > Tony > > > > > >>> I don't need to create > > > >>> WAL device, just primary on HDD and DB on SSD, and WAL will > > be using > > > >>> DB device cause it's faster. Is that correct? > > > >> > > > >> Yes. > > > >> > > > >> > > > >> But be aware that the DB sizes are limited to 3GB, 30GB and > > 300GB. > > > >> Anything less than those sizes will have a lot of untilised > > space, > > > >> e.g a 20GB device will only utilise 3GB. > > > > > > > > I have 1 480GB SSD and 7 2TB HDDs. 7 LVs are created on SSD, > > each is > > > > about 64GB, for 7 OSDs. > > > > > > > > Since it's shared by DB and WAL, DB will take 30GB and WAL will > > take > > > > the rest 34GB. Is that correct? > > > > > > > > Is that size of DB and WAL good for 2TB HDD (block store and > > object > > > > store cases)? > > > > > > > > Could you share a bit more about the intention of such limit? > > > > > > > > > > > > Thanks! > > > > Tony > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph- > > users(a)ceph.io> To unsubscribe send an > > > > email to ceph-users-leave(a)ceph.io <mailto:ceph-users- > > leave(a)ceph.io> > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph- > > users(a)ceph.io> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > <mailto:ceph-users-leave@ceph.io> > > > >

2 1

Fwd: Ceph Upgrade Issue - Luminous to Nautilus (14.2.11 ) using ceph-ansible
by Suresh Rama 28 Aug '20

28 Aug '20

Hi All, We encountered an issue while upgrading our Ceph cluster from Luminous 12.2.12 to Nautilus 14.2.11. We used https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-mimic-o… and ceph-ansible to upgrade the cluster. We use HDD for data and NVME for WAL and DB. *Cluster Background:* HP DL360 24 x 3.6T SATA 2x1.6T NVME for Journal osd_scenario: non-collocated current version: Luminous 12.2.12 & 12.2.5 type: bluestore The upgrade went well for MONs (though I had to overcome the systemd masking issues). While testing OSD upgrade with one OSD node, we encountered issue with OSD daemon failing quickly after startup. After comparing and checking the block devices mapping, everything looks fine. The nodes was up for more than 700+ days and then I decided to do a clean reboot. After that noticed the mount points are completely missing and also ceph-disk is no longer part of nautilus. Had to manually mount the partitions after checking disk partitions and whoami information. After manually mounting the osd.108, now it's throwing permission error which I'm still reviewing (bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Permission denied). Enclosed the log of the OSD for full review - https://pastebin.com/7k0xBfDV. *Questions*: What could have went wrong here and how can we fix this? Do we need to migrate the Luminous cluster from ceph-disk to ceph-volume before attempting upgrade or any other best practice can be followed? What's the best upgrade method using ceph-ansible to move from Luminous to Nautilus? Manual upgrade of Ceph-ansible? Started thinking now Octopus release which uses container, what is the best transition path for long run? We don't want to destroy and rebuild the entire cluster but we can do one node at a time but that would be a very lengthy process for 2500+ systems of 13 clusters. Looking for help and expert comments on the transition path. Any help would be greatly appreciated. 2020-08-27 14:41:01.132 7f0e0ebf2c00 0 bdev(0xb7e2a80 /var/lib/ceph/osd/ceph-108/block.wal) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-108/block.wal failed: (22) Invalid argument 2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bdev(0xb7e2a80 /var/lib/ceph/osd/ceph-108/block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard supported 2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-108/block.wal size 1 GiB 2020-08-27 14:41:01.132 7f0e0ebf2c00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-08-27 14:41:01.132 7f0e0ebf2c00 -1 rocksdb: Invalid argument: Can't parse option compaction_threads 2020-08-27 14:41:01.136 7f0e0ebf2c00 -1 /build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_upgrade_super()' thread 7f0e0ebf2c00 time 2020-08-27 14:41:01.135973 /build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 10249: FAILED ceph_assert(ondisk_format > 0) ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846368] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846543] 3: (BlueStore::_upgrade_super()+0x4b6) [0xd62346] 4: (BlueStore::_mount(bool, bool)+0x592) [0xdb0b52] 5: (OSD::init()+0x3f3) [0x8f5483] 6: (main()+0x27e2) [0x84c462] 7: (__libc_start_main()+0xf0) [0x7f0e0bda3830] 8: (_start()+0x29) [0x880389] Journalctl -xu log ================ Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 osd.108 0 OSD:init: unable to mount object store Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 ** ERROR: osd init failed: (13) Permission denied Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Main process exited, code=exited, status=1/FAILURE Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Unit entered failed state. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Failed with result 'exit-code'. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Service hold-off time over, scheduling restart. Aug 27 20:18:39 pistore-as-b03 systemd[1]: Stopped Ceph object storage daemon osd.108. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Start request repeated too quickly. Aug 27 20:18:39 pistore-as-b03 systemd[1]: Failed to start Ceph object storage daemon osd.108. Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' -- Regards, Suresh Note: Resent as the size was big and waiting on moderator approval

2 1