Den tors 25 jan. 2024 kl 03:05 skrev Henry lol <pub.virtualization@gmail.com>:

Do you mean object location (osds) is initially calculated only using its name and crushmap, and then the result is reprocessed with the map of the PGs?

and I'm still skeptical about computation on the client-side. is it possible to obtain object location without computation on the client because ceph-mon already updates that information to PG map?

The client should not need to contact the mon for each object access and every client can't have a complete list of millions of objects in the cluster, so it does client-side computations. The mon connection will more or less only require new updates if/when OSDs change weight or goes in/out. This way, clients can run on "autopilot" even if all mons are down, as long as OSD states don't change. -- May the most significant bit of your life be positive.

Den tors 25 jan. 2024 kl 11:57 skrev Henry lol <pub.virtualization@gmail.com>:

It's reasonable enough. actually, I expected the client to have just? thousands of "PG-to-OSDs" mappings.

Yes, but filename to PG is done with a pseudorandom algo.

Nevertheless, it’s so heavy that the client calculates location on demand, right?

Yes, and I guess the client has some kind of algorithm that makes it possible to know that PG 1.a4 should be on OSD 4, 93, 44 but also if 4 is missing, the next candidate would be 51, if 93 isn't up either then 66 would be the next logical OSD to contact for that copy and so on. Since all parts (client, mons, OSDs) have the same code, when osd 4 dies, 51 knows it needs to get a copy from either 93 or 44 and as soon as that copy is made, the PG will stop being active+degraded but might possibly be active+remapped, since it knows it wants to go back to OSD 4 if it comes back with the same size again.

if the client with the outdated map sends a request to the wrong OSD, then does the OSD handle it somehow through redirection or something?

I think it would get told it has the wrong osdmap.

Lastly, not only CRUSH map but also other factors like storage usage are considered when doing CRUSH? because it seems that the target OSD set isn’t deterministic given only it.

It doesn't take OSD usage into consideration except at creation time or OSD in/out/reweighing (or manual displacements with upmap and so forth), so this is why "ceph df" will tell you a pool has X free space, where X is "smallest free space on the OSDs on which this pool lies, times the number of OSDs". Given the pseudorandom placement of objects to PGs, there is nothing to prevent you from having the worst luck ever and all the objects you create end up on the OSD with least free space. -- May the most significant bit of your life be positive.

On 1/25/24 13:32, Janne Johansson wrote:

It doesn't take OSD usage into consideration except at creation time or OSD in/out/reweighing (or manual displacements with upmap and so forth), so this is why "ceph df" will tell you a pool has X free space, where X is "smallest free space on the OSDs on which this pool lies, times the number of OSDs". Given the pseudorandom placement of objects to PGs, there is nothing to prevent you from having the worst luck ever and all the objects you create end up on the OSD with least free space.

This is why you need a decent amount of PGs, to not run into statistical edge cases. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

...

...

forth), so this is why "ceph df" will tell you a pool has X free space, where X is "smallest free space on the OSDs on which this pool lies, times the number of OSDs".

To be even more precise, this depends on the failure domain. With the typical "rack" failure domain, say you use 3x replication and have 3 racks, you'll be limited to the capacity of the smallest rack. If you have more racks than failure domains, though, you are less affected racks that vary somewhat in CRUSH weight. With respect to OSDs, the above is still true, which is one reason we have the balancer module. Say your OSDs are on average 50% full but you have one that is 70% full. The most-full outlier will limit the reported available space. The available space for each pool is also a function of the replication strategy -- replication vs EC as well as the prevailing full ratio setting.

...

...

Given the pseudorandom placement of objects to PGs, there is nothing to prevent you from having the worst luck ever and all the objects you create end up on the OSD with least free space.

This is why you need a decent amount of PGs, to not run into statistical edge cases.

Yes, just take the experiment to someone with one PG only, then it can only fill one OSD. Someone with a pool with only 2 PGs could at the very best case only fill two and so on. If you have 100+ PGs per OSD, the chances for many files to end up only on a few PGs becomes very small.

Indeed, a healthy number of PG shards per OSD is important as well for this reason. I use an analogy of filling a 55 gallon drum with sportsballs. You can fit maybe two beach balls in there with a ton of air space, but you could fit thousands of pingpong balls in there with a lot less air space. Having a power of 2 number of PGs per pool also helps with uniform distribution -- the description of why this is the case is a bit abstruse so I'll spare the list, but enquiring minds can read chapter 8 ;)

and every client can't have a complete list of millions of objects in the cluster, so it does client-side computations.

This is one reason we have PGs -- so that there's a manageable number of things to juggle, while not being so few as to run into statistical and other imbalances.

-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io