hey Gal and Eric, in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release?
Hi, Just to expand a bit. For development purposes. As Casey says: + EPEL9 has libarrow-9 If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module) And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing (I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.) I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.) On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Kaleb
hey Gal and Eric, nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule? an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that? On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Kaleb
bump! On Tue, Jan 23, 2024 at 10:38 AM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule?
an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that?
On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Kaleb
i had checked version 10 on fedora 38 (the standalone application) it seems ok beyond that(version 10) some of arrow API's are modified (it breaks the build) ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Monday, February 5, 2024 6:57 PM To: Eric Ivancich <ivancich@ibm.com>; Gal Salomon <gsalomon@ibm.com> Cc: dev@ceph.io <dev@ceph.io>; Gal Salomon <gsalomon@redhat.com>; Eric Ivancich <ivancich@redhat.com>; Kaleb Keithley <kkeithle@redhat.com> Subject: [EXTERNAL] Re: arrow submodule version bump! On Tue, Jan 23, 2024 at 10:38 AM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule?
an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that?
On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Kaleb
I’m happy to update my code to track a later version. What version would you prefer to go to, Gal? Eric (he/him) On Feb 5, 2024, at 12:05 PM, Gal Salomon <gsalomon@ibm.com> wrote: i had checked version 10 on fedora 38 (the standalone application) it seems ok beyond that(version 10) some of arrow API's are modified (it breaks the build) ________________________________ From: Casey Bodley <cbodley@redhat.com<mailto:cbodley@redhat.com>> Sent: Monday, February 5, 2024 6:57 PM To: Eric Ivancich <ivancich@ibm.com<mailto:ivancich@ibm.com>>; Gal Salomon <gsalomon@ibm.com<mailto:gsalomon@ibm.com>> Cc: dev@ceph.io<mailto:dev@ceph.io> <dev@ceph.io<mailto:dev@ceph.io>>; Gal Salomon <gsalomon@redhat.com<mailto:gsalomon@redhat.com>>; Eric Ivancich <ivancich@redhat.com<mailto:ivancich@redhat.com>>; Kaleb Keithley <kkeithle@redhat.com<mailto:kkeithle@redhat.com>> Subject: [EXTERNAL] Re: arrow submodule version bump! On Tue, Jan 23, 2024 at 10:38 AM Casey Bodley <cbodley@redhat.com<mailto:cbodley@redhat.com>> wrote:
hey Gal and Eric,
nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule?
an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that?
On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com<mailto:kkeithle@redhat.com>> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com<mailto:cbodley@redhat.com>> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
--
Kaleb
we discussed this in the rgw standup. version 10 sounds the least risky and easiest to validate for the squid release, since there's no API breakage. Gal was going to do some smoke testing with s3select before opening a pull request On Tue, Feb 6, 2024 at 11:38 AM Eric Ivancich <ivancich@ibm.com> wrote:
I’m happy to update my code to track a later version. What version would you prefer to go to, Gal?
Eric (he/him)
On Feb 5, 2024, at 12:05 PM, Gal Salomon <gsalomon@ibm.com> wrote:
i had checked version 10 on fedora 38 (the standalone application) it seems ok
beyond that(version 10) some of arrow API's are modified (it breaks the build)
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Monday, February 5, 2024 6:57 PM To: Eric Ivancich <ivancich@ibm.com>; Gal Salomon <gsalomon@ibm.com> Cc: dev@ceph.io <dev@ceph.io>; Gal Salomon <gsalomon@redhat.com>; Eric Ivancich <ivancich@redhat.com>; Kaleb Keithley <kkeithle@redhat.com> Subject: [EXTERNAL] Re: arrow submodule version
bump!
On Tue, Jan 23, 2024 at 10:38 AM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule?
an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that?
On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Kaleb
i had tested it with fedora38/arrow-10 (docker build = https://pad.ceph.com/p/r.53a4cf1228c7da4ef344be8802397240 ) and also with ubunto20.04/arrow-10 (docker build = https://pad.ceph.com/p/r.0823d1a8ec97699ccecc274f4646c049) p.s we can try move forward, eventually we will need that. ________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Tuesday, February 6, 2024 6:54 PM To: Eric Ivancich <ivancich@ibm.com> Cc: Gal Salomon <gsalomon@ibm.com>; dev@ceph.io <dev@ceph.io>; Gal Salomon <gsalomon@redhat.com>; Kaleb Keithley <kkeithle@redhat.com> Subject: [EXTERNAL] Re: arrow submodule version we discussed this in the rgw standup. version 10 sounds the least risky and easiest to validate for the squid release, since there's no API breakage. Gal was going to do some smoke testing with s3select before opening a pull request On Tue, Feb 6, 2024 at 11:38 AM Eric Ivancich <ivancich@ibm.com> wrote:
I’m happy to update my code to track a later version. What version would you prefer to go to, Gal?
Eric (he/him)
On Feb 5, 2024, at 12:05 PM, Gal Salomon <gsalomon@ibm.com> wrote:
i had checked version 10 on fedora 38 (the standalone application) it seems ok
beyond that(version 10) some of arrow API's are modified (it breaks the build)
________________________________ From: Casey Bodley <cbodley@redhat.com> Sent: Monday, February 5, 2024 6:57 PM To: Eric Ivancich <ivancich@ibm.com>; Gal Salomon <gsalomon@ibm.com> Cc: dev@ceph.io <dev@ceph.io>; Gal Salomon <gsalomon@redhat.com>; Eric Ivancich <ivancich@redhat.com>; Kaleb Keithley <kkeithle@redhat.com> Subject: [EXTERNAL] Re: arrow submodule version
bump!
On Tue, Jan 23, 2024 at 10:38 AM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
nothing has changed since reef, and we're still using the ancient arrow version 6.0.1 for this submodule. can we please choose a more recent version and pull up the submodule?
an issue was recently raised in https://github.com/ceph/ceph/pull/52629 where the arrow build fails due to https://github.com/apache/arrow/issues/15281 which was fixed in arrow 14.0. can we use that?
On Thu, Feb 9, 2023 at 3:09 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
Hi,
Just to expand a bit. For development purposes.
As Casey says: + EPEL9 has libarrow-9
If you're developing on RHEL8 or CentOS 8: + EPEL8 has libarrow-8 (requires installing subversion-devel module and utf8proc-devel from the module)
And if you're developing on Fedora: + Fedora 38 and f39/rawhide are getting libarrow-11 now + Fedora 37 has libarrow-9 + Fedora 36 has libarrow-8 + Fedora 35 (EOL) has nothing
(I originally thought 9, 10, and 11 were released closer together than they actually are. Although 6-8 weeks between them is still a lot of churn IMO. Independent of that, there's still an opportunity to leave Fedora 38 on libarrow-10 if it makes sense to have libarrow-10 available in a Fedora release.)
I dare say "we" could build ORC and Arrow .debs for Ubuntu in a Launchpad PPA. Would that be useful or usable? I.e. could teuthology consume packages from a PPA? (Pretty sure I don't want to own that though. Def don't want to be the only owner of it.)
On Thu, Feb 9, 2023 at 2:17 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Gal and Eric,
in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release
as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Kaleb
participants (4)
-
Casey Bodley
-
Eric Ivancich
-
Gal Salomon
-
Kaleb Keithley