Python 2 exodus is happening now [Was: Re: [sepia] Transition to Python 3]
On Wed, Apr 18, 2018 at 6:27 AM Nathan Cutler <ncutler@suse.cz> wrote:
That would be at odds to what Nathan is suggesting though, which is a hard change to Python 3.
Hm, not sure what hard/soft means in this context. For any given script, either it runs with Python 3, or it doesn't. And this is determined by the shebang. (Unless the shebang is omitted, of course.)
I was very surprised to find out that, in SLES and openSUSE, the symlink /usr/bin/python -> /usr/bin/python2 will not be changed even when the migration of the underlying distro to Python 3 is complete.
But then my colleagues explained why that is, and I "saw the light". Since every single script in the distro has to be audited for Python 3 compatibility, anyway, it makes sense to have the shebang be an explicit declaration of said compatibility.
By retaining the symlink at it is, all scripts start out the migration process with an explicit declaration that they are compatible with Python 2. Compatibility with Python 3 is signalled not by saying "it's OK with Python 3, we tried it". It's signalled by changing the shebang.
And this isn't unique to SUSE. Fedora is treating the shebang in the same way, apparently. [2]
Seems that if you only have python3 installed in Fedora31 this is *not* the case. # python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3 See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package "there is no /usr/bin/python" So the two distros are quite divergent in their approach apparently?
It may be true that a given script is fine with Python 3, but as long as the shebang says "python" (i.e. python2), there's no way to really find out, is there? (Barring things like Josh's suggestion of changing the shebang on the fly via a teuthology task/workunit, which is fine if we decide we need a transition period, which it looks like we will.)
Nathan
[1] https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/ce... [2] https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin.... _______________________________________________ Sepia mailing list Sepia@lists.ceph.com http://lists.ceph.com/listinfo.cgi/sepia-ceph.com
-- Cheers, Brad
On Sun, Nov 17, 2019 at 8:09 PM Brad Hubbard <bhubbard@redhat.com> wrote:
On Wed, Apr 18, 2018 at 6:27 AM Nathan Cutler <ncutler@suse.cz> wrote:
That would be at odds to what Nathan is suggesting though, which is a hard change to Python 3.
Hm, not sure what hard/soft means in this context. For any given script, either it runs with Python 3, or it doesn't. And this is determined by the shebang. (Unless the shebang is omitted, of course.)
Apologies if this has already been mentioned, but FYI, Fedora (and RHEL and CentOS by extension) do not allow ambiguous python shebangs – they _require_ explicit versioned shebangs. I presume that is what is meant by a hard change to Python3. things like: #!/usr/bin/env python #!/usr/bin/python are not permitted. (Also not #!/usr/bin/env $anything) Shebangs – if there is one – in .py files must be either #!/usr/bin/python3 or #!/usr/bin/python2 AFAIK Fedora is enforcing it, kinda. Package builds in fedora run rpmlint which checks for these kinds of things, although I'm not sure this is a test yet or if there is a test that it scores high enough to fail a build. (Ditto for package builds in the OpenSUSE OBS, i.e. they also run rpmlint, although I haven't noticed python shebang tests there either.)
I was very surprised to find out that, in SLES and openSUSE, the symlink /usr/bin/python -> /usr/bin/python2 will not be changed even when the migration of the underlying distro to Python 3 is complete.
But then my colleagues explained why that is, and I "saw the light". Since every single script in the distro has to be audited for Python 3 compatibility, anyway, it makes sense to have the shebang be an explicit declaration of said compatibility.
By retaining the symlink at it is, all scripts start out the migration process with an explicit declaration that they are compatible with Python 2. Compatibility with Python 3 is signalled not by saying "it's OK with Python 3, we tried it". It's signalled by changing the shebang.
And this isn't unique to SUSE. Fedora is treating the shebang in the same way, apparently. [2]
Seems that if you only have python3 installed in Fedora31 this is *not* the case.
# python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3
See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package
"there is no /usr/bin/python"
So the two distros are quite divergent in their approach apparently?
It may be true that a given script is fine with Python 3, but as long as the shebang says "python" (i.e. python2), there's no way to really find out, is there? (Barring things like Josh's suggestion of changing the shebang on the fly via a teuthology task/workunit, which is fine if we decide we need a transition period, which it looks like we will.)
Nathan
[1]
https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/ce...
[2]
https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin....
_______________________________________________ Sepia mailing list Sepia@lists.ceph.com http://lists.ceph.com/listinfo.cgi/sepia-ceph.com
-- Cheers, Brad _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, Nov 18, 2019 at 12:17 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
On Sun, Nov 17, 2019 at 8:09 PM Brad Hubbard <bhubbard@redhat.com> wrote:
On Wed, Apr 18, 2018 at 6:27 AM Nathan Cutler <ncutler@suse.cz> wrote:
That would be at odds to what Nathan is suggesting though, which is a hard change to Python 3.
Hm, not sure what hard/soft means in this context. For any given script, either it runs with Python 3, or it doesn't. And this is determined by the shebang. (Unless the shebang is omitted, of course.)
Apologies if this has already been mentioned, but FYI, Fedora (and RHEL and CentOS by extension) do not allow ambiguous python shebangs – they _require_ explicit versioned shebangs.
I presume that is what is meant by a hard change to Python3.
things like: #!/usr/bin/env python #!/usr/bin/python are not permitted.
(Also not #!/usr/bin/env $anything)
Shebangs – if there is one – in .py files must be either #!/usr/bin/python3 or #!/usr/bin/python2
AFAIK Fedora is enforcing it, kinda. Package builds in fedora run rpmlint which checks for these kinds of things, although I'm not sure this is a test yet or if there is a test that it scores high enough to fail a build. (Ditto for package builds in the OpenSUSE OBS, i.e. they also run rpmlint, although I haven't noticed python shebang tests there either.)
Does this mean you can no longer write a script that supports both python 2 and 3? Or, you can write it that way but it will only ever be run against python2 OR python3 so there is no point? Is it expected you would need two versions of a script if you wish to support both? I realise questions of supporting both become less important going forward and that the emphasis should be on migration to python 3 but I'm interested in the answers to these questions nonetheless (not placing the burden of answering them solely on your shoulders Kaleb).
I was very surprised to find out that, in SLES and openSUSE, the symlink /usr/bin/python -> /usr/bin/python2 will not be changed even when the migration of the underlying distro to Python 3 is complete.
But then my colleagues explained why that is, and I "saw the light". Since every single script in the distro has to be audited for Python 3 compatibility, anyway, it makes sense to have the shebang be an explicit declaration of said compatibility.
By retaining the symlink at it is, all scripts start out the migration process with an explicit declaration that they are compatible with Python 2. Compatibility with Python 3 is signalled not by saying "it's OK with Python 3, we tried it". It's signalled by changing the shebang.
And this isn't unique to SUSE. Fedora is treating the shebang in the same way, apparently. [2]
Seems that if you only have python3 installed in Fedora31 this is *not* the case.
# python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3
See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package
"there is no /usr/bin/python"
So the two distros are quite divergent in their approach apparently?
It may be true that a given script is fine with Python 3, but as long as the shebang says "python" (i.e. python2), there's no way to really find out, is there? (Barring things like Josh's suggestion of changing the shebang on the fly via a teuthology task/workunit, which is fine if we decide we need a transition period, which it looks like we will.)
Nathan
[1] https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/ce... [2] https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin.... _______________________________________________ Sepia mailing list Sepia@lists.ceph.com http://lists.ceph.com/listinfo.cgi/sepia-ceph.com
-- Cheers, Brad _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Cheers, Brad
On Mon, Nov 18, 2019 at 12:43 PM Brad Hubbard <bhubbard@redhat.com> wrote:
On Mon, Nov 18, 2019 at 12:17 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
On Sun, Nov 17, 2019 at 8:09 PM Brad Hubbard <bhubbard@redhat.com> wrote:
On Wed, Apr 18, 2018 at 6:27 AM Nathan Cutler <ncutler@suse.cz> wrote:
That would be at odds to what Nathan is suggesting though, which is a hard change to Python 3.
Hm, not sure what hard/soft means in this context. For any given script, either it runs with Python 3, or it doesn't. And this is determined by the shebang. (Unless the shebang is omitted, of course.)
Apologies if this has already been mentioned, but FYI, Fedora (and RHEL and CentOS by extension) do not allow ambiguous python shebangs – they _require_ explicit versioned shebangs.
I presume that is what is meant by a hard change to Python3.
things like: #!/usr/bin/env python #!/usr/bin/python are not permitted.
(Also not #!/usr/bin/env $anything)
Shebangs – if there is one – in .py files must be either #!/usr/bin/python3 or #!/usr/bin/python2
AFAIK Fedora is enforcing it, kinda. Package builds in fedora run rpmlint which checks for these kinds of things, although I'm not sure this is a test yet or if there is a test that it scores high enough to fail a build. (Ditto for package builds in the OpenSUSE OBS, i.e. they also run rpmlint, although I haven't noticed python shebang tests there either.)
Does this mean you can no longer write a script that supports both python 2 and 3? Or, you can write it that way but it will only ever be run against python2 OR python3 so there is no point? Is it expected you would need two versions of a script if you wish to support both? I realise questions of supporting both become less important going forward and that the emphasis should be on migration to python 3 but I'm interested in the answers to these questions nonetheless (not placing the burden of answering them solely on your shoulders Kaleb).
Ignore this. I must have been having a mega brain-fade when I wrote it and I still can't understand how I even formulated these questions. Sorry for the noise.
I was very surprised to find out that, in SLES and openSUSE, the symlink /usr/bin/python -> /usr/bin/python2 will not be changed even when the migration of the underlying distro to Python 3 is complete.
But then my colleagues explained why that is, and I "saw the light". Since every single script in the distro has to be audited for Python 3 compatibility, anyway, it makes sense to have the shebang be an explicit declaration of said compatibility.
By retaining the symlink at it is, all scripts start out the migration process with an explicit declaration that they are compatible with Python 2. Compatibility with Python 3 is signalled not by saying "it's OK with Python 3, we tried it". It's signalled by changing the shebang.
And this isn't unique to SUSE. Fedora is treating the shebang in the same way, apparently. [2]
Seems that if you only have python3 installed in Fedora31 this is *not* the case.
# python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3
See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package
"there is no /usr/bin/python"
So the two distros are quite divergent in their approach apparently?
It may be true that a given script is fine with Python 3, but as long as the shebang says "python" (i.e. python2), there's no way to really find out, is there? (Barring things like Josh's suggestion of changing the shebang on the fly via a teuthology task/workunit, which is fine if we decide we need a transition period, which it looks like we will.)
Nathan
[1] https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/ce... [2] https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin.... _______________________________________________ Sepia mailing list Sepia@lists.ceph.com http://lists.ceph.com/listinfo.cgi/sepia-ceph.com
-- Cheers, Brad _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Cheers, Brad
-- Cheers, Brad
On Sun, Nov 17, 2019 at 9:17 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
On Sun, Nov 17, 2019 at 8:09 PM Brad Hubbard <bhubbard@redhat.com> wrote:
On Wed, Apr 18, 2018 at 6:27 AM Nathan Cutler <ncutler@suse.cz> wrote:
That would be at odds to what Nathan is suggesting though, which is a hard change to Python 3.
Hm, not sure what hard/soft means in this context. For any given script, either it runs with Python 3, or it doesn't. And this is determined by the shebang. (Unless the shebang is omitted, of course.)
Apologies if this has already been mentioned, but FYI, Fedora (and RHEL and CentOS by extension) do not allow ambiguous python shebangs – they _require_ explicit versioned shebangs.
I presume that is what is meant by a hard change to Python3.
things like: #!/usr/bin/env python #!/usr/bin/python are not permitted.
(Also not #!/usr/bin/env $anything)
Shebangs – if there is one – in .py files must be either #!/usr/bin/python3 or #!/usr/bin/python2
AFAIK Fedora is enforcing it, kinda. Package builds in fedora run rpmlint which checks for these kinds of things, although I'm not sure this is a test yet or if there is a test that it scores high enough to fail a build. (Ditto for package builds in the OpenSUSE OBS, i.e. they also run rpmlint, although I haven't noticed python shebang tests there either.)
I can speak somewhat authoritatively on the Fedora situation, as I've helped with implementing it. In Fedora 31, we have transitioned all "unversioned" binaries from the Python stack from Python 2 to Python 3. All "unversioned" package names have also transitioned. See https://fedoraproject.org/wiki/Changes/Python_means_Python3 In Fedora 32, the aim is to retire nearly all of the Python 2 stack. The interpreter will remain and small selection of module packages will exist too (probably in the order of dozens). See https://fedoraproject.org/wiki/Changes/RetirePython2 We have enforced packages to pick either Python 2 or Python 3 for shebangs since Fedora 30 through a build-root policy script: https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error Somewhat relatedly, we automatically generate Python distribution dependencies by default since Fedora 30: https://fedoraproject.org/wiki/Changes/EnablingPythonGeneratorsByDefault Also related, this generator can be manually activated for EL7 and EL8 by adding the following to the top of a spec: # Enable Python dependency generation %{?python_enable_dependency_generator} You can see an example of that here: https://src.fedoraproject.org/rpms/python-hvac/blob/51406e2c7383bf15303ecf5c... openSUSE is orders of magnitude behind Fedora on this. All packages are still being built dual-stack, and the preference has not been switched from Python 2 to Python 3. There is no dependency generator for Python module dependencies, so getting a clear picture of the dependency web is much more complex. My understanding is that openSUSE will break the world on this only after Python 2 is retired from Factory in January 2020. It's going to be hard and painful, and it's because openSUSE is taking a *really* slow path to get there. -- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Dec 2, 2019 at 2:10 PM Neal Gompa <ngompa13@gmail.com> wrote:
Somewhat relatedly, we automatically generate Python distribution dependencies by default since Fedora 30: https://fedoraproject.org/wiki/Changes/EnablingPythonGeneratorsByDefault
Also related, this generator can be manually activated for EL7 and EL8 by adding the following to the top of a spec: # Enable Python dependency generation %{?python_enable_dependency_generator}
You can see an example of that here: https://src.fedoraproject.org/rpms/python-hvac/blob/51406e2c7383bf15303ecf5c...
I forgot to point this out, but the generator only works for Python 3 stuff in EL7, it works for all Python stacks shipped in RHEL for EL8. -- 真実はいつも一つ！/ Always, there's only one truth!
On Tue, Dec 3, 2019 at 5:16 AM Neal Gompa <ngompa13@gmail.com> wrote:
On Mon, Dec 2, 2019 at 2:10 PM Neal Gompa <ngompa13@gmail.com> wrote:
Somewhat relatedly, we automatically generate Python distribution dependencies by default since Fedora 30: https://fedoraproject.org/wiki/Changes/EnablingPythonGeneratorsByDefault
Also related, this generator can be manually activated for EL7 and EL8 by adding the following to the top of a spec: # Enable Python dependency generation %{?python_enable_dependency_generator}
You can see an example of that here: https://src.fedoraproject.org/rpms/python-hvac/blob/51406e2c7383bf15303ecf5c...
I forgot to point this out, but the generator only works for Python 3 stuff in EL7, it works for all Python stacks shipped in RHEL for EL8.
Good to know Neal. Thanks for the information.
-- 真実はいつも一つ！/ Always, there's only one truth!
-- Cheers, Brad
Seems that if you only have python3 installed in Fedora31 this is *not* the case.
# python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3
See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package
"there is no /usr/bin/python"
So the two distros are quite divergent in their approach apparently?
In Fedora 31, if I have a script with the following shebang: #!/usr/bin/python3 will it not work? Nathan
On Fri, Nov 22, 2019 at 6:53 PM Nathan Cutler <ncutler@suse.com> wrote:
Seems that if you only have python3 installed in Fedora31 this is *not* the case.
# python --version Python 3.7.5 # /usr/bin/python --version Python 3.7.5 # ls -l /usr/bin/python lrwxrwxrwx. 1 root root 9 Nov 18 00:57 /usr/bin/python -> ./python3
See https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... and https://fedoraproject.org/wiki/Changes/RetirePython2#The_python27_package
"there is no /usr/bin/python"
So the two distros are quite divergent in their approach apparently?
In Fedora 31, if I have a script with the following shebang:
#!/usr/bin/python3
will it not work?
Sure it will (and I'm sure you've tried it already :) ) I don't think I said it wouldn't. I was pointing out the difference between your statement that "/usr/bin/python -> /usr/bin/python2 will not be changed" in SLES and openSUSE and the current reality in Fedora 31 and that's where the "two distros are quite divergent in their approach apparently" came from. I'm not convinced we're on the same page.
Nathan
-- Cheers, Brad
Hi Brad:
I don't think I said it wouldn't. I was pointing out the difference between your statement that "/usr/bin/python -> /usr/bin/python2 will not be changed" in SLES and openSUSE and the current reality in Fedora 31 and that's where the "two distros are quite divergent in their approach apparently" came from. I'm not convinced we're on the same page.
What I'm trying (poorly) to say is: as long as we explicitly state "python3" everywhere and diligently avoid "python", we'll be OK: we will not need to care what "python" points to in a given distro. The motivation for keeping "python -> python2" is/was to avoid breaking legacy scripts. I guess the Fedora folks are not so concerned about that. And it seems reasonable to expect even the more conservative distros to switch over to "python -> python3", at some point in the future - when Python 2 recedes into the cosmic background noise and the fear of breaking older scripts will have subsided. Nathan
Got it. :) On Mon, Nov 25, 2019 at 6:55 AM Nathan Cutler <ncutler@suse.com> wrote:
Hi Brad:
I don't think I said it wouldn't. I was pointing out the difference between your statement that "/usr/bin/python -> /usr/bin/python2 will not be changed" in SLES and openSUSE and the current reality in Fedora 31 and that's where the "two distros are quite divergent in their approach apparently" came from. I'm not convinced we're on the same page.
What I'm trying (poorly) to say is: as long as we explicitly state "python3" everywhere and diligently avoid "python", we'll be OK: we will not need to care what "python" points to in a given distro.
The motivation for keeping "python -> python2" is/was to avoid breaking legacy scripts. I guess the Fedora folks are not so concerned about that. And it seems reasonable to expect even the more conservative distros to switch over to "python -> python3", at some point in the future - when Python 2 recedes into the cosmic background noise and the fear of breaking older scripts will have subsided.
Nathan
-- Cheers, Brad
participants (4)
-
Brad Hubbard
-
Kaleb Keithley
-
Nathan Cutler
-
Neal Gompa