perl 5.38.0

[jamesjer]

The Fedora Linux distribution just updated to perl 5.38.0 in Rawhide, which will become Fedora 39 in the fall. Since polymake 4.10 is incompatible with perl 5.38.0, this leaves the polymake package unbuildable and uninstallable. As the polymake package maintainer, I need to decide what to do. As I see it, I have 3 options:

1. Leave polymake in its broken state for now and wait for a new release.
2. Beg the perl package maintainers to add a downstream patch to temporarily expose the hidden functions needed by polymake. (In this case, I will need a list of the affected functions.)
3. Remove polymake from Fedora and deal with the consequences (such as removing the GAP polymaking package as well).

Do you anticipate a redesign of polymake to avoid the now-hidden APIs? If so, do you have any guesses as to how long it will take?

[gawrilow]

Indeed, we are aware of the incompatibility problem with perl 5.38. There is no simple cure against the symbol hiding introduced there. Eventually polymake will lose the mandatory perl bindings, but this refactoring is still work in progress and there is no clear timeline yet for the completion. The only solution for now, besides withdrawing polymake package altogether, could be to add a dedicated package with perl 5.36 to the new distro. As such, it's technically feasible, one can install as many different perl versions side by side as one wants, but somebody must take care of that package and of the accompanying CPAN stuff required by polymake (such as JSON, Term::Readline etc.) which might need to be frozen at older versions too.

[jamesjer]

Sadly, the approach of keeping perl 5.36 around is not really feasible. It might be technically feasible, but the amount of time and attention it would require is far more than I can take on. I think I will have to withdraw polymake from Fedora for now. I wish you good luck and good speed in moving beyond this issue.

[jamesjer]

I announced my intention to retire the polymake package on the Fedora developers list. Kevin Kofler remarked that pointers to Perl_ck_fun can be found in PL_check array, and that the body of Perl_save_pushptrptr can be copied verbatim into the polymake code. That got me wondering how many more such tricks could be found. Starting with Kevin's ideas, I came up with this patch: https://jjames.fedorapeople.org/polymak ... 5.38.patch. It is not complete. Polymake can be compiled and linked successfully, but there is a heap corruption issue. Just running "polymake --reconfigure" only works about 1 out of 8 tries, otherwise polymake segfaults. I can see that an SV on the free list (PL_sv_root) has a bad "next" pointer (sv_any). Once that SV is popped off of the free list, the root pointer is bad, leading to the segfault. I haven't yet figured out how that happens.

One disadvantage of this approach is that the code in lib/core/src/perl/perl.cc was copied from the perl 5.38.0 release. It carries the license notice of other perl code. That code would have to be compared with upstream perl code on every subsequent polymake release.

I was careful to wrap changes in #if PerlVersion >= 5380 ... #endif, even when the code in question should work for earlier perl versions. Most changes should be fairly obvious, but the intercept_pp_anoncode change is due to the change described here: https://perldoc.perl.org/perl5380delta# ... hancements.

Does this approach interest you at all?

[gawrilow]

Thanks for your heroic attempts to save the package! However, I'm afraid that this approach won't be sustainable. My guess is that perl maintainers introduced the symbol hiding on purpose, probably they plan to massively refactor the interpreter guts and with this measure they aimed exactly at catching "malicious" extensions like polymake early enough, to avoid later complaints about crashes, broken APIs and such. Even if we would trick somehow around the hiding barrier in 5.38, the probable impending changes in the near future will cause much more dramatic failures. Moreover, polymake extension relies not only on the check functions stored in the PL_check table but also on other routines manipulating the op-tree. They all have become inaccessible now.

The future direction of polymake development is clearly steering off from perl towards Julia as a language for user interaction and integration with other software in the computer algebra domain. The marriage with perl was long and happy, but now this story comes to a natural end.

[jamesjer]

I understand. I look forward to seeing your Julia work.

[Joachim Zobel]

This means that polymake will vanish from all linux distributions for now. For Debian it will most likely not be in trixie.

Do we know why the perl developers are doing this? Has a workaround (temporary would be good enough) been requested?

[jamesjer]

Presumably the perl maintainers want to make some changes to the internals. As for temporary measures, let me point out the incomplete patch for perl 5.38 mentioned above: https://jjames.fedorapeople.org/polymak ... 5.38.patch. That patch lets polymake compile, and "polymake --reconfigure" works. However, when polymake is given an actual problem, the rules search finds almost no viable rules. I haven't yet tracked down that issue. If anybody is interested in working with me on completing this patch, please get in touch: loganjerry@gmail.com.

[Joachim Zobel]

I did a tiny bit of research. Perl is now using a gcc attribute (hidden) to hide internal functions. That is seemingly what hit polymake. See perldelta 5.38.0.

[gawrilow]

... I haven't yet tracked down that issue. If anybody is interested in working with me on completing this patch, please get in touch

I fully understand and appreciate your wish to help us in this misery, but I'm afraid any hacks we would try here will be in vain. Once the perl maintainers decided to hide important parts of the API, they will surely keep changing it in whatever manner and frequency and we won't be able to catch up. The time has come to say good bye to perl in polymake. Actually we were warned early enough by Mongo driver maintainers who announced the end-of-life of perl support already few years ago. Unfortunately, we haven't started refactoring activities on full scale right away...