Gentoo's package manager

Why is Gentoo's package manager made in Python?


wiki.gentoo.org/wiki/Portage

I just want to know why it is written in Python and why it has not been rewritten

Pic related btw.

Attached: gentoo_loves_python.png (653x451, 90.37K)

Other urls found in this thread:

wiki.gentoo.org/wiki/Handbook:AMD64/Working/Portage
vimeo.com/108441214
en.wikipedia.org/wiki/Big_ball_of_mud
en.wikipedia.org/wiki/Coupling_(computer_programming)
en.wikipedia.org/wiki/Smalltalk
wiki.gentoo.org/wiki/Package_Manager_Specification
dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html
dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html
dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html
dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html
twitter.com/NSFWRedditGif

wiki.gentoo.org/wiki/Handbook:AMD64/Working/Portage

Portage is inspired by the seemingly simple nature of ports on *bsd systems.
Therefore it's designed in a way that makes it transparent for the enduser.
Making it in python/bash has many advantages, such as easily being able to make changes to portage without having to recompile, this makes portage a very fast and versitile packagemanager.
However, there is one huge downside to python and that is indeed speed and it's reliance on python.
If you've programmed in python before and also use gentoo you probably know about the issues gentoo has when it comes to pip python's offical packagemanager
When using Pip, you have the change to completely overwrite some key dependencies of portage when using it outside of the user mode.
On the speed part of portage, it's trivial these days as new version of portage support a multi-thread setting, the only part you will have actuall issues with is searching for things, which thankfully is handled with eix.

I remember there used to be a project that was working on a portage written in C, i don't think they're working on it anymore.

Portage is first and foremost a horrible mess, which makes it hard to replace and me glad that its authors aren't writing it in C. Generally I think a package manager should be as self-contained as possible to avoid the problems in , but that conflicts with compilation a bit. Maybe an ideal package manager could bundle its high level language's runtime or something.

Because C makes everyone run screaming

portage is not like other package managers
it's a complex package management system and it's written in python for the sake of maintainability, readability and hackability
writing it in C would only make development more cumbersome while offering pretty much no real advantage in return
you can try paludis and you'll see it won't resolve dependencies any faster
i'm not a huge fan of python myself, but it unironically is the right tool for the job in this case

Fuck off. Paludis is a meme by trolls, not the good kind of /b/ trolls either. It is nice to have a alternative to the portage codebase should anything go horrible wrong though.

There's a alternative to portage called pkgcore which is written in C and python but the slow parts of portage were rewritten in C. If you use their latest git, not their 2017 version. and a minimalist system it makes everything go much faster for re/installing packages. As an example it cut my ram usage by a tenth of what portage uses to do the same thing along with being faster. @world rebuilds for a full GUI setup with browser went from ten hours to eight just by switching to pkgcore for my setup. It might not work with esoteric packages like systemd or gnome shitware. But if you are using shitware why are you going for a minimalist and fast installer?

Because the creator of gentoo was lazy/not serious at the time of making portage and it has been a cobbled together mess since day 1. There's even a specification against which features must be supported for a gentoo package manager if you wanted to rewrite it in say rust or ada. I forget the name of it now.

This is truly a dumb nitpick.

For just pulling data and running system commands, Python does not need impressive performance. The difference to the user is negligible.
Compiling is still the bulk of time used for an emerge, and it's running on a compiler written in C.
I agree it's at odds with the "I compile with special flags to make my stuff run a millisecond faster" mentality of many Gentoo regulars, myself included.
But there's no reason to write it in C, and the best alternative would be a Source Mage style "all bash scripts" system. Better performance? Sure. But not worth the time to rewrite and much harder to maintain.


This is a fundamental flaw with Python versioning. Use pyvenv for everything you do. It's a dumb hack workaround but it's standard at this point. Maybe "Python as a kernel" can be the first step to fixing it but it seems the core leadership have huge blindspots about what Python developers actually need so I'm not too optimistic yet.

Did nobody ever read "Clean Code: A Handbook of Agile Software Craftsmanship" ? or "Working Effectively with Legacy Code" ?

youtube-dl --user-agent "Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0" vimeo.com/108441214

en.wikipedia.org/wiki/Big_ball_of_mud
en.wikipedia.org/wiki/Coupling_(computer_programming)
en.wikipedia.org/wiki/Smalltalk

This. The runtime cost of compiling an app is the overwhelming majority when compared with the runtime cost of running Portage itself. Python makes programming easy. C is too much effort for the task of package manager. This is an application package manager whose task is to download compile and install instructions for packages, activate the compiling work and then organizing the final output.

Well the advantage of source mage is that you don't have python dependencies which are not only heavy but dangerous considering what happened this year.

Interesting, I'll keep this in mind.
I'm assuming you mean this. wiki.gentoo.org/wiki/Package_Manager_Specification

Actually the most time is spent on autotools bullshit checking whether you have a faulty fortran compiler from 1980 :^)

The A.I is correct. Most of the time building packages is spent in libtool/autotools with the exception of monsters like firefox and libreoffice. Libtool and autotools is single threaded and not parrellelized whatsoever. Libtool is a neccessity on every unix/macosx/linux/solaris install. You could remove autotools and convert everything to meson and use the openbsd kernel with busybox+musl userland but everything else dependent on configure.ac scripts still requires libtool. Libtool needs a security audit ASAP.
The following specifications were used in developing and evaluatingthis system: 1. The system must be as elegant as possible. 2. The system must be fully integrated with the GNU Autoconf and Automake utilities, so that it will be easy for GNU maintainers to use. However, the system must not require these tools, so that it can be used by non-GNU packages. 3. Portability to other (non-GNU) architectures and tools is desirable.
They utterly failse at point 2.
=========================Even before libtool was developed, many free software packages built andinstalled their own shared libraries. At first, these packages wereexamined to avoid reinventing existing features. Now it is clear that none of these packages have documented thedetails of shared library systems that libtool requires. So, otherpackages have been more or less abandoned as influences.
Which is to say there are none.

Why is libtool even needed? I might be a brainlet, but wouldn't it be possible to pass a variable to make like "PUT_SHITE_HERE=/usr/lib" explicitly? A human could do it by hand (would be tedious), but more importantly, a package manager could do it automatically properly, instead of relying on a magic tool.

I don't think you understand how evolved ebuilds, the EAPI and portage are, rewriting them in C would be dumb. Written some modules in a statically typed no GC language would help though, but extending Python is ugly as shit compared to something like Tcl or Lua.
By the way, aren't some parts like the depend atom parsing already written in C or cython?

They should make package manager in fortran.

Portage is anything but simple though, in fact I'm pretty sure we could say it's the most advanced package manager out there (though admittedly also the best).

OP here glad to see this got some traction.


I missed that from the handbook, that helps to explain.


I thought portage's source code was compiled into "pyc" files but I guess I was wrong.

What about the text emerge outputs to the screen if you do not use the quiet flag? Does that not harm performance since it is Python doing that output?

Python's compilation is just a transparent caching mechanism common to many high-level languages, not really comparable to replacing a C binary. Output is a syscall so pathological cases aside the language doesn't matter much. The biggest CPU bottleneck in Portage is probably massaging the dependency graph; I remember how back when I used Haskell, update previews would take ages simply because that language's overlay has so many micro packages.

My (admittedly novice) take is that any delays introduced by the use of Python for portage are minimal compared to the time required to compile the packages. Why not use a scripting language for scripting?

I'm hooked on gentoo. Serves me well on my 8c/16t desktop and on my C2D laptop and in everything in between.

I wish I knew more about the sysadmin domain... I would love to use something like puppet to ensure that my various installs stayed in sync as far as DE/WM limitations allow.

What do you mean by this?

What other package manager lets you hotswap out your standard C library or your system SSL library in about five commands? What other package manager has the concept of binary vs source code split besides guixs'. What other package manager, besides windows' internal compilation script nightmare, lets you compile things statically, shared, or both at a whim and keeps track of it all? What other package manager lets you install to new architectures that the *insert developers here* haven't compiled binaries for? What other package manager lets you set a single flag to disable or enable a feature going by different names in different packages for your entire system?

I don't know but you should install it haha

Guix, Nix

If you're talking about openssl vs libressl you still have to recompile your whole system though don't you?

ports

You do, which is hilarious considering that to solve this problem is literally what dynamic linking was invented for. Swapping out your libc can have "fun" effects too, especially when downgrading. I once had 80% of all programs fail at load time because I downgraded to a version before bzero support. The rest of the system was of course still compiled for it.

There was a niche distro once based on a build system called tup, where basically the system was a huge makefile. In a sense, the package manager kept almost-copiled programs around and could quickly relink or partially recompile when something changed. In my opinion, this would be far superior to the current dynamic linking and full recompilation shitfest.

take a look at emerge(1), ebuild(5), make.conf(5) and porage(5) and you'll see what he means
dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html
dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html
dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html
dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html


guix and nix are cool, but porage has overwhelmingly more features


ports are fine if you only plan on making modifications to a couple of packages which won't affect other packages
otherwise, it's not really feasible unless you've got time to maintain your own port tree

The only reason the libressl vs openssl recompilation is a think is because openssl broke the abi with pkcs11 vs pkcs9 split or something like that. Programs support both but some programs have the split into different code depending on what's avaliable at link time, with the other being left out of the symbols. No excuse for the shit joke that is dynamic linking in cy+24

You can do that with portage it is called ccache. It does exactly as you described by saving the object files and only recompiling a program based on changes to the objects.

Yes Portage has more features, but those features are irrelevant because of Guix's features.

Do these "features" include the trannycock and being written in language maintained by an ultra faggot?

G-d I'm so tired of this zoomer meme.
Portage being written in Python is a literal non-issue. It works, and it works well, better than most other package managers written in your beloved C.
USE flags cause calculating dependencies to be inherently slower. Paludis exists, and it wasn't much faster than Portage.

I don't see any dicks in the source code, if you do, you should visit a doctor.

I'm surprised it's not written in GO

Ah, so that's why emerge takes ten years to check dependencies.