Written by m4r3k and tagged by openSUSE.
Hi, as you probably know OBS is again under big overload, waiting queue is pretty long (i586: 2746 waiting, x86_64: 2729 waiting) so I come with a few ideas how to help to figure out this situation and I would like to hear your comments and opinions about these optimizations, so here we go:
Optimization #1
Packages are after build checked with rpmlint and in some projects there are a lot of packages not able to pass rpmlint test in openSUSE:Factory and I think it could save some builds if new Factory build of package foo is not triggered by scheduler if same package (revision of package) failed rpmlint a few days or hour ago. You know these problem in package are not getting to fix itself just because whole openSUSE:Factory project got rebuilded.
But for the case of some changes in Factory rpmlint rules, there could be as I told few hours or days timeout. After this timeout new rebuild of package foo could be triggered automatically.
This rule should not be applied to manual triggered rebuild such a osc rebuildpac and osc commit.
Optimization #2
Sometimes package build get freeze for not obvious reason. Like right now package aqemu in project home:rom-ale, job time is about 6 hours I don’t wanna imagine how big package have to be to get builded for 6 hours (do we have any package getting build for 6 hours in distribution?) so I think there could be some watchdog watching job time and after exceeding some kind of barrier (2 hours, 4 hours? for discussion) obsworker with this package should be automatically rebooted or something.
For case of really big package there could be some kind of configuration on package meta where user can manually change this barrier or turn it off, but by default this barrier should be turned on.
Optimization #3
This is not my idea, but idea of my friend Michal Smrž from czech openSUSE community. He think it could be useful to have cross build on those ppc64 nodes we have an OBS right now. You know they are idle almost all the time (right now we have 25 idling ppc64 build servers) so they can be used this way in time of idling if it is possible to build x86_64 and/or i586 packages using cross build platform in OBS, to be honest I am not sure if it is possible, but I think it is…
Call-up
Well all ideas a wrote in this article about are just ideas and I would like to discuss these ideas with you, openSUSE community members and openSUSE Build Service users, I hope we will figure something out together to have more optimized build power for ourself. :-)
Also, you guys who can use _aggregate function instead of _link do so! You’re helping not only yourself, but also others OBS users. :-)
December 20th, 2008 at 5:35 pm
I fully agree with Marek.
I have one note to Optimization #2.
There could be two watchdogs.
Firts, with shorter time, which will look on last activity time in build log.
Second, with longer time, will look on time of whole build, no matter what is in build log (in case, some recursive error appears in building process).
December 21st, 2008 at 11:14 am
Well, one thing I certainly doesn’t like is if I commit package, waiting for package to be build and my “powerful” local machine is just idling. I would prefer if I can build package locally and upload it back or if I can connect with icecream or something similar to actively help with getting my package built. I know that uploading locally built package can have some security issues but it can be marked somehow as dirty (maybe less dirty with icecream) and let people decide whether they trust packager that much… And these packages can be rebuilt later to get some more thrust but there is no rush.
Other thing I’m thinking about is whether there is some way how to prioritize/deprioritize packages. I guess it’s more important to build package which blocks many others then some leaf packages… And I can imagine in situations like nowadays that I’ll want to say that I need this package on this architecture faster and I don’t need these other packages so fast so they can wait. It happened to me already that I was working on several packages and the package I wanted to get build first get built as a last one. I know that partial solution is to disable building of other packages, but prioritizing one of them at the cost of the others would be nicer cause I wouldn’t forget to enabling build of the other packages.
January 2nd, 2009 at 2:10 am
To #1: Well, what happens, if we’ve a problem in rpmlint checks? Perhaps the “broken” package will build if a new rpmlint package is checked in?
To #2: Look at OpenOffice if you search for a package building nearly 12hours (24h on low-CPU hosts). Even the kernel-source needs more than 2hours on some hosts. People not knowing a default value (for example 2hours build job) would be very surprised if they start their new, big package over night and see something like “please add variable ‘foo’ to your config” the other day - instead of a clean build result. We shouldn’t start to use this feature (which is already available in the code IMO) until we’ve a good documentation about it and even a webfrontend for disabling it.
To #3: we could use the ppc hosts for building noarch packages. But therefor we need a “noarch” scheduler…