generate-filelist: create fewer DB sessions

Signed-off-by: Torsten Werner <twerner@debian.org>

run g-p-s in unchecked

Signed-off-by: Joerg Jaspert <joerg@debian.org>

missing t

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'twerner/gf' into merge

* twerner/gf:
  drop an obsolete view by db update34
  generate-filelist: implement incremental mode

Signed-off-by: Joerg Jaspert <joerg@debian.org>

drop an obsolete view by db update34

Signed-off-by: Torsten Werner <twerner@debian.org>

generate-filelist: implement incremental mode

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  examine-package: replace lock by thread.local()

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' into show-new

examine-package: replace lock by thread.local()

- thread local storage is more robust than locks
- the code is more correct than before

Signed-off-by: Torsten Werner <twerner@debian.org>

Serialize by suite

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Slightly more useful tempfile names

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'gps' into merge

* gps:
  Use the new script in dinstall
  generate-packages-sources
  generate-packages-sources
  dont replace the path in DAILY_APT_CONF right now, make it nicer later in an AptCOnf class. also dont call with the suite name, it knows it anyways
  generate-packages-sources
  Add start of a generate_packages_sources command

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Use the new script in dinstall

Signed-off-by: Joerg Jaspert <joerg@debian.org>

generate-packages-sources

finalize the first step. there needs to be a rework of the apt-trees part
into an aptconfig class. Much to inflexible the way it is now, but we can
start testing this out in real life now.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge commit 'ftpmaster/master'

Extend a hack

Signed-off-by: Mark Hymers <mhy@debian.org>

Remove an entirely unused config section

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  show-new: reduce DB sessions even more

Signed-off-by: Joerg Jaspert <joerg@debian.org>

show-new: reduce DB sessions even more

Signed-off-by: Torsten Werner <twerner@debian.org>

We use upload.d.o now

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  show-new: reduce number of open DB sessions

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' into show-new

show-new: reduce number of open DB sessions

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'lamby/master' into merge

* lamby/master:
  Popen.communicate calls stdout.close() which looks like is causing errors w/threads..
  Finally kill m68k
  Support +b[0-9] versions in autobyhand for di

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Popen.communicate calls stdout.close() which looks like is causing errors w/threads..

Finally kill m68k

Signed-off-by: Mark Hymers <mhy@debian.org>

Support +b[0-9] versions in autobyhand for di

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
  Finally kill m68k
  Support +b[0-9] versions in autobyhand for di

Merge remote branch 'lamby/master' into merge

* lamby/master:
  Fix race condition in our commands.getstatusoutput monkeypatch

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Finally kill m68k

Signed-off-by: Mark Hymers <mhy@debian.org>

Support +b[0-9] versions in autobyhand for di

Signed-off-by: Mark Hymers <mhy@debian.org>

Fix race condition in our commands.getstatusoutput monkeypatch

Previously we were returning from this method before the process had actually
finished, introducing issues where we relied on process actually finishing or
simply not returning all the data the process was going to produce.

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  examine-package: do not create useless DB sessions

Signed-off-by: Joerg Jaspert <joerg@debian.org>

examine-package: do not create useless DB sessions

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  show-new: close ressources earlier

Signed-off-by: Joerg Jaspert <joerg@debian.org>

show-new: close ressources earlier

Signed-off-by: Torsten Werner <twerner@debian.org>

generate-packages-sources

some more changes:
 - chdir before we call the generate function
 - dont include architecture all
 - use commands.getstatusoutput
    this breaks the try, finally: block. using os.system() it works nicely,
    but the output is a hell. using commands.getstatusoutput the finally:
    is called to early, removing the config for apt-ftparchive, breaking it.
    ugh, bad.
 - the whole mess with the config inside dbconn.py needs to go, as well
   as the special cases for the tree configs. should be an aptconf class,
   but thats a task for when this fully works

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' into gps

* master:
  examine-package: lock access to printed_copyrights
  show-new: multithreading support
  show-new: no longer fiddle with sys.stdout
  examine_package: make most functions functional
  examine-package: really support option -H
  examine-package: add missing newlines to output
  examine-package: make foldable_output functional()
  examine-package: make headline() functional
  call check_deb() with correct arguments
  Add config for volatile

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  examine-package: lock access to printed_copyrights
  show-new: multithreading support
  show-new: no longer fiddle with sys.stdout
  examine_package: make most functions functional
  examine-package: really support option -H
  examine-package: add missing newlines to output
  examine-package: make foldable_output functional()
  examine-package: make headline() functional
  call check_deb() with correct arguments
  Add config for volatile

Signed-off-by: Joerg Jaspert <joerg@debian.org>

dont replace the path in DAILY_APT_CONF right now, make it nicer later in an AptCOnf class. also dont call with the suite name, it knows it anyways

Signed-off-by: Joerg Jaspert <joerg@debian.org>

examine-package: lock access to printed_copyrights

FIXME: the global variable printed_copyrights is still broken...

Signed-off-by: Torsten Werner <twerner@debian.org>

show-new: multithreading support

Signed-off-by: Torsten Werner <twerner@debian.org>

generate-packages-sources

generate packages sources files, threaded, one thread per suite/arch

Signed-off-by: Joerg Jaspert <joerg@debian.org>

show-new: no longer fiddle with sys.stdout

Signed-off-by: Torsten Werner <twerner@debian.org>

examine_package: make most functions functional

Signed-off-by: Torsten Werner <twerner@debian.org>

examine-package: really support option -H

Signed-off-by: Torsten Werner <twerner@debian.org>

examine-package: add missing newlines to output

Signed-off-by: Torsten Werner <twerner@debian.org>

examine-package: make foldable_output functional()

Signed-off-by: Torsten Werner <twerner@debian.org>

examine-package: make headline() functional

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' into show-new

call check_deb() with correct arguments

Signed-off-by: Torsten Werner <twerner@debian.org>

Add start of a generate_packages_sources command

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Add config for volatile

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge commit 'ftpmaster/master'

Add volatile testing dists files

Signed-off-by: Mark Hymers <mhy@debian.org>

Less backgrounding before a-f

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'twerner/show-new' into merge

* twerner/show-new:
  show-new: open a new session for every source package

Signed-off-by: Joerg Jaspert <joerg@debian.org>

show-new: open a new session for every source package

Signed-off-by: Torsten Werner <twerner@debian.org>

Sigh

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'twerner/msfl' into merge

* twerner/msfl:
  introduce defaultThreadCount
  add key Common::ThreadCount to dak.conf

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Use the real path, not my local one - whoops

Signed-off-by: Mark Hymers <mhy@debian.org>

introduce defaultThreadCount

... make it an int and let it default to 1 (single threading)

Signed-off-by: Torsten Werner <twerner@debian.org>

Move franck's dak to 5433

Signed-off-by: Mark Hymers <mhy@debian.org>

What, are we still using postgres 7.4 or something?

Signed-off-by: Mark Hymers <mhy@debian.org>

Remove hardcoding of database name, port etc

Signed-off-by: Mark Hymers <mhy@debian.org>

We really don't want environment variables to cause us to drop the wrong damn database

Signed-off-by: Mark Hymers <mhy@debian.org>

Don't need a database connection for this

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge branch 'master' into msfl

add key Common::ThreadCount to dak.conf

That is the default number of threads created for the ThreadPool() but
the value can be overwritten when calling the constructor.

Signed-off-by: Torsten Werner <twerner@debian.org>

Allow getting db config out easily

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'twerner/msfl' into merge

* twerner/msfl:
  make generate-filelist multithreaded

Signed-off-by: Joerg Jaspert <joerg@debian.org>

make generate-filelist multithreaded

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'twerner/msfl' into merge

* twerner/msfl:
  add threadpool implementation
  improve multithread robustness

Signed-off-by: Joerg Jaspert <joerg@debian.org>

And more public

Signed-off-by: Joerg Jaspert <joerg@debian.org>

add threadpool implementation

... and its license information

Signed-off-by: Torsten Werner <twerner@debian.org>

improve multithread robustness

- create a separate db session per suite and component in writeSourceList()

Signed-off-by: Torsten Werner <twerner@debian.org>

And this also public. ohmy

Signed-off-by: Joerg Jaspert <joerg@debian.org>

use template1

Signed-off-by: Joerg Jaspert <joerg@debian.org>

debianqueued

rename does not work across devices. Use move from File::Copy instead.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

And one more in public

Signed-off-by: Joerg Jaspert <joerg@debian.org>

And thats in public view

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Unfortunately DSA doesnt give us all rights anymore for no good reason to do a sane backup, including our roles, so deactivate that part. :(

Signed-off-by: Joerg Jaspert <joerg@debian.org>

One would think at some point one got all

Signed-off-by: Joerg Jaspert <joerg@debian.org>

One would think at some point one got all

Signed-off-by: Joerg Jaspert <joerg@debian.org>

One would think at some point one got all

Signed-off-by: Joerg Jaspert <joerg@debian.org>

No, webdir is public, damnit

Signed-off-by: Joerg Jaspert <joerg@debian.org>

No, webdir is public

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Remove a tag

Signed-off-by: Joerg Jaspert <joerg@debian.org>

And one more path

Signed-off-by: Joerg Jaspert <joerg@debian.org>

And one more path

Signed-off-by: Joerg Jaspert <joerg@debian.org>

One more

Signed-off-by: Joerg Jaspert <joerg@debian.org>

New path stuff

Signed-off-by: Joerg Jaspert <joerg@debian.org>

More changes for franck

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' into franck

* master:
  Add the apache configs

Add the apache configs

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' into franck

* master:
  No more pgp keyring

No more pgp keyring

Signed-off-by: Joerg Jaspert <joerg@debian.org>

path changes

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'drkranz/master' into merge

* drkranz/master:
  Add a newline between two changelog entries
  Check if upload already has a changelog entry
  Replace base_filename with os.path.basename

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Add a newline between two changelog entries

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>