First pass at extra_src_references table

Signed-off-by: Mark Hymers <mhy@debian.org>

Don't contradict ourselves within 2 lines

Signed-off-by: Mark Hymers <mhy@debian.org>

add mvo quote

Signed-off-by: Mark Hymers <mhy@debian.org>

Prepare for pg 9.0

Signed-off-by: Mark Hymers <mhy@debian.org>

gps timeout/error handling

return the result of the a-f call.
additionally get it using a timeout.
when either a TimeoutError occurs or the result is non-zero, complain about it
and then exit non-zero, so dinstall could stop.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

mhy, das kruemelmonster

Signed-off-by: Joerg Jaspert <joerg@debian.org>

database backup foo

no longer dump the full database on each dinstall run. instead we just
save a transaction id. DSA is doing the backup dumps in whatever cycle,
and keeping the WAL stuff around, so we can tell them to get us to
whatever point in our db which we might need, ie. the specific transaction
id.

as those are just 9 bytes per file, and not 4gigabytes, we now also save
wth we have been before and after unchecked runs, which makes our backup
be nearer to actual filesystem state than ever.

Signed-off-by: Joerg Jaspert <joerg@debian.org>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

remove pgdakdev

Signed-off-by: Joerg Jaspert <joerg@debian.org>

parse scriptvars, not . vars

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Fix a comment in dbconn.py.

Signed-off-by: Torsten Werner <twerner@debian.org>

Fix copy-paste error.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ftp-master.debian.org:/srv/ftp.debian.org/git/dak

Ignore more warnings from python-apt.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
  Add filename de-duplication to contents scanning.

updates

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'dbtests'

Add filename de-duplication to contents scanning.

Signed-off-by: Torsten Werner <twerner@debian.org>

byhand-di: allow YYYYMMDD+<suite>x version numbers

To: debian-dak@lists.debian.org
Cc: Julien Cristau <jcristau@debian.org>, Otavio Salvador <otavio@ossystems.com.br>
Date: Mon, 14 Mar 2011 14:11:39 +0100
X-Spam-Status: No, score=-1.775 tagged_above=-99 required=4.6 tests=[BAYES_00=-1.9, SARE_SUB_OBFU_OTHER=0.135, T_RP_MATCHES_RCVD=-0.01] autolearn=no
X-Mailer: git-send-email 1.7.2.5
Resent-From: debian-dak@lists.debian.org

stable updates need to have a higher version than binNMUs, but
YYYYMMDD<suite>x << YYYYMMDD+b1.

Signed-off-by: Julien Cristau <jcristau@debian.org>
Cc: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

Break hardlink when writing Contents files.

Signed-off-by: Torsten Werner <twerner@debian.org>

we do want -n1, not -n16. we want as many parallel xz as we can

Signed-off-by: Joerg Jaspert <joerg@debian.org>

dinstall - mirror

exclude the temporary files from apt-ftparchive (*.new), and delete excluded
files too.
Should keep the mirrors happy, cos those files arent readable for others.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Handle empty packages in 'dak contents scan'.

Signed-off-by: Torsten Werner <twerner@debian.org>

allow squeeze-updates to be reset

Signed-off-by: Mark Hymers <mhy@debian.org>

Switch to 'dak contents generate'.

It replaces APT::FTPArchive::Contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

Enable the new contents generation for experimental.

... as a first test

Signed-off-by: Torsten Werner <twerner@debian.org>

Write full section to Contents files.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' into dbtests

Add workaround for duplicate filenames in contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
  bugfixes
  Contents: support override suites.

dinstall - use xz for pg dumps, and use a | xargs instead of while

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
  Use 'dak contents generate_helper'.
  bugfixes
  Add a new subcommand 'generate_helper' to 'dak contents'.
  Switch to multiprocessing module and reap child gzips.

dinstall - dont recompress .xz files

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Contents: support override suites.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' into dbtests

Use 'dak contents generate_helper'.

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Add a new subcommand 'generate_helper' to 'dak contents'.

Signed-off-by: Torsten Werner <twerner@debian.org>

Switch to multiprocessing module and reap child gzips.

Signed-off-by: Torsten Werner <twerner@debian.org>

And make the whitespace something useful

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Fixup the regex

Signed-off-by: Joerg Jaspert <joerg@debian.org>

goodbye comments

Signed-off-by: Joerg Jaspert <joerg@debian.org>

during unchecked no longer recreate testing indices, only do them when importing new data there

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Scan 10.000 files per unchecked run.

Signed-off-by: Torsten Werner <twerner@debian.org>

Include all files (except directories) in bin_contents table.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'dbtests'

Add missing 'dists' directory to method output_filename().

Signed-off-by: Torsten Werner <twerner@debian.org>

Fully rename bootstrap_bin to scan.

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Rewrite most of dak/contents.py.

Signed-off-by: Torsten Werner <twerner@debian.org>

Update ContentsTestCase to the current formatting.

Signed-off-by: Torsten Werner <twerner@debian.org>

some bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Implement class method ContentsWriter.write_all().

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Add header to Contents files.

Signed-off-by: Torsten Werner <twerner@debian.org>

Save a dummy filename for packages without files.

Signed-off-by: Torsten Werner <twerner@debian.org>

Call dak contents bootstrap_bin from unchecked.

It is limited to 1000 packages per run.

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Rewrite 'dak contents bootstrap_bin' to use ContentsScanner.

Signed-off-by: Torsten Werner <twerner@debian.org>

Yield utf-8 encoded strings from DBBinary.scan_contents().

Signed-off-by: Torsten Werner <twerner@debian.org>

Fix contents testcase.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' into dbtests

Conflicts:
tests/dbtest_contents.py

bugfix

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'dbtests' of ftp-master.debian.org:public_html/dak into dbtests

Add workaround for Unicode support in database.

Signed-off-by: Torsten Werner <twerner@debian.org>

s/colour/color/

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Add and enable data collection and graphing for the DEFERRED queues.

Link to the graphs instead of including them in deferred.html

Merge branch 'dbtests'

Add a test for the new cascade rule.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add an explicit cascade rule for BinContents.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add an explicit cascade rule for BinContents.

Signed-off-by: Torsten Werner <twerner@debian.org>

Make SQLAlchemy's pool parameters configurable.

Signed-off-by: Torsten Werner <twerner@debian.org>

Return some useful values from scan_all().

Signed-off-by: Torsten Werner <twerner@debian.org>

Reap zombie children in scan_contents().

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'dbtests' into merge

Add classes ContentsWriter and ContentsScanner as a preparation for the new
contents command.

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfix

Signed-off-by: Torsten Werner <twerner@debian.org>

Support utf-8 encoding as a primary encoding for filenames.

Signed-off-by: Torsten Werner <twerner@debian.org>

Assume that all filenames are iso8859-1 encoded.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add missing filter for query in scan_all().

Signed-off-by: Torsten Werner <twerner@debian.org>

Add new class ContentsScanner.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add method scan_contents() to class DBBinary.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add small test package hello_2.2-1_i386.deb.

Signed-off-by: Torsten Werner <twerner@debian.org>

Implement and test class ContentsWriter.

Signed-off-by: Torsten Werner <twerner@debian.org>

s/MD5/MD5Sum/

Signed-off-by: Joerg Jaspert <joerg@debian.org>

its MD5Sum

Signed-off-by: Joerg Jaspert <joerg@debian.org>

dak.conf, checksums

for now put md5 back into all release files. (pending deletion sometime
not too far away)

Signed-off-by: Joerg Jaspert <joerg@debian.org>

add squeeze-updates to md5

Signed-off-by: Joerg Jaspert <joerg@debian.org>

generate-releases

add a list of hashes for generate-release with the suite names they are for
remove the double indirection with print_HASH_files, directly call print_hash_files
and use the hashfuncs dict to know which hash function to call

Signed-off-by: Joerg Jaspert <joerg@debian.org>

drop generation of md5sum output in release files

Signed-off-by: Joerg Jaspert <joerg@debian.org>

generate-releases

include debian-installer MD5SUMS files into our release files. (closes: #611087)

Signed-off-by: Joerg Jaspert <joerg@debian.org>

currently no deferred queue graphs

Signed-off-by: Joerg Jaspert <joerg@debian.org>

graphs are just one link to the stat.html

Signed-off-by: Joerg Jaspert <joerg@debian.org>

fixup code

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Write the RRD files to webdir/stat and enable graph rendering.

Improve the label text and formatting of the labels for the queue graphs.

Rename the queue graph labels to ones shorter and of equal length.

Don't attempt to merge the numbers into the label, use cur instead.

Also use labels that are more indicative of their meaning.

Don't force the numbers to 3 digits in length.

Turn Min/Max/Avg in the labels to lower case.

Improve the title of the queue graphs.

Include the correct path to the graph images in new.html

Make rrdtool errors non-fatal to reduce distruption.