This describes the steps I took to create miniminidvodbs (and smaller) to try to find the corrupt SMF. Step 1: Find the cam_ids used in !ThreePi.283. You can find them with this mysql query: {{{ select cam_id from camRun join camProcessedExp using (cam_id) join addRun using (cam_id) join minidvodbRun using (minidvodb_name) where minidvodb_name like '%283'; }}} Then, take the first half and the last half and create 2 minidvodbs, using one addtool command for each cam_id. This shows 2 examples: {{{ addtool -definebyquery -destreaked -set_dvodb DEBUG.THREEPI.283.1 -set_minidvodb_group debug.ThreePi.283.1 -set_label debug.ThreePi.283.1 -dbname gpc1 -cam_id 148883 addtool -definebyquery -destreaked -set_dvodb DEBUG.THREEPI.283.2 -set_minidvodb_group debug.ThreePi.283.2 -set_label debug.ThreePi.283.2 -dbname gpc1 -cam_id 149485 }}} Edit site.config in ~ipp/ippconfig to know about DEBUG.THREEPI.283.1 and DEBUG.THREEPI.283.2 in addstars pantask: {{{ addstar.off minidvodb.merge.off add.minidvodb Debug.ThreePi.283.1 DEBUG.THREEPI.283.1 DEBUG.THREEPI.283.1 1 GPC1 add.minidvodb Debug.ThreePi.283.2 DEBUG.THREEPI.283.2 DEBUG.THREEPI.283.2 1 GPC1 add.label Debug.ThreePi.283.1 add.label Debug.ThreePi.283.2 (remove all the old labels) }}} Wait until there are entries for !Debug.!ThreePi.% in the minidvodbRun table. When there are, type this in addstars pantasks: {{{ addstar.on }}} Now run addstar and relphot: {{{ addstar -resort -D CAMERA GPC1 -D CATDIR /data/ipp004.0/gpc1/catdirs/Debug.ThreePi.283.1/ relphot -averages -update -D CAMERA gpc1 -D CATDIR /data/ipp004.0/gpc1/catdirs/Debug.ThreePi.283.1 }}} Results: * !ThreePi.283 - segfault * !ThreePi.283.1 (first half of !ThreePi.283) - completed succesfully * !ThreePi.283.2 (second half of !ThreePi.283) - segfault * !ThreePi.283.4 (second half of !ThreePi.283.2) - completed successfully * !ThreePi.283.3 (first half of !ThreePi.283.2) - segfault * !ThreePi.283.5 (half of !ThreePi.283.3) - completed successfully * !ThreePi.283.6 (half of !ThreePi.283.3) - completed successfully * !ThreePi.283.7 (half of !ThreePi.283.6, before I found out it didn't segfault) * !ThreePi.283.8 (half of !ThreePi.283.6, before I found out it didn't segfault) So, chopping in half didn't work. !ThreePi.283.3 segfaults, but it's 2 halves do not. Current attempts: * Taking a copy of !ThreePi.283.6, and do the following: 1. add a cam_id from !ThreePi.283.5 (6 and 5 make up the 2 halves of !ThreePi.283.3) 2. run relphot on !ThreePi.283.6. Does it segfault. If not, go back to step 1. * recreating !ThreePi.283.3 using the ipp-20101029 tag. Does it segfault with relphot?