Changes between Version 2 and Version 3 of PS1_IPP_Czarlog_20190304

Timestamp:

Mar 4, 2019, 6:15:07 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190304

@@ -6 +6 @@
 === Monday : 2019.03.04 ===
 
-JRF: ippb21 was unreachable this morning. This is after one of the SSDs had been replaced by Haydn. Rebooted it twice, seems to be behaving after the second reboot.
+JRF (morning): ippb21 was unreachable this morning. This is after one of the SSDs had been replaced by Haydn. Rebooted it twice, seems to be behaving after the second reboot.
 
-ippb22 has also started reported similar problems. So we, JRF & EAM, have asked Haydn to have a look or think about what else might be a problem on those machines.
+JRF (morning): ippb22 has also started reported similar problems. So we, JRF & EAM, have asked Haydn to have a look or think about what else might be a problem on those machines.
+(update) - Both b21 and b22 have now have disks replaced and been rebooted. They are behaving well at the moment so they have been set to repair in nebulous so that they get used.
+
+
+'''Czar training:'''
+
+Today we are changing the configuration of PS1 stdscience so that the warp stage will be targeting specific machine and 
+
+The file that contains the config for which machines are targeted for which stage is:
+{{{
+ipphosts.mhpcc.config
+}}}
+
+Located in the src directory directory:
+{{{
+~ippitc/src/ipp-20170121/ippTasks
+}}}
+
+Best practice is to create a backup of the current version, and another to modify and eventually replace the current one with.
+
+
+At the moment ipp123-126 should now be targeted for any warp processing. Warps use the skycells, so this is the meta data we should change. We are mapping these skycells to these 16 specified machines. The remaining skycells will repeated with pattern. The new entry:
+{{{
+ipphosts METADATA
+  camera STR skycell
+  count S32 16 
+  sky00 STR  ipp123.0
+  sky01 STR  ipp124.0
+  sky02 STR  ipp125.0
+  sky03 STR  ipp126.0
+  sky04 STR  ipp123.1
+  sky05 STR  ipp124.1
+  sky06 STR  ipp125.1
+  sky07 STR  ipp126.1
+  sky08 STR  ipp123.0
+  sky09 STR  ipp124.0
+  sky10 STR  ipp125.0
+  sky11 STR  ipp126.0
+  sky12 STR  ipp123.1
+  sky13 STR  ipp124.1
+  sky14 STR  ipp125.1
+  sky15 STR  ipp126.1
+END
+}}}
+
+Note that there is a similar setup for the current mops tasks:
+{{{
+~ippmops/psconfig/ipp-20170121.lin64/share/pantasks/modules
+}}}
+
+
+Now that we have changed the source we would normally have to rebuild...however this is dangerous as it will incorporate any changes that have been checked in (the in production could should not really be modified, or at least this config should be separate...but these are separate issues). We can skip all that by just putting it into the production code:
+{{{
+~ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules
+}}}
+To incorporate the changes in stdscience pantasks we need to restart it.
+
+
+'''Manually restarting a pantasks with a new label added to config (and also how not to):'''
+
+On purpose, we have gone to the machine where ps1 pantasks stdscience is running and started the server (to know where it is running check the {{{ippitc/stdscience/ptolemy.rc}}}):
+{{{
+start_server.sh stdscience
+}}}
+
+This means there are two pantasks servers running now, which is not a great state to be in. You can confirm by checking the process list
+{{{
+ps -ef | pantasks_server
+}}}
+If there are two entries listed -> you have two servers running.
+
+The issue with this is that when running pantasks_client you might end up connecting to either one of the servers, and if you are in the more recently created one then 'status' might not work and you can't see jobs that could be running on the other one. The best way to get out of this situation is to shut them both down:
+{{{
+$> pantasks_client 
+pantasks: stop
+pantasks: shutdown
+}}}
+Between the stop and shutdown check that the jobs have finished, if any, by querying the 'status' or 'controller status'
+
+
+
+For the second part of this task we want a label to be added automatically each time pantasks stdscience starts up. To do this add the following line to the 'input' file in the stdscience directory:
+{{{
+add.label update.OSS
+}}}
+
+To restart the pantasks_server, do what we did before:
+{{{
+start_server.sh stdscience
+}}}
+
+Then to load the config and start it running, do the following in pantasks:
+{{{
+pantasks: server input input
+pantasks: setup
+pantasks: run
+}}}
+The first loads the modified config, the second activates all the tasks with the loaded config, then run actually starts the server running.
+
+This new label currently has the highest priority, as it defaults to highest if there is no entry... this is not what we want. To add the label to the list of labels, and set the priority, use the label tool!
+To check if it is present first:
+{{{
+labeltool -dbname gpc1 -listlabel -simple | grep update
+}}}
+
+Add label and set priority:
+{{{
+labeltool -dbname gpc1 -definelabel -set_label update.OSS -set_priority 300
+}}}
+In this case we want it to be a lower priority than MOPS.2 which was 310, so we chose 300.
+
+Give it a few minutes and ippmonitor should reflect these changes by showing it in the correct place.
+
+
+'''Other Things:'''
+
+Some weird spikes in loads going off at 16:30...not sure why.
+
   
 === Tuesday : 2019.03.05 ===
+
+
 
 === Wednesday : 2019.03.06 ===