\newcommand{\QueueExposuresWarping}{%
  \texttt{queue\_exposures\_for\_warping.py}}
\newcommand{\QueueExposuresStacking}{%
  \texttt{queue\_exposures\_for\_stacking.py}}

\chapter{Operators Manual}
\label{chap-operation}

\section{Condor Framework Availability on the IPP Production Cluster}

The Condor framework is not available on all hosts of the IPP
production cluster. The purpose of this section is to give some
details about the different nodes which are used and their roles in
Condor. The set of all machines on which Condor is installed is called
the \definition{Condor pool}{Condor pool}{Condor!Pool}.

The Condor \definition{central manager}{Central
  manager}{Condor!Central manager} is \texttt{ippdb03}. From the
Condor documentation (section~3.1), the role of the central manager is
to "[collect] information, and [negotiate] between resources and
resource requests". It "plays a very important part in the Condor pool
and should be reliable" since "[if it] crashes, no further matchmaking
can be performed within the Condor system". Moreover "[all] queries go
to the central manager".

The other nodes of the IPP production cluster where Condor is
installed are all both Execute and Submit Condor nodes. The complete
list of those nodes is available in the file
\texttt{\~{ }ippdor/local/share/CondorNodes}. Note that a node on which
Condor is installed is not necessary available for processing in
Condor since it can be deactivated (see section~\ref{sec-oper-hosts}
for details).

An \definition{Execute}{Execute node}{Condor!Execute node} node is a
node where processing can be performed. From the Condor manual
section~3.1, "the only resource that might matter is disk
space"... Until now, it has never been a matter.

A \definition{Submit}{Submit node}{Condor!Submit node} node is a node
where Condor jobs can be submitted (see
section~\ref{sec-ippdor-run}). It is suggested (again this is from the
Condor manual, section~3.1), that since "every job that you submit
that is currently running on a remote machine generates another
process on your submit machine [, ...]  if you have lots of jobs
running, you will need a fair amount of swap space and/or real
memory". Until now, jobs were generally submitted from one of the
\texttt{ippc\emph{xy}} nodes.

Note: For Condor gurus, there is no Checkpoint Server since all
processing is performed in the \emph{vanilla} Condor universe.

\section{Condor Dedicated User: \texttt{ippdor}}
\label{sec-oper-login}

To use the Condor framework you need to log into the IPP production
cluster as the user \texttt{ippdor}. Note that since Condor is not
installed on \texttt{ippc18}, you will have to login again into
another node for the condor commands to be usable. \texttt{ipp004} or
\texttt{ippdb03} are other "open" nodes where you can connect.

Note: The \texttt{ippdor} account uses \texttt{bash}.

Theoretically it should be possible to use the Condor framework with
your own login but 1) It has not been tested; 2) No support will be
provided for that; 3) So... RTCM.

As detailed in section~\ref{sec-adm-ippdor}, the \texttt{ippdor}
account uses files that may be localized on the Condor node. If you
get a \texttt{command not found} error, make sure you are on one of
the node where Condor is installed\footnote{that is, the command
  \texttt{"grep \`{ }hostname\`{ } \~{
    }ippdor/local/share/CondorNodes"} should display the host name. If
  you don't see anything, read the first paragraph of this section.}.

\section{Managing Condor Execution on the IPP Production Cluster
  Nodes}
\label{sec-oper-hosts}

Note 1: To install Condor on a new node, see
section~\ref{sec-adm-newnode}.

Note 2: Except for a few home-grown commands, information was
extracted from the Condor Manual section~3.10.

\subsection{Node Availability}
\label{sec-oper-hosts-availability}
\definition{}{Node availability}{Node availability}%
\definition{}{}{Condor!Node availability}%
To check if a node can be used by Condor, check the result of the
\texttt{ipp\_condor\_status <hostname>} command. You should get some
information about the host from the Condor point of view. For
instance, \texttt{ipp060} potentially has 12 potential processors for
Condor execution\footnote{To get more information about what the
  columns mean, RTCM}:
\begin{verbatim}
ippdor@ippc20 ~ $ ipp_condor_status ipp060
Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot10@ipp060.ifa. LINUX      X86_64 Unclaimed Idle     0.000  4032  0+05:54:29
slot11@ipp060.ifa. LINUX      X86_64 Owner     Idle     0.400  4032  0+00:00:06
slot12@ipp060.ifa. LINUX      X86_64 Owner     Idle     1.000  4032  0+00:00:07
slot1@ipp060.ifa.h LINUX      X86_64 Owner     Idle     0.840  4032  0+00:00:04
slot2@ipp060.ifa.h LINUX      X86_64 Unclaimed Idle     0.000  4032  0+14:34:09
slot3@ipp060.ifa.h LINUX      X86_64 Unclaimed Idle     0.000  4032  0+06:56:32
slot4@ipp060.ifa.h LINUX      X86_64 Unclaimed Idle     0.000  4032  0+14:25:44
slot5@ipp060.ifa.h LINUX      X86_64 Unclaimed Idle     0.000  4032  0+14:33:57
slot6@ipp060.ifa.h LINUX      X86_64 Claimed   Idle     0.000  4032  3+11:22:07
slot7@ipp060.ifa.h LINUX      X86_64 Owner     Idle     1.000  4032  0+00:00:29
slot8@ipp060.ifa.h LINUX      X86_64 Owner     Idle     0.790  4032  0+00:00:03
slot9@ipp060.ifa.h LINUX      X86_64 Unclaimed Idle     0.000  4032  0+06:55:39
                     Total Owner Claimed Unclaimed Matched Preempting Backfill

        X86_64/LINUX    12     5       1         6       0          0        0

               Total    12     5       1         6       0          0        0
\end{verbatim}

If you don't get anything, the node has been deactivated for Condor
processing). For instance (while the author writes these lines),
\texttt{ipp065} will not execute any Condor jobs:
\begin{verbatim}
ippdor@ippc20 ~ $ ipp_condor_status ipp065
ippdor@ippc20 ~ $ 
\end{verbatim}

\emph{Note:} \texttt{ipp\_condor\_status} is an alias of
\texttt{condor\_status} defined in \texttt{\~{ }ippdor/.bashrc}.

\subsection{Make a Node Unavailable}
\label{sec-oper-hosts-remove}

To remove a node from the list of execution nodes, execute the command
\texttt{ipp\_condor\_off <hostname>} from any node where Condor is
installed. No job will be started/executed and submitted from that
host. \texttt{ipp\_condor\_off} is a function defined in \texttt{\~{
  }ippdor/.bashrc}.

To stop Condor execution and submission from \definition{all}{Stopping
  Condor on stare nodes}{Stopping Condor on stare nodes}stare nodes,
the \texttt{stare\_nodes\_off} and \texttt{stare\_nodes\_on} scripts
are provided. As it can be expected, those scripts just call
\texttt{ipp\_condor\_off} or \texttt{ipp\_condor\_on} for each of the
stares nodes.

\subsection{Make a Node Available}
\label{sec-oper-hosts-available}

To make a node available for Condor after it has been stopped, execute
the command \texttt{ipp\_condor\_on <hostname>} from any node where
Condor is installed.

\emph{Note:} If the following message is displayed,
\begin{verbatim}
bash: condor_master: command not found
\end{verbatim}
it simply means that Condor is not installed on the node where you try
to start it.

\section{Condor Jobs and DAG}

\subsection{Running Jobs in Condor}
\label{sec-oper-jobs}
We will call \definition{job}{job}{job} an elementary unit of
processing. It consists of the execution of one instance of an
executable (we will assume that an executable is a program that can be
called directly from the command line). For instance, "/bin/echo One"
and "/bin/echo Two" are two jobs, each job being an instance of the
same executable "/bin/echo".

In our case, defining a correct Condor job is as simple (not to say
simplistic) as defining:
\begin{itemize}
\item The name of the executable;
\item The Condor universe of execution (in the IPP case, to avoid
  recompilation of programs we use the \emph{vanilla} universe);
\item The file where the standard output of the executable will go (on
  an NFS directory since we don't know where the executable will be
  run);
\item The file where the error output of the executable will go (on
  an NFS directory since we don't know where the executable will be
  run);
\item The file where Condor will write the log of the job (on a local
  directory);
\item The requirements about the node where execution will take place;
\item Some argument to tell that we want to keep the environment
  variables;
\item The arguments for each execution of the executable.
\end{itemize}

With the example given on \texttt{/bin/echo}, the contents of the
condor job file (let's call this file \texttt{echo\_test.condor}) are:
\begin{verbatim}
Executable   = /bin/echo
Universe     = vanilla
Output       = /home/panstarrs/ippdor/tmp/tests/out.echo_test
Error        = /home/panstarrs/ippdor/tmp/tests/err.echo_test
Log          = /tmp/echo_test.log
Requirements = LoadAvg < 0.3

getenv       = True

Arguments    = One
Queue

Arguments    = Two
Queue
\end{verbatim}

To execute those jobs, they just need to be \definition{submitted}{Job
  submission}{Job submission} from any Submit node to the Condor
system with:

\texttt{condor\_submit echo\_test.condor}

The jobs are (usually) not executed immediately and there is no way of
guessing the order on the jobs execution (if the order of jobs
execution is a matter, look at paragraph~\ref{sec-oper-tasks}).

Note: From what the author experienced, \texttt{condor\_submit} and
the condor jobs definition file can be executed and placed on any kind
of file system (local and/or NFS) which is different from
\texttt{condor\_submit\_dag}.

\subsection{Running tasks}
\label{sec-oper-tasks}

\subsubsection{DAGMan}

We will call \definition{task}{task}{task} a set of jobs where jobs
chaining depend on each other. Condor provides a mechanism of
organizing jobs according to their dependencies:
\definition{DAGMan}{DAGMan}{DAGMan} (which stands for Directed Acyclic
Graph Manager). Exhaustive documentation about DAGMan can be found in
the Condor documentation section 2.10. In this section I will just
give details about what is used for the IPP.

\subsubsection{DAG Description File}

To make it simple, a DAG is a high-level description of how jobs are
sequenced. If the job B has to be performed after the job A, the
dependency is formalized this way:
\begin{verbatim}
JOB    A     /some/directory/A.condor
JOB    B     /some/directory/B.condor
PARENT A CHILD B
\end{verbatim}

Assuming the previous text is saved in the file named \texttt{AB.dag},
the execution of the DAG can be \definition{submitted}{DAG
  submission}{DAG submission} from any Submit node to the Condor
system using:

\texttt{condor\_submit\_dag AB.dag}

The output of that command will look like:
\begin{verbatim}
-----------------------------------------------------------------------
File for submitting this DAG to Condor           : AB.dag.condor.sub
Log of DAGMan debugging messages                 : AB.dag.dagman.out
Log of Condor library output                     : AB.dag.lib.out
Log of Condor library error messages             : AB.dag.lib.err
Log of the life of condor_dagman itself          : AB.dag.dagman.log

Submitting job(s).
1 job(s) submitted to cluster 71201.
-----------------------------------------------------------------------
\end{verbatim}

All those files are created in the directory of the \texttt{AB.dag}
file. The \texttt{AB.dag.dagman.out} file is the most interesting log
file. 

\subsubsection{Files and Directories Locations}

The DAG file can be localized on any file system: local or
NFS. However for performance concerns, I recommend to put it on the
local file system of the submit node. Monitor to the log growth rate
(the logs are not huge though... For instance, full processing of 2300
MD06 exposures from Chip to Warp stages in the IPP created a log of 15
MB for the \texttt{.out} file and of a total of about 50 kB for the 4
other files).

The \texttt{/some/directory} directory (where the condor jobs are
defined) can be placed on any file system. Again, I suggest to put
them on the local file system of the Submit node (mostly for
performance concerns).

\begin{center}
  \fbox{\parbox{6in}{\textbf{Important note:} The
      \texttt{condor\_submit\_dag} command \textbf{must} be executed
      \textbf{from a NFS file system} or the jobs will be indefinitely
      \texttt{Held} (shown with \texttt{H} status in
      \texttt{condor\_q}. See section~\ref{annex-heldjobs} for more
      details.}}
\end{center}

One restriction is that there cannot be cycles, i.e. circular
dependencies. For instance, job Egg depends on job Chicken, which
depend on job Egg. If you submit a DAG with a cycle, looking at the
\texttt{dagman.out} file, you will see a message like:
\begin{verbatim}
[...]
11/25/11 12:22:02 ERROR: DAG finished but not all nodes are complete -- 
                  checking for a cycle...
11/25/11 12:22:02 ... ERROR: a cycle exists in the dag, plese check input
[...]
11/25/11 12:22:03 **** condor_scheduniv_exec.71211.0 (condor_DAGMAN) 
                  pid 20996 EXITING WITH STATUS 1
\end{verbatim}

\subsubsection{Advanced DAGMan Features}
\label{sec-oper-dagman}

DAGMan offers other convenient functions for a DAG description. As for
the other Condor features the reader is invited to check the Condor
documentation for in-depth details about them.

\paragraph{SubDAG:} A graph node of a DAG can be itself a DAG making
the development and the analysis of a processing chain easier. A DAG
in a DAG is called a \definition{SubDAG}{SubDAG}{SubDAG} and can be
added to a DAG using this syntax:
\begin{verbatim}
SUBDAG EXTERNAL <job name> <dag file name>
\end{verbatim}

\paragraph{Script:} Scripts canbe executed before or after a node
execution. They can be added using the following syntax:
\begin{verbatim}
SCRIPT PRE <job name> <Executable name> [<arguments>]
SCRIPT POST <job name> <Executable name> [<arguments>]
\end{verbatim}

From the Condor documentation, "the \texttt{<job name>} specifies the
node to which the script is attached. The \texttt{<Executable name>}
specifies the script to be executed, and it may be followed by any
command line \texttt{<arguments>} to that script". In the IPP context
we currently use only the \texttt{SCRIPT POST} feature.

\subsection{Condor Interrupted}
\label{sec-oper-condor-interrupted}

If the execution of Condor DAG is interrupted, for instance, because
of an operator \texttt{condor\_rm} or because of errors, Condor marks
the jobs or subdags that were successfully completed by adding the
\texttt{DONE} keyword to the DAG file. The resulting DAG file is saved
in a \definition{rescue DAG file}{Rescue DAG file}{Rescue DAG file},
which is simply a copy of the original DAG file with a
\texttt{rescue<running ID>} file extension (\texttt{<running ID>} is
an auto-incrementing integer coded on three digits, starting at
\texttt{001}). 

To \definition{rescue processing}{Rescuing processing}{Rescuing
  processing} you just need to execute \textbf{exactly} the same
submission command: Condor will use the last rescue file that was
produced.

Note: If you know what you are doing you can create you rown rescue
file or add/remove \texttt{DONE} keywords to/from a rescue file. Some
utility programs can do that for you for some IPP stages
(see~\ref{TODO}).

\section{IPP Tasks}
\label{sec-oper-ipp}

The IPP defines complex tasks. The purpose of this section is to
provide a user manual about the programs that need to be run to
prepare the set of Condor files which are needed to perform some of
the IPP tasks. Developers should look at chapter~\ref{chap-software}
for in-depth details.

Each subsection of this section details: 1) The software role; 2) Its
required inputs; 3) Its outputs; 4) Its IPP products.

Currently the IPP parts that are implemented are:
\begin{itemize}%
  \setlength{\itemsep}{0pt}\setlength{\topsep}{0pt}\setlength{\partopsep}{0pt}\setlength{\parsep}{0pt}%
\item Chip to Warp stages (paragraph~\ref{sec-oper-ipp-chip2warp});
\item Chip to Stack stages (paragraph~\ref{sec-oper-ipp-chip2stack});
\end{itemize}

\subsection{Chip to Warp Processing: \QueueExposuresWarping{}}
\label{sec-oper-ipp-chip2warp}

Since this section is supposed to belong to the operators manual, no
detail about the software will be provided. The developer will read
section~\ref{sec-TODO} for such details.

\subsubsection{Role}
The role of \QueueExposuresWarping{} is to prepare the set of Condor
and IPP elements which are required to process a set of exposures from
the Chip to the Warp stage.

The \QueueExposuresWarping{} script has four roles:
\begin{enumerate}[topsep=0pt, partopsep=0pt, itemsep=0pt, parsep=0pt, itemindent=10pt]
\item Assert that the label has not already been used and that the
  input exposures exist;
\item Create entries in the \texttt{chipRun} table;
\item Create the DAG files which are necessary for the processing of
  all exposures;
\item Display the \texttt{submit\_condor\_dag} command that needs to
  be run by the operator.
\end{enumerate}

Note that the operator has to run the \texttt{submit\_condor\_dag}
command.

\subsubsection{Submit Node}

A node needs to be chosen as Submit node. TODO: add details.

\subsubsection{Inputs}
The inputs are:
\begin{itemize}
\item A \textbf{label} The label used for processing. If the label has
  already been used, the program is aborted. As for the processing in
  IPP, don't use spaces or non-ASCII characters for the label (for
  instance, \texttt{My.New\_Label});
\item An \textbf{e-mail address} who gets the e-mails (for instance,
  \texttt{someone@mail.com});
\item A \textbf{tessellation ID} (e.g. \texttt{MD04.V3});
\item A \textbf{list of exposures} to be processed (for instance,
  \texttt{o4926g0064o}, \texttt{o4926g0068o}, \texttt{o4926g0069o},
  \texttt{o4926g0071o}, and \texttt{o4940g0237o}).
\end{itemize}

The \QueueExposuresWarping{} script can be called in two ways:
\begin{itemize}
\item Either they are provided in a file which has the format of a
  Properties-style file. With the previous parameters:
\begin{verbatim}
label     = My.New_Label
tessId    = MD04.V3
recipient = someone@mail.com
exposure  = o4926g0064o
exposure  = o4926g0068o
exposure  = o4926g0069o
exposure  = o4926g0071o
# Comment line
exposure  = o4940g0237o  # Line followed by a comment
\end{verbatim}
  Assume that the file is stored under the name 
  \texttt{~{ }ippdor/examples/ChipsToWarps.ippdor} 
  the script is called with (note the \texttt{-f} option flag):
\begin{verbatim}
queue_exposures_for_warping.py -f ~ippdor/examples/01/ChipsToWarps.ippdor
\end{verbatim}
\item Or the input parameters are provided as arguments, that is (the
  arguments have to be in this order):
\begin{verbatim}
queue_exposures_for_warping.py My.New_Label MD04.V3 someone@mail.com \
      o4926g0064o o4926g0068o o4926g0069o o4926g0071o o4940g0237o
\end{verbatim}
\end{itemize}

\subsubsection{Outputs. Entries on GPC1 Database}
The script writes its outputs to the directory named
\texttt{<LOCAL\_TMP\_DIR>/<label>} where
\definition{\emph{\texttt{<LOCAL\_TMP\_DIR>}}}{\texttt{LOCAL\_TMP\_DIR}}{\texttt{LOCAL\_TMP\_DIR}}{}
is defined in \IppPython{Constants.py} and \texttt{<label>} is the
label which has been provided to the script. For instance, with the
current configuration if executed on \texttt{ippc10} in the
\texttt{ippdor} production environment, and for the label
\texttt{My.New\_Label}, the various files will be written to the
directory \texttt{/export/ippc10.0/ippdor/processing/My.New\_Label}.

The main file, also called DAG of DAGs, is \texttt{ch2wp.dag}. Its
name is defined by
\texttt{ChipWarpBuilder.Constants.DAG\_OF\_DAGS\_FMT}. This DAG lists
all the steps which are necessary to build the IPP products from the
list of exposures.

The script also creates one directory for each exposure in
\texttt{<LOCAL\_TMP\_DIR>}. 


\subsubsection{IPP Products}

Upon successfull completion, the IPP Products are those which are
produced by successfull chip, cam, fake, and warp stages. Look at the
IPP documentation to find at what they are.

If the execution of a SubDAG fails, an e-mail is sent to the
\texttt{recipient} address telling about the nature of the failure. To
avoid too many failures each processing stage is reverted up to 3
times. The failure e-mail is therefore sent after 3 failed revert
attempts. It is expected from the operator to fix the problem and
\textbf{manually} revert \textbf{all} failures before restarting the
Condor processing.

%%%%%%%%%%%%%%%%%%% BARF

\subsection{Chip to Stack}
\label{sec-oper-ipp-chip2stack}


\paragraph{Required Inputs}

The name of the utility is \QueueExposuresStacking. Running it requires
parameters. Those parameters can be provided through command line
arguments or in a file (which filename is provided through the command
line interface). Those parameters are:
\begin{itemize}
\item \texttt{label} The label used for processing. If the label has already
  been used, the program is aborted. As for the processing in IPP,
  don't use spaces or non-ASCII characters for the label.
\item \texttt{tesselation ID} like MD08.V2
\item \texttt{E-mail address} who gets the e-mails
\item \texttt{List of exposures}
\end{itemize}

The file format is a Properties-style file, for instance:
\begin{verbatim}
label     = My.New_Label
tessId    = MD06.V3
recipient = someone@mail.com
exposure  = o4926g0064o
exposure  = o4926g0068o
exposure  = o4926g0069o
exposure  = o4926g0071o
# Comment line
exposure  = o4940g0237o  # Followed by a comment
\end{verbatim}

The command-line equivalent is:
\begin{verbatim}
queue\_exposures\_for\_stacking.py \
    My.New_Label MD06.V3 someone@mail.com \
    o4926g0064o o4926g0068o o4926g0069o o4926g0071o o4940g0237o
\end{verbatim}

\paragraph{What \QueueExposuresStacking{} does}
\begin{enumerate}
\item It checks that each exposure exists;
\item It creates the appropriate entries in the \texttt{chipRun} table;
\item It generates the various Condor DAGs: one global DAG and one DAG
  for each exposure. The files are located
\item It displays the \CondorSubmitDag{} command that has to be run
\end{enumerate}

\section{Running a task}
\label{ippdor-run}
The \CondorSubmitDag{} command has to be run on the host where
\QueueExposuresStacking{} was run. But the \CondorSubmitDag{} has to
be run from a NFS directory (e.g. the \texttt{ippdor} home directory)
otherwise Condor will hold jobs.

The output of the \CondorSubmitDag{} command looks like\footnote{Long lines were wrapped}: 
\begin{verbatim}
Can't open directory "/export/ippc20.0/ippdor/condor_local/config" as \
   PRIV_UNKNOWN, errno: 2 (No such file or directory)
-----------------------------------------------------------------------
File for submitting this DAG to Condor           : \
  /export/ippc20.0/ippdor/processing/condor_MD06.V3_01/ch2st.dag.condor.sub
Log of DAGMan debugging messages                 : \
  /export/ippc20.0/ippdor/processing/condor_MD06.V3_01/ch2st.dag.dagman.out
Log of Condor library output                     : \
  /export/ippc20.0/ippdor/processing/condor_MD06.V3_01/ch2st.dag.lib.out
Log of Condor library error messages             : \
  /export/ippc20.0/ippdor/processing/condor_MD06.V3_01/ch2st.dag.lib.err
Log of the life of condor_dagman itself          : \
  /export/ippc20.0/ippdor/processing/condor_MD06.V3_01/ch2st.dag.dagman.log

Submitting job(s).
1 job(s) submitted to cluster 62384.
-----------------------------------------------------------------------
\end{verbatim}

Don't bother about the warning on the first line. You should have a
look (or even \texttt{tail~-f}) the \texttt{ch2st.dag.dagman.out} file
since it is the place where Condor dagman is logging. It tells how the
jobs progress, what is happening in terms of successes and failures,
if jobs are held...

Note the \texttt{1 job(s) submitted to cluster 62384} last line. It
tells the Condor cluster id which is the mother of all other Condor
jobs in the DAG. More precisely, if you want to stop the Condor
processing (see~\ref{ippdor-rm}), you can either stop each Condor job
individually (somewhat annoying when you have 2000 jobs), or stop the
mother.

\section{Supervising a Task Execution}

As mentioned in the previous paragraph, the
\texttt{ch2st.dag.dagman.out} file is the file to look at to get
information from Condor.

The \texttt{condor\_q} command can be used to monitor the
progress. The \texttt{long\_condor\_q} command is a home-grown command
that displays more information about the jobs which are running (it
doesn't display anything about idle or held jobs).

\texttt{condor\_q -analyze <job\_id>} can be useful to get more
information about a job.

\section{Ending a task}

\subsubsection{Regular}
\label{ippdor-end}

When \texttt{condor\_q} does not show the Condor cluster ID.

\subsection{Forced Termination}
\label{ippdor-rm}

The command to stop a Condor job is \texttt{condor\_rm}. 

An argument is necessary. You will generally use the Condor cluster ID
that you got from the \CondorSubmitDag{} command. For instance, to
stop all processing from example shown in paragraph~\ref{ippdor-run}:
\begin{verbatim}
condor_rm 62384.0
\end{verbatim}

The \texttt{-all} option also exists. Use at your own risk... You're
maybe not the only one to run a Condor instance.

\section{FAQ}

\begin{description}
\item[All jobs are held] 
\end{description}

