Next: 11. Command Reference Manual
Up: 10. Version History and
Previous: 10.4 Development Release Series
10.5 Stable Release Series 8.4
This is a stable release series of HTCondor.
As usual, only bug fixes (and potentially, ports to new platforms)
will be provided in future 8.4.x releases.
New features will be added in the 8.5.x development series.
The details of each version are described below.
- HTCondor version 8.4.12 not yet released.
- HTCondor version 8.4.11 released on January 23, 2017.
- Added a new config knob, IGNORE_LEAF_OOM, which defaults to
False. When True, it tells HTCondor not to
kill and hold a job that is within its memory allocation, even if
other processes within the same cgroup have exceeded theirs.
- Fixed a bug where the effects of a startd cron job would not be used by
the condor_startd when making policy decisions (e.g., evaluating the
START expression) until an UPDATE_INTERVAL had passed. This
was generally only noticeable if you set STARTD_CRON_AUTOPUBLISH
to a value other than NEVER, which could cause the startd to reject claims
from the negotiator that had been made based on the cron-updated value(s).
- Fixed a bug in pslot preemption where it could cause matching jobs to
not start for a long time.
- Fixed a bug that caused a job to not be cleaned up when the job
lease expires, if glexec is in use.
- Fixed a problem, found while testing on Ubuntu 16, where the negotiator
would crash at the end of the negotiation cycle.
- Updated the default configuration in the Debian and Ubuntu packages to
look for the Ganglia shared libraries in the proper place.
- Updated the Enterprise Linux 7 RPM to require the proper SELinux
utilities for its post-install script.
- HTCondor version 8.4.10 released on December 13, 2016.
- Added additional SELinux type enforcement rules on Enterprise Linux 7 for
the condor_shared_port daemon and Linux tuning script. The RPM post install
script makes the HTCondor SELinux domains permissive.
- Fixed a performance problem in the condor_schedd when
RequestCpus was an expression. Added a new parameter
SCHEDD_SLOT_WEIGHT, which may be needed
if SLOT_WEIGHT is not the default value of "Cpus", and refers to
expressions in the job ad.
- When transferring a job's sandbox, the permissions on sub-directories
are now preserved in the same that they are for regular files.
Previously, the permissions were modified by HTCondor daemon's umask,
and directories transferred from a Windows machine to a UNIX machine had
no permissions enabled.
- Fixed bug in the HOLD_IF_CPUS_EXCEEDED configuration template metaknob.
- Fixed a bug in the LIMIT_JOB_RUNTIMES configuration template metaknob
so that it works in the face of a non-default MaxJobRuntime.
- Fixed a bug that made it so a restart of the condor_schedd was required
in order to change REMOVE_SIGNIFICANT_ATTRIBUTES to remove an attribute that
the condor_schedds had already marked as significant.
- When creating a Google Compute Engine instance, instruct the server
to delete the auto-created disk image when the instance is removed.
- Fixed a bug in our support for Google Compute Engine which would
cause authorization tokens to be renewed five minutes after the deadline
instead of five minutes before. Naturally, this led to ten minutes of
interrupted service for jobs (or workloads) which lasted longer than the
initial valid duration of the tokens.
- Fixed a bug that would cause the condor_schedd to crash when a
condor cron job was scheduled to start during the one hour gap in daylight
savings time when the clocks are moved backwards one hour.
- Fixed a bug that would cause a core dump when running the condor_history
command against a remote schedd. The results would be returned correctly, but a
core file would appear in the log directory after the command exited.
- In the Python bindings, fixed bugs in the LogReader and EventIterator
classes that could cause invocations of poll() to return prematurely with
- Fixed a bug in condor_dagman that caused a file named
'' (two single quotes) to be created if the
DAGMAN_SUPPRESS_JOB_LOGS configuration macro was set
- Fixed a bug that can cause the condor_starter to crash if the
connection to the condor_shadow is lost during file transfer.
- Fixed a bug that allowed a user to bypass the MAX_JOBS_PER_OWNER
limit by specifying an accounting group or nice_user in their submit file.
- When there are no GPUs on a machine, condor_gpu_discovery would write
to stderr in addition to its normal output, this made it hard to use the
-config option as intended. condor_gpu_discovery has been changed so
that it will never write to stderr when the -config option is specified,
instead it will write error messages as configuration comments to stdout.
- Removed obsolete ControlGroup option from HTCondor's systemd service
unit configuration file.
- Compiled benchmarking programs as a Position Independent Executable.
Position Independent Executables are a requirement for entry into Debian 9.
- Fixed a denial of service vulnerability when using the condor_credd
on the Windows platform.
- Fixed a bug where the -pool argument would be ignored by
condor_ssh_to_job under certain circumstances.
- HTCondor version 8.4.9 released on September 29, 2016.
- Increased the maximum number of unique attributes that can be
set by the condor_chirp command set_job_attr_delayed from
50 to 100, and added the configuration knob CHIRP_DELAYED_UPDATE_MAX_ATTRS.
See section 3.5.12 for more information.
- Fixed a bug where if the condor_startd crashed while running a
Docker universe job, the job would be left running and not removed when
the condor_startd restarted. The condor_startd now removes any
orphaned Docker universe jobs on restart.
- Fixed a bug that printed spurious locking-related warnings to
the StarterLog when running Docker universe jobs.
- The Job Router and HTCondor-C now properly send a RESCHEDULE
command to the condor_schedd after submitting a job.
- Fixed bugs in the Job Router that could cause a routed job to be
aborted if the UPDATE_JOB_INFO hook printed attributes to be set in
the job ad.
- The Job Router now uses the correct name for the configuration
parameter for the JOB_FINALIZE hook.
Previously, the Job Router used the name JOB_EXIT, counter to what was
- Updated systemd configuration to start HTCondor after NIS has started.
- Updated systemd configuration to start HTCondor after local LDAP name
service daemon has started.
- Updated systemd configuration to attempt restart of HTCondor daemons
after 1 minute.
- In the RPM packages, move the systemd tmpfiles configuration file to
the recommended directory (/usr/lib/tmpfiles.d).
- Fixed a bug introduced in 8.4.5 that caused configuration variables starting with
STARTD. or STARTER. to be ignored.
- Fixed a typo in the desired value of `rmem_max' in the Linux kernel
tuning script. Improved logging of Linux kernel tuning script by including
the name of the file (not) being changed.
- Fixed a bug that could cause the condor_master to crash after
restarting the condor_shared_port daemon.
- Fixed a bug that could cause the wrong dynamic slots to be preempted
for a match when ALLOW_PSLOT_PREEMPTION is set to True.
- Fixed a bug in cream_gahp that caused it to delegate
RFC-format X.509 proxies incorrectly to the CREAM service.
- Fixed a bug where the Windows version information was set to
a single value for multiple programs.
This resulted in crash boxes for most of the HTCondor tools being reported
as a crash of condor_gpu_discovery
- Fixed a bug whereby the condor_collector process would exit with
an error several times per hour if the configuration knob NO_DNS
is set to True.
- Fixed monitoring of memory and CPU usage of running jobs on Mac OS X.
This monitoring didn't work for a personal installation of HTCondor.
With Mac OS X 10.11 and above, this monitoring resulted in a flood of
errors messages to the system logs for a root-based installation.
- Fixed a bug when attempting to authenticate using multiple
methods wherein if a method failed, the remaining methods were not
- Fixed a bug where condor_userprio may fail to display
the correct priority factor value for a user associated
with a group.
- Fixed a bug that can cause the condor_procd to crash.
Fixed a bug that prevented other daemons from talking to the
condor_procd when it is restarted after a crash.
- If the condor_procd crashes, the condor_master now tries to
restart it several times. Previously only one restart attempt was done.
- Fixed a bug that resulted in the condor_starter crashing when
attempting to run a BOINC backfill job.
- Fixed a bug in the configuration language where an if defined
test would reject a valid variable name when it had both an underscore and a digit.
- Fixed a bug that caused the condor_ssh_to_job command to fail
when using the HTCondor RPM installation.
- HTCondor version 8.4.8 released on July 5, 2016.
- Fixed a memory leak in the condor_q client code that impacted
users of the Python API call htcondor.Schedd().query().
- Fixed a bug that caused file transfers to fail when using Bosco.
- Fixed a bug that could cause the condor_schedd to crash when using
- The condor_schedd now rejects job submissions when the job owner
doesn't have a user account on the machine.
Previously, the condor_schedd would accept such jobs and then fail
to run them.
- Fixed a bug introduced in the 8.4.7 release that resulted in the remote
condor_history command failing unless the -limit argument is used.
- Fixed a bug in condor_history that caused it to treat all unrecognized arguments
as user names
- The high-availability daemon now properly detects changes to the
HAD_LIST when reconfigured.
- The high-availability daemon now properly internalizes the
HAD_LIST when reconfigured.
- Fixed a bug that caused the condor_master to stop responding after it
restarted a child daemon when shared port is enabled on Windows. This bug could also
result in a hang on shutdown.
- Fixed a bug that could cause condor_status or condor_q to
crash when the -xml option is used.
- Fixed a bug introduced in the 8.4.7 release that resulted in a parse error from condor_submit
when JobAdInformationAttrs was set in the configuration variable SUBMIT_ATTRS.
- HTCondor version 8.4.7 released on June 6, 2016.
- Docker universe jobs now drop all Linux capabilities by default.
The new knob DOCKER_DROP_ALL_CAPABILITIES, which defaults to true
can be set to false to revert to the old behavior.
- Added configuration variable MAX_TIME_SKIP to control
how much system clock skip is allowed before the HTCondor daemons
restart. See Section 3.5.4 for more information.
- On Linux, HTCondor appropriately tunes kernel parameters
root_maxkeys and root_maxbytes to prevent condor_master
startup failures on older Linux kernels.
- The configuration variable SUBMIT_ATTRS now understands the +Attr
syntax that condor_submit uses to inject attributes directly into the job ClassAd.
- The condor_submit variable job_lease_duration can now be an
- All $function macro substitutions in in configuration files
will now correctly handle variables with subsystem and localname prefixes
as well as self references. In particular
VAR = $F(VAR) now substitutes
correctly rather than hanging forever.
- Fixed a bug in Docker universe where the job would
not run with the correct group id.
- Fixed a performance problem in the condor_schedd that could
cause it to become unresponsive for several minutes after the
set of significant attributes for negotiation changes.
- Fixed a bug where the python bindings ClassAd parser would fail to detect
whether old or new format ClassAds were present in a stream, even though the
ClassAd format was specified in advance.
- Fixed a bug where some floating point values would have an extra
.0 appended to the end when printed (e.g. 2E40.0).
These values could not be read properly by normal number parsing functions.
- When using GRIDMANAGER_SELECTION_EXPR, grid ads from
different condor_gridmanager instances will no longer overwrite
each other in the condor_collector.
- In addition to logging to the file KERNEL_TUNING_LOG,
the default LINUX_KERNEL_TUNING_SCRIPT now also logs to
syslog and /etc/systcl.d/99-htcondor.conf.
- Fixed a bug on condor_history that could result in truncation of
the job id field.
- On Windows, configuring HTCondor to restrict the range of outbound
port numbers may cause substantial delays when using the command-line
tools. Since we now know that it's not free to do so, LOWPORT
and HIGHPORT no longer restrict the port numbers of outbound
connections on Windows. If you still require this functionality, use
OUT_LOWPORT and OUT_HIGHPORT.
- Fixed a bug that would cause condor_submit to create extra, incorrectly named
output and error files when
$$ substitution is used as part of the filenames.
- Fixed a bug that would cause the condor_history_helper to be invoked using
the wrong name on Windows
- Fixed a bug that would sometimes cause configuration variables with a subsystem prefix
to be ignored.
- Fixed a bug that could cause HAD to fail if a machine has an IPv6
- Fixed a bugs in condor_history when fetching history from a remote condor_schedd. The
bugs caused complete failure when the remote condor_schedd was running Windows, and
would corrupt some string values when the remote condor_schedd was any other operating system.
- HTCondor version 8.4.6 released on April 21, 2016.
- condor_advertise -multiple now tolerates multiple blank lines in the
input file. It no longer quits parsing on the first first blank line that does not
follow a valid ClassAd.
- Fixed bug where when partitionable slots were
enabled in the condor_startd, a job would be unable
to start running on that machine in some cases.
- Fixed a bug that would cause the condor_startd
to crash when ALLOW_PSLOT_PREEMPTION was enabled.
- Fixed a bug introduced in version 8.3 that
removed the attribute REMOTE_GROUP_RESOURCES_IN_USE
from the job ad in the negotiator.
- Fixed a bug where HTCondor would regard as invalid text representations
of IPv6 addresses which were the longest possible. This bug typically
manifested as a failure to contact hosts which were advertising IPv6 addresses
of this sort.
- Fixed a memory leak in the condor_negotiator when
ALLOW_PSLOT_PREEMPTION was enabled.
- Fixed a bug where after a condor_schedd restart
the submitter attribute WEIGHTED_JOBS_RUNNING
would be incorrectly computed.
- Fixed a bug when using CLAIM_PARTITIONABLE_LEFTOVERS
Machines from a remote pool could be treated as if they were in the local
As a result, the RemotePool attribute would not be set in the ads
of jobs running on these machines, and the FlockedJobs and
RunningJobs attributes of submitter ads would have incorrect
- Fixed a bug that could cause a job's supplemental groups to be set
incorrectly when SOFT_UID_DOMAIN is set to True.
- Fixed a bug that caused supplemental groups to be set incorrectly
when executing file transfer plugins and various hooks.
- Fixed a bug that resulted in Windows 10 being reported as
WindowsUnknown in the OPSYSNAME attribute of the condor_startd
- Fixed a typo in the LIMIT_JOB_RUNTIMES policy
configuration template that prevented the policy from working as intended.
- HTCondor version 8.4.5 released on March 22, 2016.
- The default for DAGMAN_LOG_ON_NFS_IS_ERROR has
been changed from True to False. This is the result
of changes in the 8.3 series that mean that file locking is no
longer required on user logs.
- Fixed a bug where HTCondor would unconditionally retry non-successful
DNS lookups of the local system's hostname; this could cause delays of up
to sixty seconds when using command-line tools on systems whose hostname
was not in DNS. We no longer retry on errors at all, and only retry
failures which are temporary.
- Fixed a bug that would cause condor_schedds flocking to remote
pools to not send no jobs, or fewer jobs than possible to the
remote pool. This was a result of not correctly setting
the submitter attribute WeightedJobsRunning for
- Accounting group names that contain spaces are now rejected by
condor_submit and ignored by the condor_negotiator.
Previously, submitting a job with an accounting group name that contained
a space would cause the condor_negotiator to fail at startup.
- Fixed a bug whereby per-job history files (enabled by
the configuration setting PER_JOB_HISTORY_DIR) may briefly
appear to be empty or incomplete.
- Fixed a bug whereby ClassAds written into history files
may contain the same attribute multiple times.
- Fixed a bug that caused DAGMan to not work correctly with
some local universe node jobs. (This bug was introduced in version
- Fixed a bug that resulted in jobs managed by the condor_job_router
not reporting memory and disk usage of the job correctly.
- Reworked a bug fix from the 8.4.3 release that was designed to allow for
more than 100 dynamic slots to be a bit more generous in allocating Disk to
Now, those slots are less prone to fail to match subsequent jobs.
- Fixed a bug in the randomization of ports within the LOWPORT to HIGHPORT range
that would sometimes generate ports outside of this range on Windows.
- Fixed a bug in condor_off -peaceful that could result in never
sending the "off" command to machines when at least one of the machines could
not be contacted when sending the previous "peaceful" command.
- When cgroups are in use, limit the amount of file system cache in the
kernel to prevent the OOM killer from killing jobs that use a large amount of
file system cache.
- HTCondor version 8.4.4 released on February 4, 2016.
- Fixed a bug that caused the condor_collector to crash if
CONDOR_DEVELOPERS_COLLECTOR failed to resolve.
- Fixed a bug that caused Condor-C jobs to fail when
JobLeaseDuration was set to less than one hour (3600 seconds).
The remote job would be aborted due to the job lease not being renewed.
- Fixed a bug that could cause HTCondor to misreport an EC2 job as running
when it had in fact been purged from the service. Fixed bugs that could
cause a running EC2 job to be misreported as idle. HTCondor also no longer
sends EC2 services superfluous queries. (This may increase the latency
of HTCondor job status updates.)
- The grid manager now aborts if the GAHP hangs, which we detect by
the absence of a response after GRIDMANAGER_GAHP_RESPONSE_TIMEOUT
seconds. The EC2 GAHP now performs many fewer memory allocations in the
course of normal operations, which improves stability on some systems.
- Fixed a bug that caused the condor_master to fail if a shared port
daemon address file written by a version of HTCondor prior to 8.4.0
- Fixed a bug that caused updates to the job attribute
TimerRemove to not be respected while the job was being managed
by the condor_shadow, condor_gridmanager, or the Job Router.
- Fixed a bug where the job policy expression of a job could appear
in one of the Reason attributes of another job.
- Fixed a bug, that occurred on the Windows platform, that would cause
the condor_shadow to hang while trying to delete old shadow logs when the
value of MAX_NUM_SHADOW_LOG was larger than the default value of 1.
This bug would also sometimes result in the condor_schedd hanging.
- HTCondor version 8.4.3 released on December 16, 2015.
- Fixed a bug that caused the -append option to be handled too
late to apply to the first Queue statement in a condor_submit file.
- Fixed a bug that prevented running more than 100 slots on a single
condor_startd with partitionable slots.
- Fixed a bug which caused ec2_iam_profile_name
not to work for Spot instances.
- Fixed a bug where the cgroup VM limit would not be set for sizes over
- Fixed bugs that prevented the HTCondor daemons from working promptly at
startup when the condor_shared_port daemon was in use on Windows platforms.
- Added SELinux type enforcement rules to allow the condor_schedd
to use sendmail on Enterprise Linux 7 platforms.
- Fixed a bug where HTCondor service would not start if the
condor_master.pid file was empty on Linux platforms.
- HTCondor version 8.4.2 released on November 17, 2015.
- condor_history no longer reports an error when run on a system that does
not have a history file.
This change was made because the history file is not created until after the
first job runs.
So, users were always seeing an error message on a fresh installation of
- Fixed a bug introduced in 8.4.1 that could cause the condor_schedd
This affected remote submit, HTCondor-CE, and HTCondor-C.
- The TCP_FORWARDING_HOST is now honored by
HTCondor client programs.
- Fixed a problem where Standard Universe jobs could not restart
from a checkpoint in the Enterprise Linux 6 RPM distribution.
- Fixed bugs in the function of the DAGMan
especially for node jobs that create multiple procs.
- Fixed a problem where the RPMs would claim to publicly provide
Globus shared libraries that are in a private location.
- Added a default request_memory for condor_submit -interactive
of 512 megabytes. Formerly, the default was one, which is
insufficient in environments that strictly enforce memory
- Fixed a problem were the condor_classad RPM would claim to
provide a replacement for the classad RPM in EPEL.
- HTCondor now applies the configuration settings
when running grid universe jobs for EC2 or Google Compute Engine.
- Fixed a crash in the condor_schedd that happened when the
schedd was under load and being shutdown in the fast mode.
- Added a timeout to the condor_fetchlog command so that it
will not hang forever waiting for a unresponsive daemon.
- Fixed a problem that prevented HTCondor from building on some 64-bit Linux
platforms such as Arm64.
This was reported by Debian maintainers as their Bug 804386.
- Fixed a problem where the platform string was incorrect in the RPM
- The DAGMan workflow log file is not correctly written for local
universe DAG node jobs that have no log file specified in the submit file,
which causes DAGMan to wait forever, thinking the jobs have not completed.
Note that this problem can be worked around by specifying any
log file for the job, even log = /dev/null.
(This bug is a regression that was introduced some time since version
- DAG node retries do not work correctly with DAG node submit files
that create more than one proc in the resulting cluster (such nodes
cause DAGMan to hang if the retry is activated).
We believe that this bug has existed since DAGMan first supported
multi-proc node jobs.
- HTCondor version 8.4.1 released on October 27, 2015.
- Remote submit to an 8.4.1 condor_schedd is broken if file transfer is
used. This also means HTCondor-CE and HTCondor-C are broken. This bug will
be fixed in version 8.4.2.
- TCP_FORWARDING_HOST is disregarded by HTCondor clients
starting in version 8.3.6. This bug will be fixed in version 8.4.2 and 8.5.1.
- Added support to allow an admin to always volume mount
certain directories into docker universe containers running
on a host.
- Added four policy metaknobs to simplify configuring a policy
to either preempt or hold jobs that use more memory
or CPU cores than provisioned in the slot. See the POLICY
category of metaknobs in section 3.5.1 for
- Added configuration variables and documentation so that we uniformly prefer
<var>_ATTRS over <var>_EXPRS but support both. This includes
STARTD_ATTRS, STARTD_JOB_ATTRS and SUBMIT_ATTRS
which are often used by HTCondor sites which customize the configuration. These
configuration variables are now exclusively for use by HTCondor administrators;
The former default values for these variables have been moved into other configuration
which is reserved for use by HTCondor developers. This is done to prevent administrators
from accidentally removing the necessary defaults.
A warning about use of STARTD_EXPRS has been disabled unless
STARTD_ATTRS or SLOT_TYPE_<n>_STARTD_ATTRS is also used, since
the use all three of these at the same time is not supported.
- When condor_reconfig and condor_restart are run as root
they will check to see if the condor user has read access to all of the
configuration files before sending the command. This is done to prevent aborting the daemons
accidentally by sending reconfig after the admin creates a new config file and
forgets to give the condor user read access to that file.
- Added the -natural sort option to condor_status to sort the slots
in numerical order rather than alphabetical order.
- HTCondor version 8.4.0 released on September 14, 2015.
- Fixed a bug introduced in HTCondor version 8.3.7 that caused the
condor_shared_port daemon to leak file descriptors.
Also made HTCondor work better when some HTCondor daemons
are using shared port, but the condor_master is not.
- The condor_starter lowers the OOM (out of memory) score of jobs
so the OOM killer is more likely to chose an HTCondor job rather than
an HTCondor daemon or other user process.
- Job submission fails if X.509 certificates are advertised with EC2
grid universe jobs.
Therefore EC2 grid universe jobs no longer advertise their access keys.
Next: 11. Command Reference Manual
Up: 10. Version History and
Previous: 10.4 Development Release Series