Scheduler Developer Guide

This page describes some of the current implementation details of the Scheduler CSC. It may be that some details described here are not yet in line with the Scheduler requirements and some requirements are still not fully implemented. These will be corrected in future updates to the software and documentation as we work towards fulfilling all the requirements.

The primary classes in the Scheduler CSC are:

• SchedulerCSC; Commandable SAL Component that manages the interaction with the observatory control system and computes the observing queue.

• Driver; A base class that implements an API for integrating scheduling algoriths in the context of the Rubin Observatoy.

These two main classes are intrinsically interconnected, with the SchedulerCSC using the Driver to generate a list of observations and interacting with the other parts of the observatory control software to execute them. While doing so, the SchedulerCSC makes some assumptions about the Driver implementation and overall behavior. When implementing a new scheduling algorithm to use with the SchedulerCSC, one must take into account these assumptions and make sure they are not violated or, at least, that they understand their impact on the output of their underlying implementation.

In the following sections we will explore each one of these modules individually.

Scheduler CSC

The SchedulerCSC follows the standard definition of CSCs in the context of the Rubin Observatoy Control System. In short the CSC works as follows;

• While in STANDBY state, the CSC will not perform any work, and simply establishes itself as alive and ready.

  In this state the CSC will not hold an instance of the Driver or any other object it uses for operation. One should assume that any state gathered by the CSC when it is in other states is wiped out when the CSC is in STANDBY.

  This is particularly important when developing new Driver classes, since one should not expect state to be persisted when the CSC is transitioned to this state.

  Any state that the Driver or the model classes construct is done when transitioning from STANDBY to DISABLED.

• When transitioning from STANDBY to DISABLED, the CSC will re-construct instances to all the objects it needs (Driver and models alike).

  In the default mode of operation this means the CSC will start with bare initilized objects.

  The CSC provides a mechanism to reconstruct state for the Driver through the startup_type and startup_database configuration parameters.

  See more details in Startup Modes.

• Once in ENABLED state the CSC will monitor and synchronize the observatory state with its own observatory state object.

  The CSC will not perform any additional action upon entering the ENABLED state, therefore it is safe to keep the CSC in ENABLED state at any time.

• When ready to handle control of the observatory to the SchedulerCSC, the operator must send a resume command to the CSC.

  This will cause the CSC to activate the target production loop, which will then interact with the Driver to produce a sequence of observations and later send the associated observations to the ScriptQueue.

• Operators can continue to interact with the ScriptQueue while the SchedulerCSC is operating.

  In all modes of operation, the SchedulerCSC monitors the content of the ScriptQueue and can interoperate with scripts scheduled by other parties.

  In most cases, this simply consists of waiting for the Script to finish executing before adding sending more targets for observations.

  Nervertheless, the SchedulerCSC tries to do more than simply wait. If the Script has information about telescope position and duration, the Scheduler CSC will take that information into account when determining the observatory state for future observations. For more information see the Operation Modes.

• To stop the SchedulerCSC, operators can send the stop command.

  The stop command will interrupt the target production loop. By default, the SchedulerCSC will allow the currently scheduled observations (those it already sent to the ScriptQueue) to continue and will continue to register them in the Driver.

  Alternatively, users can tell the SchedulerCSC to interrupt any scheduled observation.

Startup Modes

The SchedulerCSC supports three different types of startup modes, that controls how the Driver is initialized.

Cold Start

This is the simplest, and default, startup mode. In this mode the scheduler will only initialize the driver with the input configuration parameters and will make no additional attempt to reconstruct state.

Warm Start

In this mode, the scheduler begins by doing a cold initialization of the driver with its initial settings. After initialization, the SchedulerCSC calls the Driver, passing the value of startup_database. This method is responsible for reading the input file, in whatever format the specific driver understand and returning a list of Driver (or appropriate subclass thereof). The list is then fed back into the Driver by sequentially calling register_observation.

Here the SchedulerCSC assumes that calling register_observation is sufficient to reconstruct the state of a previous observation, without the need to call select_next_target.

Some implementations may not support this kind of operation. In these cases, users must make sure calling parse_observation_database raises an exception to interrupt the SchedulerCSC from attempting to complete the task.

Hot Start

In this mode the SchedulerCSC will also begin by doing a cold initialization of the driver with its initial settings. Nervertheless, after the initialization process, the SchedulerCSC calls reset_from_state, passing the value of startup_database. This method should be able to read a snapshot of the state of the scheduler algorithm that reconstruct a previously stored state.

The snapshot must have a similar format to that generated by calling Driver.save_state.

This mechanism of saving a snapshot and reloading it afterwards is central to the Advance scheduler mode of operation.

Operation Modes

Simple

The simple mode of operation causes the scheduler to deal with one target at a time. When operating in this mode the Scheduler will;

1. gather telemetry,

2. update the driver with the most recent telemetry,

3. request one target,

4. send the target to the ScriptQueue for execution,

5. wait for the script execution to complete,

6. if the observation executes successfully, register the observation in the driver.

This mode is mostly to be used for testing the system is a sequential way.

Advance

The advanced mode is the one intended for actual operation of the observatory. In its current implementation it exercises most of the functional requirements of the SchedulerCSC, but still lacks some important feature, most notably, the capability to compute a predicted queue in a configurable window into the future.

Nervertheless, the current implementation does have some of the traits that we hope to use as a basis to develop these additional features.

What this mode currently does is to compute a (configurable) number of targets ahead of time and commit to execute them. Furthermore, even though it is commited to execute those observations, it makes no assumption about the outcome of the observations. This means, it will still handle the cases when the observation is scheduled but fails to be taken.

The Advance mode of operation works like this:

1. Gather telemetry,

2. Synchronize the model observatory state with the current observatory state.

3. Save a snapshot of the driver state, by calling Driver.save_state. The data is stored in the LFOA and an event is published with information about the file.

3. Simulate observing any scheduled observation.

   This process involves:

   • Simulate the observation in the observatory model.

   • Register the observation in the driver with Driver.register_observation.

4. Generate a list of targets with the following procedure:

   1. Request one target by calling select_next_target.

   2. Register the target as observed.

      This process involves:

      • Simulate the observation in the observatory model.

      • Register the observation in the driver with Driver.register_observation.

   3. Repeat until n_targets are generated or if no target is returned in step 1.

5. Reset the state of the Driver by calling Driver.reset_from_state with the file generated in step 3.

6. Schedule 2 targets in the ScriptQueue for observing.

   While one target executes the second target is waiting to be executed. When the running target finishes it will send a new one to make sure there is always a target waiting to execute in the ScriptQueue.

   When a target is observed successfully (Script finishes executing) register the observation with Driver.register_observation.

   If the observation fails, the target is not registered. Note that this process is very similar to doing a short scale Warm Start start. For the scheduling algorithm in the Driver, that observation was never requested. Since we reset the state of the Driver to the one before the observations where requested, is is like select_next_target was never called.

   Therefore, it is paramount for this mode of operation to work that select_next_target does not change the state of the scheduler.

7. When all targets generated in step 4 are scheduled for observation in the ScriptQueue, start again from step 1.

Dry

The SchedulerCSC has a third mode of operation that is mainly used for testing.

When configured in this mode the SchedulerCSC will not initialize the driver and the target generation loop. It is still possible to bring the component to the ENABLED state with this mode and command it to resume and stop, but no target will be generated in any condition.

This mode is only provided for unit testing purposes and should not be used in general.

Driver

The Driver provides an interface for integrating different scheduling algoriths with the Scheduler CSC. When implementing new scheduling algorithms, developers should first subclass Driver. The default implementation provides most of the busines logic, leaving to the developer to implement a couple different methods to specify how the application is initialized, configured, consume telemetry and produce targets for observation.

The standard Driver implementation is paired with a standard set or parameters and a standard target class; DriverParameters and DriverTarget, respectively. These only provide some basic parameters used by the Driver in it standard operations. Developers can further expand the parameters and the target class on their implementation by subclassing them.

The following is the minimum set of methods that needs to be implemented by the driver:

• Driver.configure_scheduler

• Driver.update_conditions

• Driver.select_next_target

• Driver.register_observation

With this minimum set of methods implemented it is possible to run some simple integration tests. Nervertheless, for real operations of the observatory, the following methods should also be implemented.

• Driver.save_state

• Driver.parse_observation_database

• Driver.get_state_as_file_object

• Driver.reset_from_state

Finally, the following functional methods are needed for full-featured operations.

• Driver.load

• Driver.cold_start

The Feature-Based Scheduler Driver

TBD

The Sequential Driver

TBD

Code API

lsst.ts.scheduler.scheduler_csc Module

Functions

run_scheduler()

Run the Scheduler CSC.

Classes

SchedulerCSC(index[, config_dir, ...])

This class is a reactive component which is SAL aware and delegates work.

Class Inheritance Diagram

digraph inheritance0361a7296f { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "BaseCsc" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Base class for a Commandable SAL Component (CSC)"]; "Controller" -> "BaseCsc" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ConfigurableCsc" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Base class for a configurable Commandable SAL Component (CSC)"]; "BaseCsc" -> "ConfigurableCsc" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ABC" -> "ConfigurableCsc" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Controller" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that receives commands for a SAL component"]; "SchedulerCSC" [URL="../../py-api/lsst.ts.scheduler.scheduler_csc.SchedulerCSC.html#lsst.ts.scheduler.scheduler_csc.SchedulerCSC",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="This class is a reactive component which is SAL aware and delegates"]; "ConfigurableCsc" -> "SchedulerCSC" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

lsst.ts.scheduler.telemetry_stream_handler Module

Classes

TelemetryStreamHandler(log, efd_name)

Handle telemetry stream.

Class Inheritance Diagram

digraph inheritance6e98febef1 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "TelemetryStreamHandler" [URL="../../py-api/lsst.ts.scheduler.telemetry_stream_handler.TelemetryStreamHandler.html#lsst.ts.scheduler.telemetry_stream_handler.TelemetryStreamHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Handle telemetry stream."]; }

lsst.ts.scheduler.driver Package

Classes

Driver(models, raw_telemetry[, parameters, log])	The Scheduler Driver is the module that normalizes the interface between any scheduling algorithm to the LSST Scheduler CSC.
DriverParameters([night_boundary, ...])	Actual global driver configuration parameters.
FeatureScheduler(models, raw_telemetry[, ...])	Feature scheduler driver.
NoNsideError	Exception raised when nside is not defined in the feature scheduler configuration.
NoSchedulerError	Exception raised when scheduler is not defined in the feature scheduler configuration.
SequentialParameters([night_boundary, ...])	Sequential driver parameters.
SequentialScheduler(models, raw_telemetry[, ...])	A simple scheduler driver that implements a sequential scheduler algorithm.
SurveyTopology()

Class Inheritance Diagram

digraph inheritanced3337d3cde { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Driver" [URL="../../py-api/lsst.ts.scheduler.driver.Driver.html#lsst.ts.scheduler.driver.Driver",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The Scheduler Driver is the module that normalizes the interface between"]; "DriverParameters" [URL="../../py-api/lsst.ts.scheduler.driver.DriverParameters.html#lsst.ts.scheduler.driver.DriverParameters",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Actual global driver configuration parameters."]; "FeatureScheduler" [URL="../../py-api/lsst.ts.scheduler.driver.FeatureScheduler.html#lsst.ts.scheduler.driver.FeatureScheduler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Feature scheduler driver."]; "Driver" -> "FeatureScheduler" [arrowsize=0.5,style="setlinewidth(0.5)"]; "NoNsideError" [URL="../../py-api/lsst.ts.scheduler.driver.NoNsideError.html#lsst.ts.scheduler.driver.NoNsideError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception raised when nside is not defined in the feature scheduler"]; "NoSchedulerError" [URL="../../py-api/lsst.ts.scheduler.driver.NoSchedulerError.html#lsst.ts.scheduler.driver.NoSchedulerError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception raised when scheduler is not defined in the feature scheduler"]; "SequentialParameters" [URL="../../py-api/lsst.ts.scheduler.driver.SequentialParameters.html#lsst.ts.scheduler.driver.SequentialParameters",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Sequential driver parameters."]; "DriverParameters" -> "SequentialParameters" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SequentialScheduler" [URL="../../py-api/lsst.ts.scheduler.driver.SequentialScheduler.html#lsst.ts.scheduler.driver.SequentialScheduler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A simple scheduler driver that implements a sequential scheduler"]; "Driver" -> "SequentialScheduler" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SurveyTopology" [URL="../../py-api/lsst.ts.scheduler.driver.SurveyTopology.html#lsst.ts.scheduler.driver.SurveyTopology",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

lsst.ts.scheduler.utils Package

Functions

get_efd_client(efd_name[, mock])	Utility function to get an EFD client.
get_mock_efd_client(efd_name)	Create a mock EFD client.
is_uri(uri)	Check if input is a valid uri.
set_detailed_state(detailed_state)	A class decorator for coroutine to facilitate setting/resetting detailed state.
support_command(command_name)	Check if the CSC supports a particular command.

Classes

FailedToQueueTargetsError
SchedulerCscParameters([driver_type, ...])	Configuration of the LSST Scheduler's Model.
SchedulerModes(value)	An enumeration.
SchemaConverter()	Record how to convert an observation array to the standard opsim schema.
UnableToFindTargetError
UpdateTelemetryError

Class Inheritance Diagram

digraph inheritanced23a694432 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Enum" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Generic enumeration."]; "FailedToQueueTargetsError" [URL="../../py-api/lsst.ts.scheduler.utils.FailedToQueueTargetsError.html#lsst.ts.scheduler.utils.FailedToQueueTargetsError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "IntEnum" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Enum where members are also (and must be) ints"]; "Enum" -> "IntEnum" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SchedulerCscParameters" [URL="../../py-api/lsst.ts.scheduler.utils.SchedulerCscParameters.html#lsst.ts.scheduler.utils.SchedulerCscParameters",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration of the LSST Scheduler's Model."]; "SchedulerModes" [URL="../../py-api/lsst.ts.scheduler.utils.SchedulerModes.html#lsst.ts.scheduler.utils.SchedulerModes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An enumeration."]; "IntEnum" -> "SchedulerModes" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SchemaConverter" [URL="../../py-api/lsst.ts.scheduler.utils.SchemaConverter.html#lsst.ts.scheduler.utils.SchemaConverter",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Record how to convert an observation array to the standard"]; "UnableToFindTargetError" [URL="../../py-api/lsst.ts.scheduler.utils.UnableToFindTargetError.html#lsst.ts.scheduler.utils.UnableToFindTargetError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "UpdateTelemetryError" [URL="../../py-api/lsst.ts.scheduler.utils.UpdateTelemetryError.html#lsst.ts.scheduler.utils.UpdateTelemetryError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

Dependencies

The Scheduler CSC is a Python salobj based CSC, as such it depends on the Observatoy Control System core packages;

• ts_xml

• ts_sal

• ts_idl

• ts_salobj

• ts_ddsconfig

For development and testing we recommend using the ts-conda-build metapackage.

Furthermore, the Scheduler has additional test and run dependencies for the models, the supported scheduling algoriths and other observatory control packages (e.g. scriptqueue). The list includes the following packages, which are available through conda;

• ts-scriptqueue (development and testing only)

• ts-observatory-model

• ts-astrosky-model

• ts-dateloc

• rubin-sim

Build and Test

There are two supported ways to build and test the Scheduler CSC, using eups or conda/pip. Selecting which method is more suitable for you depends on what kind of changes you are planning to make.

If the update you intend to make doesn’t involves changes to CSC interface (which also means updates to ts_xml, the best option is to use conda/pip. If you need to update the CSC interface (event, command or telemetry), the preferred method is to use eups.

Let us review both options.

Developing with conda

I assume here you have a basic conda environment ready with a usable installation of OpenSpliceDDS. The latter is the library that handles the Observatory Control Middleware and is used by the CSC to communicate with other components in the system.

Start by creating a conda environment with python 3.8 and activate it:

conda create -y --name scheduler-dev python=3.8
conda activate scheduler-dev

Then, install the test dependencies:

conda install -y -c lsstts ts-conda-build=0.3 ts-idl ts-utils ts-salobj ts-scriptqueue ts-observatory-model ts-astrosky-model ts-dateloc rubin-sim

From inside the package now, install the package locally with pip:

pip install -e . --ignore-installed --no-deps

Now you should be able to run the package unit tests with:

pytest

Developing with eups

For developing with eups we strongly recommend using the development environment docker image. The image ships with all the necessary libraries to get starting with minimum setup.

The process to developing using the docker development environment and eups is as follows:

1. Pull the latest docker image:

   docker pull lsstts/develop-env:develop
   

2. Run the docker image, exporting your local development directory:

   docker run -it -v <path-to-local-development-directory>:/home/saluser/develop lsstts/develop-env:develop
   

   After running the command above you will be inside the docker container. From now on, I assume you are in the same prompt that you are left in after running the command above.

3. Install the scheduler dependencies:

   conda install -y -c lsstts ts-observatory-model ts-astrosky-model ts-dateloc rubin-sim
   

   Note that we excluded some dependencies from the previous conda install command used in the Developing with conda. This is because those packages are already available in the development environment.

4. Declare the scheduler package with eups:

   cd ~/develop/ts_scheduler
   eups declare -r . -t $USER
   setup ts_scheduler -t $USER
   

Usage

[Description on how to use the software, scripts, any useful programs etc. Basic operations such as startup/shutdown should be explained, ideally with example code/steps]

Building the Documentation

Before building the documentation page, follow one of the development installation procedures in Build and Test.

With the packages installed run:

pip install -r doc/requirements.txt

To install the libraries required to build the documentation, and then:

cd docs/
package-docs build

Contributing

Code and documentation contributions utilize pull-requests on github. Feature requests can be made by filing a Jira ticket with the Scheduler label. In all cases, reaching out to the contacts for this CSC is recommended.