Run an engine
How to interact with the Engine / Compute service¶
The entry point for using the Compute service is the IComputeClient
interface. It can be resolved from DI container (after using AddPlatformClients
extension) or constructed manually.
var collection = new ServiceCollection();
collection.AddPlatformClients(o =>
{
o.ConfigureOpenApiKey("my-open-api-key");
o.PlatformEnvironment = PlatformEnvironment.Dev;
});
using var provider = collection.BuildServiceProvider();
var computeClient = provider.GetRequiredService<IComputeClient>();
How to run an engine on data in the platform¶
The simplest case of an engine execution is when you have your files prepared inside a single folder, and you want to execute a single engine. The following example runs an engine on a lake.m21fm
file. It is assumed that this file (along with any other necessary inputs) is stored under the folder identified by projectId
.
The call to CreateExecution
returns an instance of IExecutionBuilder
, which can then be used to specify and finally invoke the execution itself.
var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);
var execution = await computeClient
.CreateExecution(projectId)
.WithModelRun(engine, "lake.m21fm")
.ExecuteAsync();
Console.WriteLine($"Execution id {execution.ExecutionId}");
¶
var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);
var execution = await computeClient
.CreateExecution(projectId)
.WithModelRun(engine, "lake.m21fm")
.ExecuteAsync();
Console.WriteLine($"Execution id {execution.ExecutionId}");
NOTE: The CreateExecution can runs only when the ENGINE_EXECUTION feature is enabled and under the allowed project.
How to run an engine on data outside of the platform¶
You can run an engine on files that are to be downloaded from a remote location to the target machine. In such a case all files have to be explicitly specified, and then the model runs are associated with the files based on the url of the file.
var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);
var execution = await computeClient
.CreateExecution(projectId)
.WithRemoteInput(
new RemoteInput("https://coreenginedevinputs.blob.core.windows.net/devdata/lake.m21fm"),
new RemoteInput("https://coreenginedevinputs.blob.core.windows.net/devdata/lake.mesh"))
.WithModelRun(engine, "https://coreenginedevinputs.blob.core.windows.net/devdata/lake.m21fm")
.ExecuteAsync();
The RemoteInput
class allows for overriding of the local file name in case it is not easily distinguishable from the url.
Note
As of 01-02-2021 it is not possible to combine inputs in such a way so that some inputs are provided as folder ids, and others as urls. Either have all your data in the platform and provide only folder ids (as platform inputs), or provide the list of all files via urls (remote inputs).
How to wait for an execution¶
The IExecutionBuilder
provides means for waiting on an execution via ExecuteAndWaitAsync
method. Awaiting this method will finish once the execution is done running.
var execution = await computeClient
.CreateExecution(projectId)
...
.ExecuteAndWaitAsync();
Another option is to invoke the execution, and then poll the status in your code using a polling loop.
var execution = await computeClient
.CreateExecution(projectId)
...
.ExecuteAsync();
while (true /*your own condition*/)
{
var status = await computeClient.GetExecutionAsync(projectId, execution.ExecutionId);
await Task.Delay(1000);
}
How to specify engine¶
For selected engines the SDK has strongly typed classes representing those engines. They are placed under DHI.Platform.SDK.Clients.Compute.Engines
namespace and implement the IEngine
interface. In some cases the versions known to the SDK are listed in an enumeration within that class. Also, f such engine has a parameter that the SDK knows, it can be set by methods found on the object itself.
var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0); // version "2021.0"
engine.WithSubdomainsPerNode(5); // set "SubdomainsPerNode" parameter
engine.WithThreadsPerSubdomain(2); // set "ThreadsPerSubdomain" parameter
If the engine you are looking for is not present in the SDK, it can still be used, but has to be defined manually. The critical information of an engine is its Name and Version. You can find these by inspecting the deployed Compute service and listing the available engines.
Once you have the name and version, you can construct a generic representation of that engine using GenericEngine
class. Parameters can also be added, but it is up to the user to know the parameter names and correct value types.
var fePestEngine = new GenericEngine("FePEST", "8.0.00");
fePestEngine.WithParameter("WorkersPerNode", "2");
Engine Options Configuration¶
WithReportLogUpdatesLines
¶
Defines the configuration for log update lines from the engine.
- Parameters:
reportLogUpdatesLines
(int
): The number of lines from the engine log to send with each update (every second). The default value is300
. Acceptable values range from-1
to1000
. If a value outside this range is provided, anArgumentOutOfRangeException
is thrown.-
logFiles
(IEnumerable<string>
): Specifies the log files from which updates are sent. This is a list of relative paths to the directory containing the setup file. If no log files are listed, only the[setup_file].log
is monitored for updates. -
Example Usage:
engine.WithReportLogUpdatesLines(500, new List<string> { "custom_log1.log", "custom_log2.log" });
WithTerminateOnNoProgressForSec
¶
Defines a custom timeout for terminating the engine if no progress is detected.
- Parameters:
-
terminateOnNoProgressForSec
(int
): The number of seconds the engine can run without making progress before it is automatically terminated.The default value is 1200. The value must be greater than0
. If a non-positive value is provided, anArgumentOutOfRangeException
is thrown. -
Example Usage:
engine.WithTerminateOnNoProgressForSec(1200); // Sets a 20-minute timeout for detecting no progress
How to add multiple engine runs and multiple input folders¶
Multiple engine runs can be invoked within one execution. They will be invoked in the order in which WithModelRun
method is called. Each run can have its model file located in a different folder - the contents of that folder will be downloaded to the working directory. If additional inputs located in other folders are needed, each folder can be added with the call to WithPlatformInput
.
var execution = await computeClient
.CreateExecution(projectId)
.WithModelRun(engine1, "input1.setup") // file "input1.setup" is assumed to be found under projectId folder
.WithModelRun(engine2, "input2.setup", dataFolder1Id) // file "input2.setup" is assumed to be found under dataFolder1Id
.WithModelRun(engine3, "data/input3.setup", dataFolder2Id) // file "input3.setup" is assumed to be found under dataFolder2Id/data
.WithPlatformInput(dataFolder3Id) // even when there are no model runs defined for this project, its contents will be downloaded into the working directory
// no need to call WithPlatformInput on dataFolder1Id and dataFolder2Id - they are included automatically
.ExecuteAsync();
How to continuously monitor an execution¶
The IExecutionBuilder
can be setup to monitor a running execution and provide information about its progress. There is a method WithMessageHandler
on IExecutionBuilder
that takes an IMessageHandler
instance as a parameter. Every time there is an update for the given execution, all provided IMessageHandler
instances will be notified via their HandleMessageAsync
method.
class MessageHandler : IMessageHandler
{
public Task HandleMessageAsync(IEngineMessage message)
{
Console.WriteLine($"Execution status {message.Status}");
return Task.CompletedTask;
}
}
var messageHandler = new MessageHandler();
var execution = await computeClient
.CreateExecution(projectId)
.WithModelRun(engine, "lake.m21fm")
.WithMessageHandler(messageHandler)
.ExecuteAndWaitAsync();
The above example will continue executing after the execution finishes. That may take some time, so maybe you only want to setup the monitoring, and then continue with your code. For that the IExecutionBuilder
has a method ExecuteAndMonitorAsync
. This method will return an object that will track the given execution as long as it is not disposed. You can keep the object around and it will provide the progress messages during its lifetime.
// The monitor object is disposable: if it is disposed, it will stop tracking the execution. So do not dispose it until the execution finished, but do dispose it at some point, otherwise you may have a resource leak.
using var monitor = await computeClient
.CreateExecution(projectId)
.WithModelRun(engine, "lake.m21fm")
.WithMessageHandler(messageHandler)
.ExecuteAndMonitorAsync();
How to specify VM type and number of nodes¶
By default the execution will run on the smallest (least powerful) VM type. For production workloads this should be changed to match your model needs and requirements.
The CreateExecution
method has an optional parameter of type ComputeOptions
which can be used to specify the VM type and number of nodes used for the execution.
The SDK provides a static enumeration of VM types known at the time of the SDK publishing, but the list of available VM types is in general dynamic and can be enhanced in the future. If you want to select a pool that is not visible in the enumeration, you can still specify it by providing the PoolType identifier retrieved at runtime from the service.
var execution = await computeClient
.CreateExecution(projectId, new ComputeOptions(ComputeOptions.PoolTypes.VM_S_5, 2)) // pick the VM-S-5 and 2 nodes
...
.ExecuteAsync();
var items = await computeClient.GetConfigurationListAsync(projectId);
var execution = await computeClient
.CreateExecution(projectId, new ComputeOptions(items.First().PoolType, 2)) // pick the first VM type arbitrarily and 2 nodes
...
.ExecuteAsync();