Run an engine

How to interact with the Engine / Compute service¶

The entry point for using the Compute service is the IComputeClient interface. It can be resolved from DI container (after using AddPlatformClients extension) or constructed manually.

var collection = new ServiceCollection();
collection.AddPlatformClients(o =>
{
    o.ConfigureOpenApiKey("my-open-api-key");
    o.PlatformEnvironment = PlatformEnvironment.Dev;
});

using var provider = collection.BuildServiceProvider();

var computeClient = provider.GetRequiredService<IComputeClient>();

How to run an engine on data in the platform¶

The simplest case of an engine execution is when you have your files prepared inside a single folder, and you want to execute a single engine. The following example runs an engine on a lake.m21fm file. It is assumed that this file (along with any other necessary inputs) is stored under the folder identified by projectId.

The call to CreateExecution returns an instance of IExecutionBuilder, which can then be used to specify and finally invoke the execution itself.

var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);

var execution = await computeClient
    .CreateExecution(projectId)
    .WithModelRun(engine, "lake.m21fm")
    .ExecuteAsync();

Console.WriteLine($"Execution id {execution.ExecutionId}");

¶

NOTE: The CreateExecution can runs only when the ENGINE_EXECUTION feature is enabled and under the allowed project.

How to run an engine on data outside of the platform¶

You can run an engine on files that are to be downloaded from a remote location to the target machine. In such a case all files have to be explicitly specified, and then the model runs are associated with the files based on the url of the file.

var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);

var execution = await computeClient
            .CreateExecution(projectId)
            .WithRemoteInput(
                new RemoteInput("https://coreenginedevinputs.blob.core.windows.net/devdata/lake.m21fm"),
                new RemoteInput("https://coreenginedevinputs.blob.core.windows.net/devdata/lake.mesh"))
            .WithModelRun(engine, "https://coreenginedevinputs.blob.core.windows.net/devdata/lake.m21fm")
            .ExecuteAsync();

The RemoteInput class allows for overriding of the local file name in case it is not easily distinguishable from the url.

Note

As of 01-02-2021 it is not possible to combine inputs in such a way so that some inputs are provided as folder ids, and others as urls. Either have all your data in the platform and provide only folder ids (as platform inputs), or provide the list of all files via urls (remote inputs).

How to wait for an execution¶

The IExecutionBuilder provides means for waiting on an execution via ExecuteAndWaitAsync method. Awaiting this method will finish once the execution is done running.

var execution = await computeClient
    .CreateExecution(projectId)
    ...
    .ExecuteAndWaitAsync();

Another option is to invoke the execution, and then poll the status in your code using a polling loop.

var execution = await computeClient
    .CreateExecution(projectId)
    ...
    .ExecuteAsync();

while (true /*your own condition*/)
{
    var status = await computeClient.GetExecutionAsync(projectId, execution.ExecutionId);

    await Task.Delay(1000);
}

How to specify engine¶

For selected engines the SDK has strongly typed classes representing those engines. They are placed under DHI.Platform.SDK.Clients.Compute.Engines namespace and implement the IEngine interface. In some cases the versions known to the SDK are listed in an enumeration within that class. Also, f such engine has a parameter that the SDK knows, it can be set by methods found on the object itself.

var engine = new FemEngineHD(FemEngineHD.FemEngineHDVersion.v2021_0);   // version "2021.0"
engine.WithSubdomainsPerNode(5);                                        // set "SubdomainsPerNode" parameter
engine.WithThreadsPerSubdomain(2);                                      // set "ThreadsPerSubdomain" parameter

If the engine you are looking for is not present in the SDK, it can still be used, but has to be defined manually. The critical information of an engine is its Name and Version. You can find these by inspecting the deployed Compute service and listing the available engines. Once you have the name and version, you can construct a generic representation of that engine using GenericEngine class. Parameters can also be added, but it is up to the user to know the parameter names and correct value types.

var fePestEngine = new GenericEngine("FePEST", "8.0.00");
fePestEngine.WithParameter("WorkersPerNode", "2");

How to add multiple engine runs and multiple input folders¶

Multiple engine runs can be invoked within one execution. They will be invoked in the order in which WithModelRun method is called. Each run can have its model file located in a different folder - the contents of that folder will be downloaded to the working directory. If additional inputs located in other folders are needed, each folder can be added with the call to WithPlatformInput.

var execution = await computeClient
    .CreateExecution(projectId)
    .WithModelRun(engine1, "input1.setup")                      // file "input1.setup" is assumed to be found under projectId folder
    .WithModelRun(engine2, "input2.setup", dataFolder1Id)       // file "input2.setup" is assumed to be found under dataFolder1Id
    .WithModelRun(engine3, "data/input3.setup", dataFolder2Id)  // file "input3.setup" is assumed to be found under dataFolder2Id/data
    .WithPlatformInput(dataFolder3Id) // even when there are no model runs defined for this project, its contents will be downloaded into the working directory
    // no need to call WithPlatformInput on dataFolder1Id and dataFolder2Id - they are included automatically
    .ExecuteAsync();

How to continuously monitor an execution¶

The IExecutionBuilder can be setup to monitor a running execution and provide information about its progress. There is a method WithMessageHandler on IExecutionBuilder that takes an IMessageHandler instance as a parameter. Every time there is an update for the given execution, all provided IMessageHandler instances will be notified via their HandleMessageAsync method.

class MessageHandler : IMessageHandler
{
    public Task HandleMessageAsync(IEngineMessage message)
    {
        Console.WriteLine($"Execution status {message.Status}");
        return Task.CompletedTask;
    }
}

var messageHandler = new MessageHandler();

var execution = await computeClient
                .CreateExecution(projectId)
                .WithModelRun(engine, "lake.m21fm")
                .WithMessageHandler(messageHandler)
                .ExecuteAndWaitAsync();

The above example will continue executing after the execution finishes. That may take some time, so maybe you only want to setup the monitoring, and then continue with your code. For that the IExecutionBuilder has a method ExecuteAndMonitorAsync. This method will return an object that will track the given execution as long as it is not disposed. You can keep the object around and it will provide the progress messages during its lifetime.

// The monitor object is disposable: if it is disposed, it will stop tracking the execution. So do not dispose it until the execution finished, but do dispose it at some point, otherwise you may have a resource leak.
using var monitor = await computeClient
                .CreateExecution(projectId)
                .WithModelRun(engine, "lake.m21fm")
                .WithMessageHandler(messageHandler)
                .ExecuteAndMonitorAsync();

How to specify VM type and number of nodes¶

By default the execution will run on the smallest (least powerful) VM type. For production workloads this should be changed to match your model needs and requirements. The CreateExecution method has an optional parameter of type ComputeOptions which can be used to specify the VM type and number of nodes used for the execution. The SDK provides a static enumeration of VM types known at the time of the SDK publishing, but the list of available VM types is in general dynamic and can be enhanced in the future. If you want to select a pool that is not visible in the enumeration, you can still specify it by providing the PoolType identifier retrieved at runtime from the service.

Use static enumUse list of VMs retrieved at runtime

var execution = await computeClient
                .CreateExecution(projectId, new ComputeOptions(ComputeOptions.PoolTypes.VM_S_5, 2)) // pick the VM-S-5 and 2 nodes
                ...
                .ExecuteAsync();

var items = await computeClient.GetConfigurationListAsync(projectId);

var execution = await computeClient
                .CreateExecution(projectId, new ComputeOptions(items.First().PoolType, 2)) // pick the first VM type arbitrarily and 2 nodes
                ...
                .ExecuteAsync();