Skip to content

Folders and subprojects

Intro

The Metadata service allows the caller to define Projects. Each project is a container for datasets (files and data in queryable storages), and can be used to control access to its contents (via setting access level and assigning users as members with a certain role). As of 10-02-2020 all projects comprise a flat structure - all projects are placed on the same level.

Subprojects

As of 11-02-2020 it is possible to nest projects within each other, effectively creating a tree-like structure. This structure can be used to emulate folders as you know them from a file system. In most cases the words project, folder and subproject can be used interchangeably, as they are all represented by the same entity and have effectively (almost) the same functionality. For the sake of clarity let's call - a project the top-level entity that may or may not contain other subprojects; - a subproject or folder the project that has been placed under a top-level project or another subproject; - a (sub)project either a top-level project or a subproject.

API

Please see Metadata Swagger for complete description of the API.

Listing projects

GET/api/metadata/project/list returns top-level projects. It supports cursor-based paging, meaning each response contains a cursor, that can be passed to the same endpoint to get the next page of results.

GET/api/metadata/project/offset-list returns top-level projects. It supports offset-based paging, meaning the caller can specify an offset within the result set to get a specific page of records.

In both cases the response is a flat list of objects. Contrary to previous version the returned projects do not contain information about the project members or user capabilites. This information can be retrieved by querying GET/api/metadata/project/{id}/member and GET/api/metadata/project/{id}/capabilities, respectively.

Both endpoints support server-side filtering and pagination. Filtering is possible by parameters:

  • namePrefix which will filter projects whose name starts with the given string;
  • role which will filter projects if the calling user has the given role in the project (inheritance is accounted for). The role can be specified multiple times, indicating the user can be a member in one of the listed roles. Empty parameter indicates no filtering by roles.
  • capability which will return only projects where the user has the given capability. The capability can have the following values:
    • CanReadContent - typically needed to access/read datasets within a project
    • CanListContent - needed for listing content of a project
    • CanCreateContent - needed to create a new content in a project (e.g. create a dataset)
    • CanUpdateContent - needed for updating an existing content (e.g. dataset)
    • CanDeleteContent - needed for deleting an existing content (e.g. dataset) Please note that the values are case sensitive. Please note that filtering by both role and capability is not allowed - use either role when looking for membership information, or capability when looking for effective permissions.

Sorting is done by specifying:

  • sortBy parameter, which is a string and can have values such as "Name" or "CreatedAt" (see swagger description to learn more);
  • sortOrder parameter, which can be either "Asc" or "Desc" to indicate sorting order.

To see the paging options please visit How to use pagination

Before the introduction of subprojects the endpoint GET/api/project/list returned all projects for the given customer. This endpoint now returns the list of all top-level projects for the given customer. The output structure contains the same information as before the introduction of subprojects.

Click to show sample request to get top-level projects 


curl -X GET "https://api.mike-cloud-test.com/api/project/list" -H "accept: text/plain" -H "api-version: 1" (bearer token and user headers omitted)
Response:
{
  "data": [
    {
      "name": "$IT_025_k1j7tkf8 - New Project",
      "description": "Description-024",
      "metadata": {},
      "settings": {},
      "accessLevel": "Shared",
      "members": [
        {
          "userId": "862281a7-2e66-4c76-af8d-29c82c723b4b",
          "role": "Owner"
        }
      ],
      "capabilities": {
        "canEdit": false,
        "canEditAccessLevel": false,
        "canDelete": false,
        "canGrantAccess": true,
        "canCreateContent": false,
        "canListContent": true,
        "canUpdateContent": false,
        "canDeleteContent": false,
        "canReadContent": true
      },
      "hasThumbnail": false,
      "inheritsMembers": false,
      "id": "5b66050a-156f-44d0-b8c6-b367d7792553",
      "createdAt": "2019-10-09T11:54:18.4251252",
      "createdBy": "862281a7-2e66-4c76-af8d-29c82c723b4b",
      "updatedAt": "2019-10-09T11:54:18.4251252",
      "updatedBy": "862281a7-2e66-4c76-af8d-29c82c723b4b"
    }
    ...,
    ]
}

Listing subprojects

GET/api/metadata/project/{id}/subproject/list returns direct subprojects of a project identified by id. It supports cursor-based paging, meaning each response contains a cursor, that can be passed to the same endpoint to get the next page of results.

GET/api/metadata/project/{id}/subproject/offset-list returns direct subprojects of a project identified by id. It supports offset-based paging, meaning the caller can specify an offset within the result set to get a specific page of records.

Same options are available as for listing top-level projects.

For listing subprojects contained within another project or subproject there is an endpoint GET/api/project/{projectId}/subprojects which will output a list of subprojects directly nested within the given project (no recursion is done).

Click to show sample request to get nested projects 


curl -X GET "https://api.mike-cloud-test.com/api/project/2a9e6d4e-063c-4f0d-8642-1a94166931fd/subprojects" -H "accept: text/plain" -H "api-version: 1" (bearer token and user headers omitted)
Response:
{
  "data": [
    {
      "name": "$IT_002_k6283p6q - New Project",
      "description": "Description-001",
      "metadata": {},
      "settings": {},
      "accessLevel": "Shared",
      "members": [
        {
          "userId": "862281a7-2e66-4c76-af8d-29c82c723b4b",
          "role": "Owner"
        }
      ],
      "capabilities": {
        "canEdit": false,
        "canEditAccessLevel": false,
        "canDelete": false,
        "canGrantAccess": true,
        "canCreateContent": false,
        "canListContent": true,
        "canUpdateContent": false,
        "canDeleteContent": false,
        "canReadContent": true
      },
      "hasThumbnail": false,
      "parentProjectId": "f536ea47-05cb-43fb-800a-078d7c67a079",
      "inheritsMembers": false,
      "id": "cb2eb616-d0ce-429e-868e-71dd860d6ef9",
      "createdAt": "2020-01-31T13:48:37.8910071",
      "createdBy": "862281a7-2e66-4c76-af8d-29c82c723b4b",
      "updatedAt": "2020-01-31T13:48:37.8910071",
      "updatedBy": "862281a7-2e66-4c76-af8d-29c82c723b4b"
    }
  ]
}

  • There is a new property in the output object - InheritsMembers. This boolean flag indicates whether the subproject inherits its members (and therefore access privileges) from its parent project or not. If not, then the members array lists all the members with their roles, just like with top-level projects.

  • Note that two users calling this endpoint with the same projectId may get different results. Some subprojects may not be accessible to the user due to different access level or membership. Those subprojects will be filtered out of the returned list.

  • Similarly, a user may have been given access to a subproject without being able to access the one or more of the parent (sub)projects. Any UI application should allow direct access to a project, by providing for example url routing with clear indication of the project id, so that the user can jump directly to the project. In the future other means (such as listing these "hidden, but accessible" subprojects) may be developed if the need arises.
  • Each subproject will have a property parentProjectId identifying the parent project. This can be useful when you receive a projectId directly (e.g. by user input) and you need to find its parent. If the need arises, we may expose endpoints for retrieving the root project, or the whole chain leading from the given project up to its root project (e.g. for the purposes of having a breadcrumb-like navigation). If this is needed, let the team Stargazer know.

Creating a subproject

Endpoint POST/api/metadata/project/{id}/subproject is meant to be used for creating subprojects within a given (sub)project. The input is exactly the same as for creating a project. You can specify a list of users with roles to be assigned to the subproject. The calling user will be added automatically as "owner" of the subproject.

Click to show sample request to create a subproject 


curl -X POST "https://api.mike-cloud-test.com/api/metadata/project/2edc81d5-dee4-40af-b7a7-9ba53416106b/subproject" -H "api-version: 3" -H "accept: text/plain" -H "api-version: 1" -H "Content-Type: application/json-patch+json" -d "{ \"name\": \"subproject1\", \"accessLevel\": \"Shared\"}" (bearer token and user headers omitted)
Response:
{
  "name": "subproject1",
  "metadata": {},
  "settings": {},
  "accessLevel": "Shared",
  "members": [
    {
      "userId": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
      "role": "Owner"
    }
  ],
  "capabilities": {
    "canEdit": true,
    "canEditAccessLevel": true,
    "canDelete": true,
    "canGrantAccess": true,
    "canCreateContent": true,
    "canListContent": true,
    "canUpdateContent": true,
    "canDeleteContent": true,
    "canReadContent": true
  },
  "hasThumbnail": false,
  "parentProjectId": "2edc81d5-dee4-40af-b7a7-9ba53416106b",
  "inheritsMembers": false,
  "id": "d3dbaf05-17e9-4a6d-9363-5c20e76c31e5",
  "createdAt": "2020-02-05T14:47:37.8898876",
  "createdBy": "aba1a3de-eadd-4937-980f-69dfdc9fad35",
  "updatedAt": "2020-02-05T14:47:37.8898876",
  "updatedBy": "aba1a3de-eadd-4937-980f-69dfdc9fad35"
}


Access rights for subprojects

  • As stated previously, by default a subproject is created with one member as Owner (the calling user). The members can be defined using endpoint PUT/api/metadata/project/{projectId}/member (the same as with regular projects). Alternatively, you can use PUT/api/metadata/project/{projectId}/members/inherit to set the subproject for member inheritance. When this endpoint is used, the subproject will inherit its members from its parent (sub)project (this works iteratively, so effectively the members are used from the first parent (sub)project that does not have this flag set), and the collection of members will be cleared for the given subproject. If you want to cancel the inheriting of members, simply set the members by calling PUT/api/metadata/project/{projectId}/members - this will set the project to use the given set of members and they will no longer be inherited from the parent.
  • There is also PATCH/api/metadata/project/ endpoint that allows you to set access level after the project is created. This works the same way for subprojects as it does for projects.

Deleting (sub)projects

DELETE/api/metadata/project/{projectId} can be used to delete a project as well as a subproject. In order to delete a project, the user must have the DELETE permission on that project (basically only an Owner can do that). In order to delete a subproject, the user must either have DELETE permission on the subproject (i.e. it must be the Owner of the deleted subproject), OR the user must have DELETECONTENT permission on the parent (sub)project (the Owner and Contributor roles have this). The presence of these permissions is indicated in the Capabilities object returned with each project object. If the user has either CanDelete flag on the deleted subproject, or the CanDeleteContent on the parent (sub)project, the delete operation will succeed. When a (sub)project is deleted, it and all of its nested subprojects will become unavailable for subsequent operations. Effectively the operation performs a soft-delete, and the target (sub)project is placed into a recycle bin. The (sub)project can be restored from the recycle bin during a grace period (this works the same way as with projects). See Recycle bin wiki for more information.

Moving subprojects

The endpoint POST/api/metadata/project/{id}/move can be used to move one subproject (identified by id) under a different subproject (identified by input:targetProjectId). Both subprojects must reside within the same root project. The user must have permissions to delete content from the current parent of the given subproject, and permissions to create content in the new parent. - Creating loops is forbidden (in other words the subproject being moved must not be on the path from the destination (sub)project to the root project). - Root projects can not be moved.

Listing datasets

The pre-existing endpoint GET/api/metadata/project/{projectId}/dataset/list can be used to get datasets placed directly within the given project.

Updating the subproject hierarchy in bulk

Hierarchy of subprojects and plain files can be updated in bulk by using POST/api/hierarchy/project/{id}/ (V3 of the metadata service). The endpoint accepts a list of paths relative to the projectId in the url. With each path you can specify whether it should be created, deleted or moved/renamed. Internally the service then maps these paths to projects, and either creates, deletes or renames the projects/datasets. The result of the bulk operation is a list of newly created paths, along with their respecitve project and dataset ids (where applicable).

This example request will create a a new subproject "level1" under , another subproject "level2" under "level1" and finally a new empty file dataset "file.txt" in the last subproject. If any one of the subfolders already exist, the action will still succeed and will reuse them, however, if the final node in the path (in this case file.txt) already exists, the action (and the whole bulk transaction) will fail.

curl -X POST "https://api.mike-cloud-test.com/api/hierarchy/project/<projectId>" \
  -H "accept: text/plain" -H "api-version: 3" \
  -H "Content-Type: application/json-patch+json" \
  <authentication headers omitted>
  --data-raw  "{ \"actions\": [ {\"type\":\"PathActionCreate\", \"path\":\"level1/level2/file.txt\", \"isFolder\":false} ]}"

Example response:
{
  "results":[
    {
      "projectId":"b06f1a7e-9a9c-475b-a3f3-4ef204dc2d5d",
      "path":"level1/",
      "sasToken":"<sas token for the project>"
    },
    {
      "projectId":"15d3ac1d-34dd-4d7d-b5a9-c3cb26cc235b",
      "path":"level1/level2/",
      "sasToken":"<sas token for the project>"
    },
    {
      "projectId":"15d3ac1d-34dd-4d7d-b5a9-c3cb26cc235b",
      "datasetId":"59f3822a-40e3-4c73-91b2-c7aec7ad575a",
      "path":"level1/level2/file.txt",
      "sasToken":"<sas token for the dataset>"
    }
  ]
}

Adapting to subprojects from flat project structure

  • In order to be able to show all datasets contained within a project, the client application must be able to traverse the hierarchical structure of nested subprojects. Each (sub)project may now contain not only datasets (see Listing datasets), but also subprojects (see Listing subprojects).
  • For performance reasons there are no endpoints that will return all datasets or subprojects for the given projects recursively (traversing all nested subprojects iteratively). Therefore any navigation must be done by sending GET requests for each level of hierarchy.
  • Since every subproject is also a project, all other pre-existing operations with projects will still work the same (provided that the user still has the correct permissions). For example the operation to import/convert a dataset (POST/api/transfer/upload-convert) still accepts a single projectId as its destination.
  • As the effective permissions for a user are now more difficult to compute, the GET/api/transfer/list-summaries endpoint has been changed to return transfer ONLY started by the calling user. This may be changed in the future to include other transfers. Right now only the DataAdmin UI application should be dependent on this endpoint. If this is an issue, please let the team Stargazer know.