How do I set up a managed bucket?

Use Dataset Manager to create a managed S3 bucket to push data to your datasets.

Overview

Once you've created your dataset you can push data for it to an AWS S3 bucket created inside Narrative's account. Hosting the bucket inside our accounts allows Narrative to read from the bucket to pull your data into our platform.

Using Dataset Manager you can to create your own managed S3 bucket by following these steps:

1. Set up an AWS Account if you don't already have one.
2. Create the bucket.
3. Set up bucket access.

Set up an AWS Account

If you already have an AWS account you can skip this step, otherwise follow the official AWS account creation guide.

The account will only be used to securely access your managed bucket.

Create the Bucket

Inside Dataset Manager, click on Sources in the navigation menu on the lefthand side of the page then New Sources.

1. Source Type

   The "Managed AWS S3 Bucket" option will be preselected for you.

2. Configuration

   Configure your managed bucket.

   • AWS Account ID: the 12 digit AWS account ID that will have access to your bucket. If you are having trouble finding this number consult the AWS guide on account identifiers.
   • Resource ID: a short identifier that will be a part of your bucket's name. A typical choice would be your company's name, lowercased and spaces replaced with -.
   • External ID (Optional): you can increase the security of your bucket by requiring external identifiers, though this is optional. Learn more.

3. Create

   Review your bucket configuration then click "Create".

Once your bucket has been created you'll be able to access the following information on the Sources page:

• Bucket Name: the name of the managed bucket created for you.
• Role ARN: the AWS IAM role which has full access your managed bucket. The next section will walk through how to use this role.
• AWS Account ID: this is the AWS account ID that you provided which has access to the bucket.

Set up Bucket Access

Once your managed bucket has been created you'll need to configure your AWS account to access it.

The example below gives an IAM user access to a managed bucket but the same can be done for any group, role, etc. inside your AWS account: Narrative does not restrict who can use the role inside your account, giving you complete control over access management. The steps to follow are:

1. Grant a user access to the role.
2. Configure your tools to use the role. Examples for the following systems will be given:
   1. AWS CLI
   2. AWS Console

Prerequisite: Grant User Access

No matter which tool you plan to use to access your bucket the first step is to grant a user inside your account permission to use the role. Narrative does not restrict who can use the role inside your account so the same steps could be followed for any group, role, etc. inside your account.

This step may require the assistance of an administrator of your AWS account as it requires AWS IAM permissions.

• Open the AWS IAM console at https://console.aws.amazon.com/iam
• Select the user that you want to be able to access your managed bucket.
• Under the Permissions tab, click Add inline policy
• On the Create Policy page, select the JSON tab and use the following policy definition:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeNarrativeBucketAccessRole",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "BUCKET_ROLE_ARN"
      ]
    }
  ]
}

• Substitute BUCKET_ROLE_ARN with the role ARN under your managed bucket on the Sources page in Dataset Manager.
• Click Review Policy. On the following page give the role a name, e.g. "narrative-managed-bucket-access", and click Create Policy.

Now your user will be able access the role. Next we'll walk through configuring different tools to use the role.

Example: Access via the AWS CLI

The AWS CLI can be used to access your managed bucket. If you've never used the AWS CLI before you'll need to:

• Follow the installation instructions: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
• Perform initial configuration: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html

You can access your role by setting up a profile to automatically use the role when using the AWS CLI. This can be done by updating your configuration file (typically ~/.aws.config ) as follows:

[profile SOURCE_PROFILE]
aws_access_key_id = ...
aws_secret_access_key = ...

[profile BUCKET_PROFILE_NAME]
role_arn = BUCKET_ROLE_ARN
external_id = EXTERNAL_ID
source_profile = SOURCE_PROFILE

Substitute the following values in the example configuration:

• SOURCE_PROFILE: replace this with any named profile that contains IAM user credentials with permissions to use the role.
• BUCKET_ROLE_ARN: use the role ARN under your managed bucket on the Sources page in Dataset Manager.
• EXTERNAL_ID: if you specified an external ID when creating your managed bucket then use it here, otherwise remove the line "external_id = EXTERNAL_ID" altogether.
• BUCKET_PROFILE_NAME: the name you will pass on the command line to access your bucket, this can be anything you choose.

Now you can use the profile you just set up to access your bucket:

aws s3 ls --profile BUCKET_PROFILE_NAME s3://BUCKET_NAME/

Where BUCKET_NAME is the name of your managed bucket.

For a more detailed guide detailing how to configure the AWS CLI to use a role, reference the official AWS guide.

Example: Access via the AWS Console

You can use the AWS console to sign in as yourself then "switch roles" to gain access to your bucket. The "switch roles" action temporarily sets aside your original user permissions and instead gives you the permissions assigned to bucket role.

Identify Your Role's Name

The first step is to identify your bucket role name using your role ARN. Your role ARN is of the form:

arn:aws:iam::NARRATIVE_ACCOUNT_ID:role/ROLE_NAME

Where:

• NARRATIVE_ACCOUNT_ID is Narrative's AWS account ID
• ROLE_NAME is the name of the role.

For example, if your role ARN is:

arn:aws:iam::123456789012:role/nio-example-t2pxc2uync7h-access

Then the ROLE_NAME is "nio-example-t2pxc2uync7h-access".

Configure the Console to Switch Roles

Sign in to your AWS account as a user who access to the role (see Prerequisite: Grant User Access). The next step is to configure the console so that you switch roles.

First, choose your user name on the navigation bar in the upper right and select Switch Roles.

You'll be presented with the following form:

Use the following values:

• Account: use the value "narrative-io"
• Role: use your ROLE_NAME, found in the previous step.
• Display Name: enter text that you want to appear on the navigation bar in place of your user name when this role is active.

Click Switch Role. You'll see your chosen Display Name on the navigation bar in the upper right indicating you successfully switched to the bucket access role.

Access Your Bucket

To access your bucket in the S3 console, you'll have to follow a link directly to your bucket and will be unable to access it from the S3 console home page. Your direct link looks like the following:

https://s3.console.aws.amazon.com/s3/buckets/BUCKET_NAME/

Where BUCKET_NAME is substituted with the name of your managed bucket. We suggest bookmarking this link for your convenience.

The requirement to follow a direct link is a limitation of console when accessing S3 buckets in third party accounts.

For official guidance on assuming a role and information on how to stop using a role inside the AWS console, reference the official guide.

Next Steps

Now that your managed bucket is set up and you can access it you are ready to start ingesting data into your datasets.

See our guide for ingesting data using your managed bucket.