File hub setup guide

Connectors

This guide collects the provider-side setup steps for every File Hub connector. Each section walks through the OAuth app creation, scopes / permissions, and the values you need to capture before you create the corresponding connection inside DataLug.

For the in-app connection wizard (the DataLug side), see the Connections guide. For per-connector behaviour notes, see the connector's own page in the sidebar.

Google Drive and Google Sheets

Availability in DataLug

Shipped in DataLug

What this provider is used for

• Google Drive for file listing, download, upload, move, delete, and folder-style file-hub operations.
• Google Sheets for structured spreadsheet access through the Google Sheets API.

One Google OAuth web application can support both connectors.

What you need before you start

• A Google account with access to Google Cloud Console
• Permission to create or manage a Google Cloud project
• Permission to configure the OAuth consent screen
• The exact Redirect URI shown in the DataLug connector modal

Provider setup steps

1. Sign in to Google Cloud Console.
2. Create a new Google Cloud project for DataLug or select an existing one.
3. Open APIs & Services -> Library.
4. Enable:
   • Google Drive API
   • Google Sheets API
5. Open APIs & Services -> OAuth consent screen.
6. Configure the consent screen.
   • Use Internal if you want to limit sign-in to your Google Workspace organization.
   • Use External if needed. If the app is not verified yet, add the users who will test sign-in.
7. Open APIs & Services -> Credentials.
8. Create OAuth client ID.
9. Choose application type Web application.
10. Add the exact DataLug Redirect URI from the modal to Authorized redirect URIs.
11. Save the app and copy the generated Client ID and Client Secret.

Permissions / scopes / roles

• DataLug uses server-side OAuth code exchange for Google.
• Keep API enablement limited to the Google services you actually use:
  • Google Drive API
  • Google Sheets API
• Restrict who can manage the app registration and connector because the connected Google account can access provider data.

Values to capture for DataLug

• Client ID
• Client Secret
• Redirect URI

DataLug-specific notes

• DataLug uses a backend-owned OAuth callback.
• Google must redirect back to the exact callback URL shown in the DataLug modal.
• DataLug stores the client secret and OAuth tokens securely on the server.
• After save, the client secret is write-only and is not returned to the browser.

Common mistakes

• Redirect URI mismatch
  • The redirect URI in Google Cloud must exactly match the one shown in DataLug.
• Google Drive API or Google Sheets API not enabled
  • OAuth may succeed, but connector actions can still fail later.
• External app without test users
  • Unverified external apps only allow listed test users.
• Wrong app reused across environments
  • DEV and PROD often need different registered redirect URIs.

Official provider documentation

• Google OAuth for web server applications: https://developers.google.com/identity/protocols/oauth2/web-server
• Google OAuth consent screen: https://developers.google.com/workspace/guides/configure-oauth-consent
• Google Drive API: https://developers.google.com/drive/api
• Google Sheets API: https://developers.google.com/workspace/sheets/api

OneDrive

Availability in DataLug

Shipped in DataLug

What this provider is used for

• OneDrive file listing, download, upload, move, delete, and folder-style operations using Microsoft Graph

What you need before you start

• Access to Microsoft Entra ID
• Permission to create or manage app registrations
• Permission to grant delegated Microsoft Graph permissions or request admin consent
• The exact Redirect URI shown in the DataLug modal

Provider setup steps

1. Sign in to Microsoft Entra admin center.
2. Open App registrations.
3. Select New registration.
4. Enter a name for the DataLug app registration.
5. Choose the supported account type that matches your tenant and sign-in requirements.
6. Under redirect URIs, add a Web redirect URI using the exact callback shown in the DataLug modal.
7. Do not register the DataLug callback under Single-page application (SPA).
8. Create the app registration.
9. Open Certificates & secrets.
10. Create a new client secret and copy the secret value immediately.
11. Open API permissions.
12. Add delegated Microsoft Graph permissions:
    • offline_access
    • openid
    • profile
    • User.Read
    • Files.ReadWrite
13. Grant consent if your tenant requires admin approval.
14. Copy the application Client ID.
15. Decide the Tenant ID value:
    • use the specific tenant directory ID for single-tenant apps
    • use common only when the app registration supports that sign-in model

Permissions / scopes / roles

• Delegated Microsoft Graph permissions:
  • offline_access
  • openid
  • profile
  • User.Read
  • Files.ReadWrite
• Use least privilege and avoid broader Graph permissions unless you truly need them.

Values to capture for DataLug

• Client ID
• Client Secret
• Redirect URI
• Tenant ID

DataLug-specific notes

• DataLug uses a backend-owned OAuth callback and redeems the authorization code on the server.
• The DataLug callback must be registered as Web, not SPA.
• After save, the client secret is write-only and not returned to the browser.

Common mistakes

• SPA selected instead of Web
  • This can trigger AADSTS9002325 because Microsoft expects PKCE for browser-style code redemption.
• Wrong Tenant ID
  • Single-tenant apps should use the actual directory ID, not common.
• Redirect URI mismatch
  • The Entra app registration must use the exact callback shown in DataLug.
• Copied the secret ID instead of the secret value
  • Microsoft only shows the secret value once.

Official provider documentation

• Register an application in Microsoft Entra ID: https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app
• Microsoft Graph permissions reference: https://learn.microsoft.com/en-us/graph/permissions-reference

Azure Blob Storage

Availability in DataLug

Shipped in DataLug

Current DataLug direction already covers:

• SAS token
• service principal
• source actions: List Files, Get File
• target actions: Upload File(s), Write Rows To File(s), Move/Rename, Delete, Create Folder

What this provider is used for

• Azure Blob container-based file-hub storage
• File operations under a selected container and optional path prefix

What you need before you start

• An Azure subscription
• A storage account and blob container
• One of these auth paths:
  • SAS token
  • service principal with Azure RBAC

Provider setup steps

Option A: SAS token

1. Sign in to the Azure portal and open the target Storage account.
2. Confirm that the target blob Container already exists, or create it first under Data storage -> Containers.
3. Open the storage account or container SAS generation experience.
4. Prefer a container-scoped SAS rather than an overly broad account-level SAS whenever your operating model allows it.
5. Choose an expiry window that matches your rotation policy.
6. For source-only scenarios, grant only the minimum permissions needed.
7. For target/write scenarios, add only the create/write/delete permissions that DataLug will actually use.
8. Generate the SAS and copy the token value securely.
9. Record the storage account name, container name, and any default path prefix you want DataLug to use.

Recommended SAS direction:

• read-only source scenarios: Read + List
• write or target scenarios: Read, Add, Create, Write, Delete, List

Option B: service principal

1. Register an app in Microsoft Entra ID.
2. Create a client secret and save it securely.
3. Capture:
   • Tenant ID
   • Client ID
   • Client Secret
4. Open the target storage account or container in Azure portal.
5. Open Access control (IAM).
6. Assign an Azure RBAC role to the service principal at the narrowest sensible scope:
   • storage account
   • or container
7. Wait for role propagation before testing from DataLug.
8. Record the storage account name, container name, and any default path prefix you want DataLug to use.

Recommended RBAC roles:

• Storage Blob Data Reader for read-only source scenarios
• Storage Blob Data Contributor for upload/write/delete scenarios

Permissions / scopes / roles

• SAS should be limited to only the container and permissions DataLug needs.
• Service principals should use Azure RBAC at storage-account or container scope.
• Prefer Storage Blob Data Contributor only when write/delete behavior is required.

Values to capture for DataLug

Shared fields:

• Storage Account
• Container
• Default Path Prefix (optional)
• Auth Method

For SAS token:

• SAS Token

For service principal:

• Tenant ID
• Client ID
• Client Secret

DataLug-specific notes

• Blob paths are treated as virtual directories under the selected container.
• Create Folder uses a hidden .datalug-folder marker blob for compatibility with folder-style file workflows.
• Default Path Prefix scopes action paths below the selected container and is a good way to separate environments or projects.
• After save, SAS tokens and client secrets are write-only and are not returned to the browser.

Common mistakes

• Expired SAS token
• SAS scoped too broadly or too narrowly
• Missing Storage Blob Data Reader or Storage Blob Data Contributor role assignment
• Wrong container name or wrong path prefix
• Assuming Blob Storage has real directories instead of virtual paths

Official provider documentation

• SAS overview: https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview
• Authorize blob access with Microsoft Entra ID: https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory
• Assign Azure role for blob data access: https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access

SharePoint Files

Availability in DataLug

Shipped in DataLug

What this provider is used for

• SharePoint document libraries and folder/file operations through Microsoft Graph
• Site and library-aware file access, beyond basic OneDrive root semantics

What you need before you start

• Access to Microsoft Entra ID
• Access to the SharePoint sites and document libraries you plan to use
• Permission to create app registrations and grant Microsoft Graph delegated permissions

Provider setup steps

1. Sign in to Microsoft Entra admin center.
2. Open App registrations.
3. Create a new app registration for DataLug or reuse an approved Microsoft Graph app registration if your governance model allows it.
4. Choose the supported account type that matches your tenant and sign-in model.
5. Add the exact SharePoint callback from the DataLug modal as a Web redirect URI.
6. Do not register the DataLug callback as Single-page application (SPA).
7. Create a client secret and store the secret value securely.
8. Open API permissions.
9. Add delegated Microsoft Graph permissions for SharePoint file access.
10. Grant consent if your tenant requires admin approval.
11. Make sure the user who will connect from DataLug can actually access the target SharePoint site and document library in Microsoft 365.
12. In DataLug, open Connections, choose SharePoint Files, and enter:

• Client ID
• Client Secret
• Redirect URI
• Tenant ID

13. Click Connect SharePoint.
14. Complete Microsoft sign-in in the popup.
15. In the modal, click Load Sites and select the SharePoint site.
16. Click Load Libraries and select the document library.
17. Optionally choose a Default Folder Path.
18. Click Test Connection.
19. Click Save.

Permissions / scopes / roles

Delegated Microsoft Graph permissions for SharePoint file work:

• offline_access
• openid
• profile
• User.Read
• Files.ReadWrite.All
• Sites.Read.All

Use broader Graph permissions only when your operating model truly needs them.

Values to capture for DataLug

• Client ID
• Client Secret
• Tenant ID
• Redirect URI
• SharePoint Site
• SharePoint Document Library
• optional Default Folder Path

DataLug-specific notes

• SharePoint adds site and document library selection on top of Microsoft Graph file auth.
• DataLug uses a backend-owned OAuth callback and server-side code redemption.
• The callback must be registered as Web, not SPA.
• Provider-side access still matters after OAuth succeeds: if the connected account cannot browse the site or library in SharePoint, DataLug cannot load it either.
• You can reuse the same Entra app as OneDrive if your governance model allows it, but separate registrations may still be preferable.

Common mistakes

• Treating SharePoint exactly like OneDrive and ignoring site/library structure
• Requesting broader Graph permissions than needed
• Registering the callback under SPA instead of Web
• Connecting the account successfully but not selecting a site and document library in the DataLug modal

Official provider documentation

• Register an application in Microsoft Entra ID: https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app
• Microsoft Graph permissions reference: https://learn.microsoft.com/en-us/graph/permissions-reference
• Selected permissions for OneDrive and SharePoint: https://learn.microsoft.com/en-us/graph/permissions-selected-overview

Dropbox

Availability in DataLug

Shipped in DataLug

What this provider is used for

• Dropbox folder and file operations through the Dropbox API

What you need before you start

• A Dropbox account or Dropbox team account
• Access to Dropbox App Console
• A decision to use App Folder access for the v1 DataLug connector

Provider setup steps

1. Sign in to Dropbox App Console.
2. Create a new Dropbox app.
3. Choose Scoped access.
4. Choose App Folder for the v1 DataLug connector.
5. Name the app and create it.
6. Open the app Permissions tab.
7. Select only the scopes needed for file and folder operations.
8. Save the permission changes before leaving the page.
9. Open the app settings page.
10. Add the exact redirect URI shown in the DataLug Dropbox modal as an OAuth 2 redirect URI.
11. Copy the App key and App secret.
12. In DataLug, open Connections, choose Dropbox, and enter:
    • Client ID
    • Client Secret
    • Redirect URI
    • optional Default Path Prefix
13. Click Connect Dropbox.
14. Complete Dropbox sign-in in the popup.
15. Confirm the connected account is shown in the modal.
16. Click Test Connection.
17. Click Save.

Permissions / scopes / roles

• Prefer App Folder for Dropbox v1 in DataLug.
• Use the smallest Dropbox scopes required for file and folder operations.
• Keep the app registration and scopes aligned to personal Dropbox or simple app-folder scenarios first.
• Dropbox Business team content adds extra complexity around team spaces, team folders, and path roots and is outside the v1 connector scope.

Values to capture for DataLug

Prepare these now:

• App Key / Client ID
• App Secret / Client Secret
• Exact Redirect URI
• Optional Default Path Prefix

DataLug-specific notes

• DataLug uses a backend-owned OAuth callback and server-side code exchange.
• The v1 connector is scoped to App Folder + personal Dropbox behavior.
• Default Path Prefix is relative to the authorized Dropbox app folder root.
• Client secrets are write-only after save and are not returned to the browser in plaintext.
• OAuth tokens and secrets are stored on the server.

Common mistakes

• Choosing Full Dropbox instead of App Folder
• Selecting too many scopes
• Assuming Dropbox Business team content behaves the same as a personal Dropbox root
• Using a redirect URI that does not exactly match the DataLug modal
• Expecting the browser to redeem the auth code instead of the DataLug backend callback

Official provider documentation

• Dropbox OAuth Guide: https://developers.dropbox.com/oauth-guide
• Dropbox File Access Guide: https://developers.dropbox.com/dbx-file-access-guide
• Dropbox Team Files Guide: https://developers.dropbox.com/en-us/dbx-team-files-guide

AWS S3

Availability in DataLug

Shipped in DataLug

DataLug now ships an AWS S3 Bucket connector with List Files, Get File, Upload File(s), Write Rows To File(s), Move/Rename, Delete, and Create Folder actions.

What this provider is used for

• S3 bucket and object storage for file-hub workflows
• Bucket + prefix based file listing, download, upload, move, and delete patterns

What you need before you start

• An AWS account
• Permission to create or manage an S3 bucket
• Permission to create or manage an IAM user and policies

Provider setup steps

1. Sign in to AWS Management Console.
2. Create or identify the S3 bucket DataLug should use.
3. Confirm the AWS Region for that bucket and keep it consistent with the value you will enter in DataLug.
4. Decide whether DataLug should use the whole bucket or a narrower prefix inside the bucket.
5. Open IAM and create a dedicated IAM user for DataLug if you are using the initial access-key path.
6. Attach the narrowest identity-based policy possible for the target bucket and, when practical, the target prefix.
7. If your organization uses restrictive bucket policies, make sure the bucket policy also allows that IAM user to perform the required S3 actions.
8. Create an access key for that IAM user.
9. Save the Access key ID and Secret access key immediately and securely.
10. Record the bucket name, region, and optional default prefix you want DataLug to use.

Permissions / scopes / roles

Initial documented path:

• access key + secret

Later direction:

• IAM role or temporary credentials

Policy direction:

• Source-only reads typically need:
  • s3:ListBucket
  • s3:GetObject
• Write/target workflows typically add:
  • s3:PutObject
  • s3:DeleteObject
• Folder-style creation also relies on s3:PutObject because DataLug writes a zero-byte marker key ending in /.
• Scope the policy to the exact bucket and, when possible, a specific prefix.

Values to capture for DataLug

Prepare these now:

• Bucket
• Region
• Default Prefix (optional)
• Access Key ID
• Secret Access Key

DataLug-specific notes

• The file-hub roadmap aligns S3 around bucket + prefix semantics.
• DataLug currently expects:
  • Access Key ID
  • Secret Access Key
  • Region
  • Bucket
  • Default Prefix (optional)
• Create Folder writes a zero-byte object key ending in / so S3 behaves like the rest of the file-hub connector family.
• Default Prefix is the easiest way to scope one bucket into separate DataLug environments or project paths without creating many buckets.
• This guide documents the initial access-key path because that is the current shipped baseline.
• Role-based or temporary-credential flows can follow later, but they are not the primary documented setup path here.

Common mistakes

• Overly broad IAM permissions
• Using the wrong bucket Region
• Forgetting s3:ListBucket for prefix discovery
• Storing access keys insecurely or failing to rotate them
• Using one shared IAM user for too many unrelated environments

Official provider documentation

• Creating S3 buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-buckets-s3.html
• Manage access keys for IAM users: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
• Identity-based policy examples for S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html
• Bucket policies for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html