Google Drive
The Google Drive connector lists, reads and writes files in My Drive or a shared drive. It signs in with your own Google OAuth client.
When to use it
- The files you need are in Google Drive or a shared drive.
- A person, not a system, puts files where the data pipeline picks them up.
- You deliver generated CSV, JSON or XLSX files to a Drive folder.
What you need
- A Google Cloud project with the Google Drive API enabled.
- An OAuth client of type Web application, created in the Google Cloud Console under APIs & Services > Credentials.
- A Google account with access to the files.
One OAuth client can serve both Google Drive and Google Sheets connections.
Provider-side setup for this connector, including the app registration, the redirect URI and the permissions, is in the File hub setup guide. Connection basics and environments are on the Connections page.
Connect
The connection form lists its steps on the left. Open any step from that list. The Environments step holds the values that differ per environment.
Test connection is in the form footer. Beside it, Test against picks Base, the connection as saved, or an environment, whose values are applied first.
| Field | Step | What to enter |
|---|---|---|
| Client ID | Authentication | The OAuth client ID, ending .apps.googleusercontent.com. |
| Redirect URI | Authentication | Filled in. Add it to the OAuth client's authorized redirect URIs. |
| Client Secret | Authentication | The OAuth client secret. |
| Connected Account, Display Name | Authentication | Filled in when you connect the Google account. |
| Default Shared Drive ID | Connection | Optional. Browse a shared drive instead of My Drive. |
| Default Folder ID | Connection | Optional. The folder where browsing and every run start. Pick it with Browse. |
| Notes | Options | Free text for your team. |
The Redirect URI is filled in for you. Copy it with the copy button and add it to the app registration exactly as shown. Use Edit beside the field only when the app is registered against a different address for DataLug.
Connect the Google account on the Authentication step. Save stays disabled until
the account is connected. DataLug requests the Google Drive scope
(https://www.googleapis.com/auth/drive) and your email address and profile.
Default Folder ID stores the folder's ID and shows its name beside it. To set it by hand, copy the ID from the folder's URL in Drive. A folder name is not accepted. A node can still name its own folder.
Supported operations
| Node | Operations |
|---|---|
| Source | List Files, Get File |
| Load | Upload File(s), Write Rows To File(s), Move/Rename, Delete, Create Folder |
Reading files. Recursive and Files only shape a listing.
Include (glob), Exclude (glob) and Regex (advanced) choose files by
name. Parse content into rows turns each CSV file into rows, with CSV has
header row and CSV Delimiter settings. Format Auto (by extension)
reads a .csv file as CSV.
Writing rows to files. Under Format & Behavior:
- Format is CSV, JSON or XLSX. Compress (.gz) compresses the file.
- File Name Template accepts the tokens
{runId},{YYYYMMDD},{HHmmss},{chunkIndex},{partIndex},{groupKey}and{field:MyField}. Append timestamp adds{YYYYMMDD}_{HHmmss}_{SSS}when the template has none. - Split is Single file, One per row or Group by. Group by takes one or more field names, separated by commas.
- Overwrite decides what happens when the file exists: Overwrite, Skip or Fail.
Choosing a path
The remote path field in Configure Action has a Browse button that lists the folder.
- Open enters a folder, and Up goes back. Use picks a file, and Use this folder picks the folder you are in.
- An operation that names a folder shows folders only.
- The field stays editable. Type a wildcard such as
/inbound/*.csv. - Browsing reads only. Nothing is written.
A folder name that contains a forward slash appears in a path as %2F.
Limits
Google applies its own per-project and per-user quotas to the Drive API.
The API limits card on the connection form is collapsed by default. Every field on it is optional, and a blank field uses the default shown below it.
| Setting | What it does | Default |
|---|---|---|
| Requests per period, Period | Paces outgoing calls. Set both to throttle. | No throttle, or the rate shown on the card. |
| Page size | Rows per request. A larger value is reduced to the provider's maximum. | The maximum shown on the card, or set by the provider. |
| Max attempts | Tries per call, including the first. | 3 |
| Cool-off (ms) | The first wait after a 429. Later waits grow from it. | 500 ms |
| Requests at once | Requests open at the same time within one run. | No limit |
A Retry-After header from the provider overrides the cool-off. The pace
applies per connection and is shared by every run on it. See
Run performance.
DataLug retries a call that the provider refused with 429, and a call that
could not connect. A read is also retried after a 408, a 5xx or a dropped
connection. A write is retried only after a 429 or a failed connection.
A Retry-After delay of up to one minute is honoured. A longer one fails the
call with the 429.
Troubleshooting
- Test connection passes and a Source lists nothing. Check the path. A folder that does not exist lists as empty, not as an error. Pick the folder with Browse.
- A file is written but does not appear where you expect. Check the target path on the Load node. A path that resolves to another folder still succeeds.
- Large files. Staging space is limited per run, and the run log names any node that spilled to disk. See Files and staging.
- A connection that worked stops working. The stored authorisation was withdrawn or has expired. Open the connection and connect the account again. Editing the other fields does not restore it.
- A run fails with "Method doesn't allow unregistered callers". The request reached Google with no token. Open the connection and connect the account again. If it happens in one environment only, check that environment's values on the Environments step.
- A run fails with "Google Drive path not found". The message names the segment it could not find and lists what that folder holds. If the folder you expected is there under a slightly different name, pick the path again with Browse.
- Google shows redirect_uri_mismatch when you connect. The Redirect URI is missing from the OAuth client, or differs from it. Copy it again and add it exactly.
Related
- Google Sheets, which can share the same OAuth client.
- Source node and Load node.
- OneDrive and Dropbox.