Skip to main content

SFTP

File storage

The SFTP/FTP connector lists, reads and writes files on an SFTP, FTP or FTPS server. You choose the protocol on the connection.

When to use it​

  • A partner drops files on an SFTP server for you to collect, or expects files delivered there.
  • You need SSH key authentication instead of a password.
  • The file is a flat format that a Message node builds or a Source node parses.

What you need​

  • The server's host name and port, reachable from the runtime.
  • A user name and a password, or for SFTP a private key in PEM format.
  • Optionally, for SFTP, the server's host key line in known_hosts format. Without it, DataLug pins the key on the first connection.

Connect​

The connection form lists its steps on the left. Open any step from that list. The Environments step holds the values that differ per environment.

Test connection is in the form footer. Beside it, Test against picks Base, the connection as saved, or an environment, whose values are applied first.

FieldStepWhat to enter
ProtocolConnectionSFTP, FTP or FTPS. Default SFTP.
TLS ModeConnectionFTPS only. See below.
HostConnectionHost name or IP address.
PortConnection22 for SFTP, 21 for FTP and explicit FTPS, 990 for implicit FTPS.
Root PathConnectionThe folder paths resolve under. Default /.
UsernameAuthenticationThe account name.
Auth MethodAuthenticationPassword or Private Key. Private Key is offered for SFTP only.
PasswordAuthenticationFor Password.
Private Key (PEM)AuthenticationFor Private Key, such as an OpenSSH private key.
Passphrase (Optional)AuthenticationFor an encrypted private key.
Strict Host Key CheckingAuthenticationSFTP only. On by default. See below.
Known Hosts (Optional)AuthenticationSFTP only. The host key line the server must match. If empty, the key is pinned on the first connection.
Timeout (ms)OptionsDefault 30000.
Keep Alive (ms)OptionsDefault 10000.

Switching Protocol to FTP or FTPS selects password authentication.

FTPS TLS mode​

TLS ModeBehaviour
ImplicitTLS from the first byte, usually on port 990.
Explicit (AUTH TLS)A plain connection that upgrades with AUTH TLS, usually on port 21.
From portPort 990 selects implicit. Any other port selects explicit.

SFTP host key verification​

With Strict Host Key Checking on, every SFTP connection checks the server's host key before it signs in.

  • Known Hosts filled in. The server's key must match the first entry.
  • Known Hosts empty. The first successful connection, from a run or from Test connection, pins the server's key. The connection screen shows the pinned key's SHA256 fingerprint, its type and when it was pinned. Every later connection to the same host and port must present the same key.

A server that presents a different key is refused with:

The SFTP server's host key changed. Check with the server owner, then reset the pinned key on the connection.

Owners and Admins can select Reset pinned key on the connection screen. The next successful connection pins the server's current key.

An imported connection does not bring pinned keys with it. It pins the key on its own first connection.

With Strict Host Key Checking off, the server's host key is not verified.

Supported operations​

NodeOperations
SourceList Files, Get File
LoadUpload File(s), Write Rows To File(s), Move/Rename, Delete, Create Folder

Reading files. Recursive and Files only shape a listing. Include (glob), Exclude (glob) and Regex (advanced) choose files by name. Parse content into rows turns each CSV file into rows, with CSV has header row and CSV Delimiter settings. Format Auto (by extension) reads a .csv file as CSV.

Writing rows to files. Under Format & Behavior:

  • Format is CSV, JSON or XLSX. Compress (.gz) compresses the file.
  • File Name Template accepts the tokens {runId}, {YYYYMMDD}, {HHmmss}, {chunkIndex}, {partIndex}, {groupKey} and {field:MyField}. Append timestamp adds {YYYYMMDD}_{HHmmss}_{SSS} when the template has none.
  • Split is Single file, One per row or Group by. Group by takes one or more field names, separated by commas.
  • Overwrite decides what happens when the file exists: Overwrite, Skip or Fail.

Choosing a path​

The remote path field in Configure Action has a Browse button that lists the folder.

  • Open enters a folder, and Up goes back. Use picks a file, and Use this folder picks the folder you are in.
  • An operation that names a folder shows folders only.
  • The field stays editable. Type a wildcard such as /inbound/*.csv.
  • Browsing reads only. Nothing is written.

Limits​

  • Plain FTP sends the password and the files unencrypted. Use SFTP or FTPS for anything sensitive.

Troubleshooting​

  • Test connection passes and a Source lists nothing. Check the path. A folder that does not exist lists as empty, not as an error. Pick the folder with Browse.
  • A file is written but does not appear where you expect. Check the target path on the Load node. A path that resolves to another folder still succeeds.
  • Large files. Staging space is limited per run, and the run log names any node that spilled to disk. See Files and staging.
  • Test connection fails with a host key error. The server's key does not match Known Hosts. Get the current key from the server's administrator and replace the line.
  • The SFTP server's host key changed. The server presented a key that does not match the pinned one. Confirm the new key with the server's owner, then ask an Owner or Admin to select Reset pinned key on the connection.
  • The connection times out on FTP or FTPS. The data channel is blocked. Check that the server's passive ports are open to the runtime, and that the TLS mode matches the server.
  • The private key is rejected. Check that the matching public key is in the user's authorized_keys on the server, and that the passphrase is correct.