Milo in the app

Guides

Milo is the AI assistant in DataLug. It answers questions about your projects and runs, builds data pipelines, writes SQL and maps fields. The Assistant only suggests; the Data Pipeline Builder changes the canvas as you answer its questions.

Open the Milo panel

Select the Ask Milo chip in the top bar. Each click moves the panel to its next state, and the chip's tooltip names it:

• closed: Open Milo opens the full panel;
• open: Minimize Milo shrinks it to a narrow icon rail;
• minimized: Expand Milo opens it again.

The panel docks on the right of the page. The rest of the page stays usable while it is open, and your conversation is kept when you minimize it.

• Drag the panel's left edge to change its width.
• Press Esc to minimize it.
• Select the close button to close it.

Choose a tab

The icon rail on the left of the panel has two tabs.

Tab	What Milo does
Assistant	Answers questions about your projects, connections and runs.
Build a data pipeline	Builds a data pipeline on the open canvas, one question at a time.

Ask the Assistant

Ask in your own words. Milo works out what you need from the question and from anything attached to it:

• Troubleshoot a failed run. Milo reads the run, the nodes that failed and the errors recorded for it, then gives the most likely cause and how to confirm it.
• Write a SQL query. Milo writes SQL against the tables and columns saved on a connection. Name the connection in your question, or ask from a page that attaches one.
• Search the docs. Milo answers from the DataLug documentation and links the page that covers the topic.

An empty chat shows three chips: Troubleshoot a failed run, Write a SQL query and Search the docs. A chip starts the question in the box for you to finish.

Type in the box at the bottom and press Ctrl+Enter (Cmd+Enter on a Mac) to send. Copy this answer copies a reply.

Earlier conversations stay in the Assistant. A message from an earlier Milo tab, or from Map fields, carries a small label such as Troubleshoot or SQL query.

To map fields with Milo, open Map fields on a Transform node. See Transform.

Build a data pipeline with Milo

The Build a data pipeline tab opens the Data Pipeline Builder. Milo asks one question at a time and draws each answer on the canvas of the data pipeline you have open. With no data pipeline open, the tab shows Open a data pipeline.

Start with a goal

Describe what the data pipeline should do, for example "Copy new orders from PostgreSQL to SQL Server every night", and select Start building. Name the source, the target and any rules.

If the canvas already has nodes, choose what Milo does with them:

• Reuse some of them keeps your nodes. Milo proposes which existing node fills each part of the pipeline: Source, Filter, Map fields, Target and Error log. Keep each proposal, pick another node of the same kind, or choose Create a new one, then select Reuse these nodes. Milo connects the chosen nodes into one pipeline from your Start node and adds only the parts that are missing. Nodes you do not choose stay on the canvas, unconnected. Answers taken from a reused node are marked From existing node and can be changed like any other.
• Replace them all removes every node on the canvas. A confirmation names how many nodes are removed before anything changes.

Milo checks every change against the canvas's connection rules before placing it. A change that would break them is not placed, and the build says which rule it breaks.

Check the plan

Before any question, the Plan step shows the data pipeline Milo plans, as a small diagram and a list of steps. For example, "Combine accounts from MySQL and Oracle, load the matching ones to Salesforce, and the non-matching ones go to Snowflake" plans two Sources, a Join, a Filter that sends the rows the Join did not match one way and the matched rows the other, and a Target on each side.

• Swap changes a step to another type that fits the same place, such as a Filter to a Router.
• Remove takes a step out and connects what came before it to what came after.
• Rebuild plan asks Milo to plan again from your goal.
• Continue starts the questions.

Milo lists what it changed in its own plan, such as a Map fields step added before each Target. A plan that cannot be placed on the canvas becomes a single chain of Source, Filter, Map fields and Target, and the Plan step says which steps could not be placed.

After Continue, the plan is read only. Change the plan goes back to it and clears every answer.

Answer the questions

While a build is open, autosave is paused for every change on the canvas, including edits you make by hand. Leaving the page asks first, as it does when autosave is off.

The build has these steps: Goal, Plan, Source, Fields, Rules, Target, Run and Review. The step bar at the top shows where you are; select a finished step to go back to it.

Each question card shows Milo's suggested answer. You can:

• select Use suggestion to put the suggestion in the card, then Save answer;
• choose your own answer and select Save answer;
• select Decide for me to let Milo answer;
• select Skip on an optional question, such as fixed values.

Answered questions stay listed above the current card with Change. Changing an answer clears the answers that depend on it, and Milo asks those again.

The questions cover:

• Source: the connection and the table or object to read. The data pipeline reads every row; set a row limit later on the Source node. New connection opens the full connection editor, the same one as on the Connections page; once saved, the new connection is in the list and selected. Milo suggests a connection only when your goal names its system.
• Webhook: when your goal starts with a webhook, the first question is which of your account's webhooks starts the data pipeline, or Read from a connection instead. A webhook adds a Webhook Trigger step in place of the Source. For a goal about leads each delivery becomes one lead row, and its fields are offered for the mapping; otherwise the payload arrives as sent and its fields are mapped on the canvas after the build. With no webhook in the account, Create a webhook opens Manage Account in a new tab, and Refresh lists the new webhook.
• Fields: the fields the data pipeline carries.
• Rules: whether to keep every row, keep rows where a field equals a value, or keep rows where an expression is true.
• Target: the connection, the table or object to write to, how rows are written, the field that matches existing records for an update or upsert, the field mapping, and optional fixed values. Email connections are not offered as a target: email is a Notification step.
• Run: when the data pipeline runs (on demand, every few hours, daily or weekly) and what happens to rows that fail to write (stop the run, or log the failed rows and continue).

For a SQL Server, PostgreSQL, MySQL, Oracle or Snowflake target, Milo asks whether to write to an existing table or create it. Create the table if it is missing adds a Create target table step to the canvas that runs before the source on the first real run. Its script is on the canvas to read and edit. A test run never runs it. Only your own answer creates a table.

On Snowflake, you then choose the database and schema for the new table. Shared databases are read only and cannot be chosen. Snowflake table and column names are written in upper case. Address and location fields from Salesforce are not added as columns; their parts are separate fields.

A connection with a long list of tables or objects shows the first 200. Search by name to find any other; a name typed in full, such as Contact or DATABASE.SCHEMA.TABLE, is checked on the connection itself. The table or object your goal names is suggested wherever it is in the list. Under the list, Showing gives how many of the matches are listed, and Show more lists the next ones. When the connection has more than the list can hold, Or type its name takes the name in full; a name the connection does not have is refused on the card.

What the list holds depends on the connection:

• Databases and Snowflake: tables and views.
• Salesforce, CRM and ticketing connections: the objects the system offers. Fields are read from the connection.
• Marketing Cloud: the Data Extensions in the business unit, by name.
• AWS S3, Azure Blob Storage, Google Drive, Dropbox and SFTP: the files in the connection's root folder. A file name typed in full, such as /exports/orders.csv, is checked on the connection. A CSV file's header row gives its fields. Set the file format on the node after the build.
• Google Sheets: spreadsheets, by name. The first sheet's header row gives the fields.

When a connection cannot be reached, the card says why and takes the name as you type it.

Some connections cannot list their tables or objects. The card says so, lists any your saved actions use, and takes the name as you type it.

On a Generic API connection the card asks for the endpoint instead. For a Source, choose the Method and enter the Path; for a Target, enter the Path, and the method is how rows are written. The path is relative to the connection's Base URL, such as /v1/orders.

When a Generic API connection has an imported spec, the card lists its operations. A DataLug API connection lists the operations of the API version it uses, with the API Studio add-on. A Source or Lookup offers read operations (GET); a Target offers write operations (POST, PUT, PATCH and DELETE). The operation's fields come from its schema. You can still enter a path instead.

While Milo works, the panel says what it is doing, such as reading a connection. After a few seconds it also says what it is waiting on.

When Milo cannot read a connection, the card says what went wrong and gives the connection's own error. From the card you can:

• Turn on connection, when the connection is turned off on the Connections page.
• Test connection, to run the same test as the Connections page.
• Edit connection, to open it in the full connection editor. Milo reads it again after you save.
• Choose another connection.
• Type the name, where the question takes a typed table or object name.
• Retry, to read the connection again. Your answers are kept.

The read order keeps a limited read to the same rows on every run and every test run. Milo suggests the field mapped to the target's key, else a created or modified date, else the primary key. The order is written into the source's query action when you save.

Fixed values set a target field to a value, or add text before or after the field mapped to it, such as DATALUG E2E before a last name. They are written into the Map fields step. A fixed value fills a field the target requires on create. Select Add a fixed value for each one, up to 20: choose the target field, Fixed value, Text before or Text after, and the text. Each row shows what the field receives, such as Name = "DATALUG E2E " + C_LAST_NAME. For text before or after, If the source value is empty chooses what the field receives when the source field is empty: Keep the text, Leave empty, or Use a default value. When you reuse a Map fields node, the fixed values it already sets are filled in and marked From existing node.

On a Salesforce text field, text Milo suggests before or after a field leaves the field empty when the source field is empty. On a field Salesforce requires on create, it writes a default instead: the text followed by Unknown. Review says what such a field receives when the source field is empty. Salesforce removes a trailing space from text, so when that changes the saved value, Review shows it, such as if empty: saved as "DATALUG E2E", and so does the fixed value row when you edit it from Review.

With Insert, running the data pipeline again writes the same rows again; Review says so.

Tables in a shared Snowflake database can be read but not written. A Salesforce upsert matches records on an external ID or lookup field. The field mapping lists only the fields the chosen write can fill: fields the target accepts on create for an insert, and on update for an update. Record IDs and the fields Salesforce sets itself are not listed. A field the target requires on create must be mapped before you can save.

When your goal sets a field from a source field, such as Name = "DATALUG E2E " + C_LAST_NAME, the mapping uses that source field. Review warns when the mapping uses another one.

Execution hooks and execution policy are not part of the build. Set them on the node after saving.

Pipelines with several steps

When the plan has more than one Source or Target, or a step such as a Join, Filter branches or an AI step, the step bar has one step per node, named as in the plan, then Run and Review. Each node asks about itself:

• Source and Target: the same questions as above, for that node.
• Map fields: comes after its Target and maps the fields that reach it.
• Join: the join type and the field on each side that must match. Milo suggests fields with the same name on both sides.
• Filter after a Join: whether KEPT holds the rows the Join did not match, or the rows it matched. The other rows leave on DROP. The Join keeps the right side's key apart as right_<field> so the Filter can tell the two apart.
• AI: the prompt the model runs for each row, and the category that holds a row back for review. Classify is the mode that runs.
• Webhook Trigger: which webhook starts the data pipeline. Owner and Admin can create one from the card: enter a name and select Create. A generated signing secret is shown once, so copy it then. A sender whose signing secret comes from the sender is created in Manage Account. Anyone else is linked to Manage Account.
• Router: the field that picks the route, then the value that sends a row down each route, named for the Target the route ends at. Rows that match no route leave on DEFAULT.
• Data Dedupe: the field that identifies the same record, such as an email, and what leaves the step: one record per match on UNIQUE, or the duplicates on OUT1 and the rest on UNIQUE.
• Lookup: the connection and table to look values up in, the field of each row to look up, and the column of the table that matches it. Each row gains the table's columns, so they can be mapped to the Target.
• Union, Fork and Log: the step's main setting, with its default suggested.
• Notification: the connection that sends it, then for email the recipients and the subject, for Slack the channel, and the message. A goal that asks to be told by email, Slack or Teams, such as "email ops@example.com when it finishes", adds this step after the Target. The message is saved as the step's action with the data pipeline.

For a SQL Server, PostgreSQL, MySQL, Oracle or Snowflake Target, Milo asks whether to write to an existing table or create it, as above. The Create target table steps run before every Source on the first real run. A data pipeline that starts with a webhook writes to tables that exist.

A Map fields step takes fixed values after its mapping.

The Run step also asks what happens when each step fails: Stop the run, or Log the failed rows, which adds a Log step on that step's error output. A run with a failed step still ends as failed.

Compute, Reshape, SQL, Script, Profile, Cache, Variable, Document, Approval, Loop, Loop Container, Sub-workflow, Node Call, Map and Message steps are placed with their defaults; Review lists each one to set up on the canvas. Saving creates one action for each Source, Target and Notification. For each Salesforce Target, Review lists its org controls under its name.

Review and test

Review lists every decision with Edit, the field mapping with each target field's status, and the checks still open. Fix on a check opens the question it belongs to. When a node uses a saved action that has no schema, Review asks you to open Configure Action and save it.

The Source and the Target get their action as soon as your answers name it: the connection and the table or object, and for the Target how rows are written and, for an update or upsert, the matching field. The action is saved with the data pipeline. While the Source or the Target has no action, Review says which question gives it one and Save pipeline stays off.

After saving, Before the first run lists what is left to set up, including the settings a saved action is missing.

For a Salesforce target, Review also lists the automation that runs when records are written to the target object: active record-triggered flows, Apex triggers, validation, duplicate, matching, workflow and assignment rules, with counts and names. Kinds the connection's Salesforce user cannot read are marked as such. To turn them off for the run and restore them after, use the Target's Configure > Advanced > Execution Hooks > Org controls.

When the Target's execution hooks turn Salesforce org controls off or on, Review lists each one under Org controls: what the run will do, or that the connection cannot do it. Apex triggers are turned off only in a sandbox or Developer Edition org, by a user with Author Apex and Modify Metadata. A control that cannot be applied stops the Target node before it writes, unless its hook action has Skip this control if it can't be applied selected; the node then runs without it and its result shows a warning.

For a Salesforce target, Milo also asks whether to write through REST or the Bulk API. Milo suggests Bulk above 2,000 rows.

Run a test reads up to 50 rows and writes nothing. The result shows, for each step, its status and the rows in and out, and lists any problems with ways to fix them. A fix with an answer applies it at once; otherwise it opens the question. After an answer changes, the result says the test run is out of date.

• On a hosted runtime, the result also shows sample output for each step.
• When the project's default environment runs on a Private Agent, the agent runs the test inside your network, for database, Snowflake and Salesforce sources. The result shows counts and errors only; sample rows stay on your agent. An agent that needs an update says so.

A pipeline with several sources or branches reads up to 50 rows from each source. The result lists each branch, ending at its target, with the steps that feed it and the rows it would write. AI and Lookup steps are not run in a test: each is marked Not previewed and its rows pass through unchanged.

A pipeline that starts with a Webhook Trigger tests with a sample body instead:

• Last delivery reads the latest delivery still stored for the webhook. Owner and Admin only.
• Paste a body takes a JSON object or array of up to 256 KB.

Fields such as passwords and tokens in the sample are masked before the test runs.

Save

Save pipeline saves the canvas once, then saves the build, and autosave resumes. Saving also creates the source's query action and the target's load action, each named after its object with (Milo), so both can be edited in Configure Action like any other. It is available once the review has no errors. The result shows how many steps are on the canvas and what to set up before the first run. When you chose a schedule, the result shows the job it created; if your plan does not include schedules, or this month's scheduled runs are used up, the result says so.

Discard build ends the build and returns the canvas to how it was before the build, edits made by hand during it included. Nothing is saved.

To start a new project from a description, choose Describe in New Project. Milo opens the Build a data pipeline tab with your description in the goal box.

Each build uses AI credits when Milo plans the goal and suggests answers. When the credits are used up, Milo cannot start a new build; a build in progress can still be answered, tested and saved.

Ask Milo from a page

Where Milo is available on your account, an Ask Milo button opens the Assistant with what you are looking at attached. The attachment shows above the box; select the cross to remove it.

• Runtime Executions: in the failure summary of a failed run. The run is attached and "Why did this run fail?" is ready to send.
• Jobs: on a job row, to ask about that job.
• Projects: in the page header, to plan a new project with your existing projects and connections in view.

Milo in Form Studio

Milo helps in three places in Form Studio.

• Draft a layout: in New form, choose Describe it to Milo and describe the form. Milo drafts the steps, sections, fields, rules and text, using the brand kit you pick. The form opens in the studio as a draft for you to review.
• Build a brand kit: in the brand kit library, New brand kit > Ask Milo starts an import from a website. On the review screen of any import, Ask Milo suggests a role for each colour and a fix for each colour pair that fails the contrast check. Apply each suggestion you want. Milo works only from what the import found.
• Review accessibility: in the studio, Check accessibility lists missing or vague labels, headings out of order, colours that fail the contrast check and images without alt text. Each finding has a suggested fix: Accept applies it as one step you can undo, Dismiss hides it and Show selects the block. A colour from the brand kit is changed in the brand kit editor.

Milo sees the form's layout and colour values, never the people who fill it in or their answers. Each request counts toward the account's AI usage, and an account can make up to 30 an hour. These features are not available when AI assistance is switched off for hosted forms on the account.

What Milo sees

Milo sees the context of the page you ask from, such as the project, the job or the run. For a failed run it sees the status, timings, the failed nodes, the error message and the run log's error and warning lines. It does not see the messages your Log nodes emit or the values in your rows.

Milo reads a limited amount of context per request. On a large account, it leaves out the least relevant items first, such as older saved actions, and the build shows "Some workspace items were left out to fit Milo's limit."

AI credits

Each Milo answer and draft uses AI credits from your plan's monthly allowance. One credit is 1,000 tokens, counting both your question with its page context and Milo's reply. A question Milo refuses uses none.

At 80% of the month's credits, a banner at the top of the Milo panel shows how much is used. When all of them are used, Milo replies with the plan's allowance and the date the credits reset, and answers again after that date or after an upgrade. Some accounts are set to continue past the allowance; the banner then shows the credits used beyond it.

See AI credits for each plan's allowance.

Related

• Milo on the docs site
• FAQ: Milo