Pipelines: Management

Synopsis

The Pipelines interface is where pipelines are created and managed from the web interface: the My Pipelines list, the create flow, the pipeline detail view (metadata plus a YAML editor for processor logic), and the review of updates for pipelines installed from the Content Hub. The pipeline YAML syntax itself is documented in Overview.

Accessing Pipelines

From the sidebar, select Content Management > Pipelines. The page opens on My Pipelines.

My Pipelines

My Pipelines lists every pipeline in your organization. A toolbar provides a Search pipelines field, a device-type filter, and a Create new pipeline button. A view switcher toggles between a card view and a table view.

The table view shows these columns:

Column	Description
Title	Pipeline display name
Device Type	Associated device type
Device Vendor	Associated vendor
Target	Associated device family
Last updated	Time of the most recent change
Created	Creation time

An In use tag marks pipelines applied on routes, devices, or targets. A renewal icon with an Update available tooltip marks pipelines that have a pending Content Hub update.

The row action menu provides:

• Manage pipeline — opens the pipeline detail view
• Delete pipeline — deletes the pipeline after confirmation

Creating a Pipeline

Click Create new pipeline to open a three-step wizard.

Creation Method

A pipeline is either built from scratch or converted from a definition written for another platform:

Method	Produces
Create new	An empty pipeline that you build processor by processor
Import from KQL	A pipeline converted from a Microsoft Sentinel ASIM parser or a raw KQL query
Import from Cribl	A pipeline converted from a Cribl Stream pipeline export
Import from Logstash	A pipeline converted from a Logstash pipeline configuration

The import methods run the same converters as the hosted MCP server's conversion tools, so the same source definition produces the same pipeline whichever route you take. What each converter translates, and where it degrades, is documented in Migration.

Selecting an import method reveals a Source definition field. Paste the definition into it, or click Upload file to read one in. The picker accepts a different set of extensions per source:

Method	Accepted files
Import from KQL	.kql, .txt, .yaml, .yml
Import from Cribl	.yml, .yaml, .json
Import from Logstash	.conf, .cfg, .txt

An upload only fills the field, so a loaded definition can still be edited before it is converted. A source definition may be at most 5 MB, enforced on the pasted text and the uploaded file alike.

The definition is converted when you continue. One that cannot be parsed keeps you on this step, and the message reports what the converter objected to rather than a generic failure.

Pipeline Details

Provide the general information:

Field	Required	Notes
Title	Yes	Display name, 2 to 149 characters
Description	Yes	Up to 99 characters
Name	Yes	Identifier; lowercase letters, digits, and underscores only, up to 150 characters
Device type	No	Optional device association
Device vendor	No	Optional device association
Target	No	Optional device family association

A KQL or ASIM source that carries its own parser name pre-fills Title and Name from it, the name lowercased with everything outside a-z, 0-9 and _ replaced by underscores — so an ASIM parser named vimDnsCiscoUmbrella arrives as vimdnsciscoumbrella. A name you have already typed is never overwritten. Cribl and Logstash definitions carry no name, so both fields stay empty for them.

Review and Complete Setup

The final step summarizes the creation method and the pipeline details, each with a Change link back to its step.

For an imported pipeline it also shows the converted pipeline preview — the YAML that will be created. Read it before completing. A converter that meets a construct it cannot translate does not fail: it degrades the construct into a Comment processor and adds a warning line to the pipeline's description. This preview is where those warnings surface, and they are the list of things still to finish by hand.

Click Complete setup to create the pipeline, or Back to make changes.

After creation, you land on the new pipeline's Pipeline Overview tab, where you define its processor logic.

Pipeline Detail

The detail view has two tabs:

• General Overview — shows the pipeline's Title, Description, and Device Information (device type, vendor, and target), alongside a Pipeline Information panel listing the created and last-updated times. Click Manage pipeline details to edit, then Save changes.
• Pipeline Overview — a YAML editor for the pipeline's processor configuration. A tree on the left lists the main pipeline and any child pipelines; select a node to edit it, use Add new child pipeline to add one, and click Save changes on selected pipeline to save. Processor syntax is covered in Overview.

The detail header provides a Pipeline Editor button and an Actions menu.

Click Pipeline Editor to open the pipeline in the full-page Pipeline Editor.

The Actions menu provides:

• Review update — shown when a Content Hub update is available (see Reviewing Content Hub Updates)
• Add child pipeline — adds a child pipeline (see Adding a Child Pipeline)
• Open pipeline debugger — opens the pipeline in the debugger
• Delete entire pipeline — deletes the pipeline

A pipeline that is applied on routes, devices, or targets cannot be deleted until it is removed from those components.

Adding a Child Pipeline

A child pipeline is added from Actions > Add child pipeline, or from Add new child pipeline in the Pipeline Overview tree. Both open the same modal, which offers the same four creation methods as the create wizard:

• Child pipeline name — lowercase letters, digits, and underscores only, up to 150 characters
• Creation method — Create new for an empty child, or an import method to convert a definition into one

An import method reveals the same Source definition field, with the same accepted file types and the same 5 MB limit. The definition is converted when you submit, and a conversion that fails leaves the modal open with the converter's message.

The new child is added to the pipeline's tree and opened in the editor.

Pipeline Staging

Staging gives a pipeline a second, editable copy so changes can be worked on and deployed to a subset of Directors before they reach the main version.

A pipeline carries a tag saying which it is: Main or Staging.

Creating a Staging Version

Create staging version, from the detail page Actions menu or the row menu in My Pipelines, makes a full copy of the pipeline — its child pipelines and its unit tests included — tagged Staging.

From that point until the staging version is deleted:

• The main pipeline is read-only. Its detail page shows a This pipeline is read-only alert with an Open staging pipeline action, and its editor is read-only too. Every change, including Content Hub updates, happens on the staging pipeline.
• The main pipeline cannot be deleted while a staging version of it exists. Delete the staging version first.
• Staging Directors deploy the staging version; every other Director keeps the main version. This is what makes a staged rollout possible.

The Actions menu also gains Open staging pipeline and Open main pipeline for moving between the two.

Syncing Back to Main

Sync changes to main copies the staging version over the main one.

warning

The main pipeline's content, settings and unit tests are overwritten — this is not a merge, and nothing is compared first. Every Director using the main version is then redeployed.

A sync leaves the staging pipeline in place, so the main pipeline stays read-only until you delete the staging version.

Creating a staging version requires PIPELINE_CREATE; syncing requires PIPELINE_EDIT.

Editing, Debugging, and Testing

Three companion pages cover working on a pipeline's processor logic:

• Editing — the full-page Pipeline Editor for editing the main pipeline and its child pipelines as YAML tabs, opened with the Pipeline Editor button.
• Debugging — the interactive debugger for running test data through the pipeline, opened from Actions > Open pipeline debugger.
• Source Control — the review step every pipeline edit passes through before it reaches your directors: uncommitted changes, commits, merging, and restoring an earlier version.
• Unit Tests — saved tests that pin a pipeline's behavior, and can be made a condition of merging.
• MCP Server — VirtualMetric's hosted MCP server that lets an AI coding agent test, validate, and author pipelines outside the web interface.

Reviewing Content Hub Updates

When a pipeline installed from the Content Hub has updates, a Pipeline update available alert appears in the detail view and Review update becomes available in the Actions menu.

The Review Update view presents the main pipeline and its child pipelines in a tree, with a side-by-side diff of the current and updated configuration for the selected pipeline. Click Accept Changes to apply the update to the selected pipeline, or Accept All Changes to apply every pipeline that has updates. Pipelines can be updated individually, so partial updates are possible.