Getting Started with Pipeline Configuration Files

Pipeline configuration files give you the ability to define pipelines that are provisioned by Conduit at startup. It's as simple as creating a YAML file that defines pipelines, connectors, processors, and their corresponding configurations.

When you install Conduit, you'll find a pipelines directory at the same level as your Conduit binary. This is where all your pipeline configuration files will be located by default:

│ # Conduit binary
├── conduit
│ # Folder with pipeline configurations
└── pipelines
    │ # Pipeline configuration file
    └── source-to-destination.yml

tip

You can also specify a different path for the pipeline configurations directory using the -p flag when starting Conduit:

./conduit run -p /path/to/pipelines

To help you get started, we've created a pipeline configuration generator that will help you create your pipeline configuration file. Take the next steps to generate your first pipeline configuration file. Check out the different specifications to learn more about the different configuration options.

1. Give your pipeline an ID

Your pipeline ID is the unique identifier for your pipeline. It's used to distinguish your pipeline from other pipelines in the system. It must be unique within your pipeline configuration file.

2. Choose your source connector

Loading connectors...

3. Choose your destination connector

Loading connectors...

4. Copy the generated pipeline configuration file

Now, you need to include this into your previously created file and start Conduit. You should seeing log lines saying that the pipeline source-to-destination was created and ready to stream process:

$ ./conduit
/.../
2023-04-01T12:34:56+00:00 INF pipeline configs provisioned component=provisioning.Service created=["file-to-file"] deleted=[] pipelines_path=./pipelines
/.../

Conduit is now running the pipeline file-to-file which continuously reads lines added to file ./example.in and copies them to file ./example.out. Try writing a line to ./example.in and checking the content of ./example.out.

$ echo "hello conduit" >> example.in
$ cat example.out
hello conduit

tip

In our Conduit repository, you can find more examples.