> ## Documentation Index
> Fetch the complete documentation index at: https://docs.allyy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data

> Documentation for setting up and managing data sources

export const ImageText = ({imageUrl, imageAlt, children, rightImagePosition = false}) => {
  const getFullImageUrl = url => {
    if (typeof window !== 'undefined' && (window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1' || window.location.hostname.startsWith('192.168.') || window.location.hostname === '[::1]')) {
      return url;
    }
    return `https://mintlify.s3-us-west-1.amazonaws.com/allyyio${url}`;
  };
  const fullImageUrl = getFullImageUrl(imageUrl);
  const containerStyle = {
    display: 'flex',
    alignItems: 'center',
    flexDirection: rightImagePosition ? 'row' : 'column',
    width: '100%'
  };
  const imageStyle = {
    width: '100%',
    marginRight: rightImagePosition ? '20px' : '0',
    marginTop: rightImagePosition ? '0' : '10px'
  };
  const textStyle = {
    width: rightImagePosition ? '80%' : '100%'
  };
  return <div style={containerStyle}>
      <div style={textStyle}>
        {children}
      </div>
      <img src={fullImageUrl} style={imageStyle} alt={imageAlt} />
    </div>;
};

Integrating data into the system is key to making your machine learning models work effectively. This process involves setting up a **data source** that can be automated once configured. All operations occur at the **dataset level**, so understanding how to create and manage datasets and data sources is crucial.

***

## **Overview of the Process**

Here’s an outline of the steps involved in setting up a data source:

<Steps>
  <Step title="Create a Dataset">
    This is the container where your data sources will live. Think of it as a project-level container that can hold multiple data sources.

    <ImageText imageUrl="/images/processes/data/data_1.png" imageAlt="Dataset creation">
      This image shows how to create a new dataset in the system. Datasets allow you to organize your data sources for different business cases.
    </ImageText>
  </Step>

  <Step title="Add a Data Source">
    Within the dataset, create multiple data sources to load the necessary data.

    <ImageText imageUrl="/images/processes/data/data_2.png" imageAlt="Add data source screen">
      This image displays the various data source connection options such as CSV, JSON, Google BigQuery, and Microsoft SQL Server.
    </ImageText>
  </Step>

  <Step title="Select the Source">
    Choose the data source you want to integrate from the list of supported sources:

    * CSV
    * JSON
    * Google BigQuery
    * Google Cloud Storage
    * Microsoft SQL Server
    * SFTP
    * Agillic
    * Active Campaign

    <ImageText imageUrl="/images/processes/data/data_3.png" imageAlt="Data source selection options" />
  </Step>

  <Step title="Enter Credentials">
    Input the credentials required to connect to the data source. This might involve entering project IDs, API keys, or authentication tokens, depending on the source.

    <ImageText imageUrl="/images/processes/data/data_4.png" imageAlt="Enter credentials">
      This image shows examples of credential input for Google BigQuery and Microsoft SQL Server.
    </ImageText>
  </Step>

  <Step title="Preview Data">
    Once connected, preview the data to ensure the source is correctly connected and the data format is as expected.

    <ImageText imageUrl="/images/processes/data/data_5.png" imageAlt="Data preview">
      In this image, you can see a preview of the data fields (e.g., ContactID, Gender, Birthdate) before confirming the connection.
    </ImageText>
  </Step>

  <Step title="Create Data Mapping">
    After previewing, map the data fields to specific entities in the **allyy data structure**. Mapping allows the system to understand the relationship between fields in your data and allyy's internal structure (Contacts, Responses, Offers, etc.).

    Read more about [data mapping here](#data_mapping).

    <ImageText imageUrl="/images/processes/data/data_6.png" imageAlt="Data mapping screen">
      This image shows the mapping process, where fields like ContactID and OfferResponse are mapped.
    </ImageText>
  </Step>

  <Step title="Save the Mapping">
    Once the mapping is saved, the data source can be accessed or synchronized at any time.
  </Step>

  <Step title="Data Synchronization or Workflow">
    For **batch data**, click the **synchronize** button to pull in data periodically or on-demand.

    For **streaming data**, use the **start streaming** or **stop streaming** buttons to manage real-time data ingestion.
  </Step>

  <Step title="View Details and History">
    You can inspect the data source for details such as source information, last sync status, errors, and history of uploads.
  </Step>
</Steps>

***

## **Streaming vs Batch**:

<CardGroup cols={2}>
  <Card title="Batch Data" icon="square-1">
    Synchronization happens manually or on a schedule. To pull in data, click the **synchronize** button or schedule it via a workflow.
  </Card>

  <Card title="Streaming Data" icon="square-2">
    Data flows in real time. Use the **start/stop streaming** buttons to manage continuous data flow. Streaming data cannot be scheduled.
  </Card>
</CardGroup>

***

## **Conclusion**

Setting up data sources allows for seamless integration of external data into the system.
By following this process, you can configure a data source once and reuse it indefinitely, either by manually pulling in data or using real-time streaming.
Ensuring that the data is correctly mapped to the **Allyy data structure** is crucial for leveraging it effectively in models and predictions.