Connecting Compatible Sources

Compatible Data Sources are ready-to-use sources in the Datastreamer pipeline but require an externally sourced API key or other authentication.

For compatible sources, Datastreamer will act as a proxy in requesting, transforming, and delivering required data. There is no need to create a Schema, as Datastreamer has already worked with compatible sources to create the schema.

Implementation checklist

To begin:

• Have the API key or authentication method to leverage the source.

1. Register the API

To register the partner API key credentials, use the "data-sources" API

Some sources require:

• API key
• Token
• Domain
• UserID
• Password
• Code

These differ per source. For the example below, we are going to proceed with an API key and private code.

curl --location --request PUT 'https://api.platform.datastreamer.io/api/data-sources/[source name]' --header 'apikey: [your API key]' --header 'Content-Type: application/json' --data-raw '{
    "apikey": "[Enter partner API key]",
    "code": "[Provided API key]"
}'

For the "source name" placeholder, enter the data source according to its Data Source Name. This can be found in the catalog listing for the source.

For example: DarkOwl's Search API would be "darkowl_search".

After registration, searching for that source will utilize your API partner-provided API key. This proxy through the partner API key occurs behind the scenes.

2. Confirm credentials have been registered.

You can confirm credentials have been registered with the following GET request.

curl --location --request GET 'https://api.platform.datastreamer.io/api/data-sources/[source_name]' --header 'apikey: [Your Datastreamer API key]'

3. Removing or replacing registered credentials.

To replace the API key you are currently using, simply register your new credentials following the above steps. It will overwrite the existing credentials.

To remove the registered credentials, send the same request as Step 1, with the type of request now being "Delete". When you perform the confirmation step (Step 2), you should now get a 404 result.

4. Search using your new connected source!

You should be all set.