Custom Functions has received a number of features and usability improvements:

• You can now resize, expand the menu to add your Custom Functions.
• Testing the Custom Function within Portal now shows detailed results and any errors detected.
• Logging on the Custom Functions component now provides detailed error logging to assist in troubleshooting.
• You can also now use the Custom Functions to log messages into the component Logging.

A number of upgrades to the user interface have been rolled out for Portal. These updates and fixes include:

1. Logging and Metrics have been rolled out to additional components.
2. Keys & Secrets page
   1. The API keys and Secrets section are now clearly separated to ensure clarity
   2. A number of styling changes have been released to better match the overall application styling.
3. Homepage: When you enter Portal, the Pipelines page shows as the default.
4. The "Pipelines" page has had a number of UI improvements:
   1. The Pipeline ID is now visible and easy to copy.
   2. Pipeline statuses are clearer.
   3. Pipeline descriptions now show on the primary list.
   4. "Last Modified" is now visible on the Pipeline table.
5. Component Picker has now received additional styling improvements to better match the key information points that customers preferred.

We've released improvements and increased documentation for the SFTP egress and ingress components. Improvements in collation types have also been created.

For additional documentation, read more here: https://docs.datastreamer.io/docs/sftp-storage-egress#/

The Databricks egress options have been expanded and split into two Databricks egress options for your Datastreamer-hosted Pipelines.

• New Databricks Files Egress: https://docs.datastreamer.io/docs/databricks-file-egress#/
• New Databricks SQL Egress: https://docs.datastreamer.io/docs/databricks-sql-egress#/

The new WebSightLine profile fetcher integrates social media profile data into JSON documents, hydrating social media posts with additional profile context.

To learn more about the details of this feature, you can access the following documentation" https://docs.datastreamer.io/docs/profile-fetcher#/

This new API helps customers to better estimate and measure Job usage for billing or monitoring.

This API retrieves the DVU count for executed jobs. Results can be grouped either by DataSource or JobId (default), depending on the parameter group_by.

To see the documentation for the the Jobs DVU Count API, you can access it here: https://docs.datastreamer.io/docs/jobs-dvu-count-api#/

Logging and metrics have been improved and added to more classifier and enrichments.

For more details on the Logging feature present in components, you can access the detailed documentation page here: https://docs.datastreamer.io/docs/view-component-logs#/

With the release of Datastreamer's MCP server, you can now integrate your AI features closer to your data pipelines!

If you have not read the "CTO Brief" on Datastreamer's strategy of being the agent interface for social data, you can access it here: https://datastreamer.io/agent-interface-for-social-data-the-cto-edition/

Inside this update, you can now access the Datastreamer MCP server and the first of our AI tools "Create Job" which allows you to create data collection jobs with natural language. This gives your new AI features, the ability to easily access the data they need.

Read more about the Create Job tool here: https://docs.datastreamer.io/docs/job-creation-agent#/

Get started connecting your MCP client here: https://docs.datastreamer.io/docs/mcp-server-setup-guide#/

When you create a Periodic Job, the default behaviour is to collect any new content matching your query since the last run time. Some customers, working to identify changes in the content's metrics, have requested greater flexibility in coverage time.

With the "Query Start Time Adjustment" option in all Periodic Jobs, you can now set an adjusted search start time. You can now set the number of seconds to adjust the query start time backwards in time. For example: search hourly, for the previous two hours. This is best used with updatable storage (like Searchable Storage) to ensure updated fields are registered.

This new component can filter out duplicates with 24-hour rolling windows by allowing you to select configurable deduplication fields. You can use this filter to move duplicate content into another stream, storage, or trash.

Read more here: https://docs.datastreamer.io/docs/document-deduplication#/