Component Configuration

You can add and configure wsl-profile-fetcher in your Pipeline by adding the component as an operation.

Profile Id JSON Path

Specifies the JSON path in your source document where the profile identifier (such as a Twitter handle) is located. The profile fetcher extracts this ID and uses it to retrieve corresponding profile details from the WSL service.

For example, if your document contains:

{
  "author": {
    "handle": "example_user"
  }
}

And you set the Profile Id JSON Path to author.handle, the component will extract "example_user" and use it to fetch profile data from WSL.

Profile Mappings

This setting defines how profile data from WSL is copied into your documents.

This setting defines how data is copied from a source JSON document into a destination JSON document.

Each mapping rule contains three fields:

type: The data type of the value (string, integer, boolean, date)
source_path: The field in the WSL profile response data
destination_path: Where to place the data in your document

Example Mapping

With the Profile ID path being author.handle and given the mapping:

{
  "mappings": [
    {
      "type": "string",
      "source_path": "userid",
      "destination_path": "author.user_id"
    },
    {
      "type": "string",
      "source_path": "name",
      "destination_path": "author.name"
    }
  ]
}

If WSL returns profile data like:

{
  "userid": "12345",
  "name": "Datastreamer User",
  "followers": 1500
}

And your original document is:

{
  "author": {
     "handle": "12345"
  },
  "post_id": "p789",
  "content": "Hello world!"
}

After processing with the mapping above, your document becomes:

{
  "post_id": "p789",
  "content": "Hello world!",
  "author": {
    "handle": "12345",
    "user_id": "12345",
    "name": "Datastreamer User"
  }
}

Common Mapping Fields

WSL profile responses typically include these fields that you can map:

userid: Twitter user ID
handle: Twitter handle/username
name: Display name
bio: Profile biography
followers: Follower count
posts_count: Number of posts
verified: Account verification status
location: Profile location
profile_image_source: URL to profile image
profile_create_date: Account creation date
bio_links: Links in profile bio
is_business_account: Business account status
twitter_friends_count: Following count
twitter_favourites_count: Likes count