Using edsl to scale data labeling tasks¶
This notebook shows how to use edsl tools for simulating surveys with AI to perform complex data labeling tasks. This is accomplished with the following generalized steps:
1. We identify data to be labeled.
2. We construct the data labeling tasks as a question or series of questions about the data, e.g., Rate the clarity of the following text on a scale from 0 to 10: {{ text }}. The questions can be qualitative or quantitative, and will be typical types of survey questions (multiple choice, free text, linear scale, etc.).
3. We draft personas for AI agents to reference in responding to the questions, e.g., You are an expert in ...
4. We administer the survey to the agents with the data as inputs to the questions.
Scaling individualized data labeling¶
We can add a layer of complexity to this generalized flow by administering the survey to each agent with only data that is relevant to the agent's persona, e.g., if we want an agent with a particular background to evaluate only the data that pertains to that background. This can be useful if our data is already sorted in some way that is important to our task. We can also use the tools to sort the data as needed.
We can visualize this modified flow as follows:
An example case: Evaluating job posts¶
Using a dataset of job categories and job posts as an example, we show how to create AI agents with relevant backgrounds and prompt them to evaluate the job posts in a variety of ways. This exercise consists of the following steps:
1. We use the tools to create a mock dataset, and show how to import a real dataset to use instead.
2. We construct questions we will ask about each of the job posts and combine them into a survey.
3. We create an AI agent with category expertise for each of the job categories.
4. We administer the survey to agent with (only) the job posts for the relevant category.
5. We show how to access the results using built-in print, SQL, dataframes and visualization methods.
Skip to any section:
Technical setup
Constructing data labeling tasks as questions
Combining questions into Surveys
Creating personas for Agents
Parameterizing questions with Scenarios
Running the survey
Accessing results
Please see our Getting Started page for more details on these methods and setting up the edsl tools:
https://www.goemeritus.com/getting-started
Technical setup¶
Here we import the edsl tools that we'll use and select LLMs. We will be prompted to enter an API key. Press return to skip entering a key.
In [3]:
from edsl.questions import QuestionMultipleChoice, QuestionFreeText, QuestionLinearScale, QuestionList
from edsl import Scenario, Survey, Agent, Model
from edsl.results import Results
In [4]:
Model.available()
Out[4]:
['gpt-3.5-turbo', 'gpt-4-1106-preview', 'gemini_pro', 'llama-2-13b-chat-hf', 'llama-2-70b-chat-hf', 'mixtral-8x7B-instruct-v0.1']
The default model is GPT 3.5. Here we choose GPT 4 instead:
In [5]:
model = Model('gpt-4-1106-preview')
Next we import a dataset. For purposes of this demo we use edsl to create a mock dataset.
In [6]:
# import csv
# data = []
# with open("data.csv", "r") as f:
# reader = csv.reader(f)
# header = next(reader)
# for row in reader:
# data.append(row)
Here we use the tools to create a dataset consisting of a column of job categories (3 different types) and a column of job posts for those categories (3 posts for each type). We'll go into more detail on the edsl methods that we use to do this in later steps.
In [7]:
# Skip this step and upload your real dataset, modifying columns as needed.
import pandas as pd
def create_data():
jobs_data = pd.DataFrame(columns=["job_category", "job_post"])
# Create a list of job categories
q_job_categories = QuestionList(
question_name = "job_categories",
question_text = "List 3 categories of job posts commonly found at an online labor marketplace."
)
job_categories_list = q_job_categories.by(model).run().select("job_categories").to_list()[0][0].split(", ")
# Create job posts for a category
q_job_posts = QuestionList(
question_name = "job_posts",
question_text = """Draft descriptions for 3 job posts in the following category of an online labor marketplace:
{{ job_category }}."""
)
for job_category in job_categories_list:
scenario = Scenario({"job_category":job_category})
# Because of how job posts are typically structured, we expect this to return a dict for each job post
# We turn each job post dict into a string to add it to our dataset
job_posts_list = q_job_posts.by(scenario).by(model).run().select("job_posts").to_list()[0]
for job_post in job_posts_list:
row_df = pd.DataFrame([[job_category, str(job_post)]], columns=["job_category", "job_post"])
jobs_data = pd.concat([jobs_data, row_df], ignore_index=True)
return jobs_data
In [8]:
df = create_data()
print(df)
job_category job_post
0 Freelance Writing {'job_title': 'Freelance SEO Content Writer', ...
1 Freelance Writing {'job_title': 'Freelance Technical Writer', 'j...
2 Freelance Writing {'job_title': 'Creative Copywriter for Ad Camp...
3 Graphic Design {'title': 'Freelance Graphic Designer', 'descr...
4 Graphic Design {'title': 'Remote UI/UX Designer', 'descriptio...
5 Graphic Design {'title': 'Senior Graphic Designer', 'descript...
6 Web Development {'jobTitle': 'Front-End Developer', 'jobDescri...
7 Web Development {'jobTitle': 'Back-End Developer', 'jobDescrip...
8 Web Development {'jobTitle': 'Full Stack Developer', 'jobDescr...
Constructing data labeling tasks as Questions¶
Next we draft our data labeling tasks in the form of questions about the job posts. We choose relevant question types—multiple choice, linear scale, free text, numerical—and construct the questions with job categories and job posts as inputs.
In [9]:
q_specific_ls = QuestionLinearScale(
question_name = "specific_ls",
question_text = """
Consider the following job category at an online labor marketplace: {{ job_category }}.
Consider the following job post: {{ job_post }}.
On a scale from 0 to 10, rate how specific the job post is compared with other job posts in the same category
(0 = Very generic, 10 = Very specific).""",
question_options = [0,1,2,3,4,5,6,7,8,9,10]
)
q_generic_ls = QuestionLinearScale(
question_name = "generic_ls",
question_text = """
Consider the following job category at an online labor marketplace: {{ job_category }}.
Consider the following job post: {{ job_post }}.
On a scale from 0 to 10, rate how generic the job post is compared to other job posts in the same category
(0 = Very specific, 10 = Very generic).""",
question_options = [0,1,2,3,4,5,6,7,8,9,10]
)
q_specific_mc = QuestionMultipleChoice(
question_name = "specific_mc",
question_text = """
Consider the following job category at an online labor marketplace: {{ job_category }}.
Consider the following job post: {{ job_post }}.
How generic or specific is the job post is compared with other job posts in the same category?""",
question_options = [
"Highly generic",
"Somewhat generic",
"Neither generic nor specific",
"Somewhat specific",
"Highly specific"]
)
Combining questions into Surveys¶
Next we combine our questions into a survey that will be administered to the AI agents.
In [10]:
jobs_survey = Survey(questions = [q_specific_ls, q_generic_ls, q_specific_mc])
Creating personas for Agents¶
Next we create descriptions for personas that we will assign to AI agents. For each job category we will construct an AI agent that is an expert in the category.
We can use the .example() method to see how an Agent is constructed:
In [11]:
Agent.example()
Out[11]:
Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})
An agent can also take an optional name and parameterized traits. For example:
In [12]:
job_category = "Web design"
base_persona = "You are an experienced freelancer on online labor marketplaces."
expertise = f"You regularly perform jobs in the following category: { job_category }."
In [13]:
job_category = "Graphic design"
example_agent = Agent(name = "Example agent", traits = {"base_persona": base_persona, "expertise": expertise})
example_agent.print()
Agent Attributes ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ _name │ 'Example agent' │ │ _traits │ {'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │ │ │ 'expertise': 'You regularly perform jobs in the following category: Web design.'} │ │ _codebook │ {} │ │ _instruction │ 'You are answering questions as if you were a human. Do not break character.' │ │ set_instructions │ False │ │ dynamic_traits_function │ None │ │ current_question │ None │ └─────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┘
Parameterizing questions with Scenarios¶
Each agent will answer the survey for the set of job posts that is relevant to the agent's expertise. We do this by creating a "scenario" for each question. We can use the example.() method again to see how a Scenario is constructed:
In [14]:
Scenario.example()
Out[14]:
{'persona': 'A reseacher studying whether LLMs can be used to generate surveys.'}
Here we show how to create a Scenario for each job category/job post pair in our dataset. (Note, however, that we will do this individually for each agent when we put it all together below, as we want each agent to only evaluate job posts in their category):
In [15]:
scenarios = [Scenario({"job_category": row["job_category"], "job_post": row["job_post"]}) for _, row in df.iterrows()]
scenarios[0].print()
Scenario Attributes ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ data │ {'job_category': 'Freelance Writing', 'job_post': "{'job_title': 'Freelance SEO Content Writer', │ │ │ 'job_description': 'Seeking an experienced SEO Content Writer to create engaging and search-engine │ │ │ optimized content for our website. Responsibilities include researching keywords, producing │ │ │ articles, blog posts, and product descriptions that rank well on search engines. Must have a strong │ │ │ grasp of English and the ability to write in various tones and styles.', 'skills_required': ['SEO', │ │ │ 'Keyword Research', 'Content Creation', 'English Proficiency', 'Adaptability in Writing Style'], │ │ │ 'project_length': '3 months', 'pay_range': '$0.06 - $0.10 per word'}"} │ └───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Running the survey¶
We administer our survey by appending the components and the .run() method. In the simplest case where we want a single agent or list of agents to answer all questions with the same scenarios, this takes the following form to generate a single Results object for the survey:
results = survey.by(scenarios).by(agents).by(models).run()
We modify this form as needed to have individual agents answer the questions for category-specific job posts. Here we create a list of Results objects for each agent/survey results:
In [16]:
def data_labeling(df, survey):
results = {}
job_categories = df["job_category"].unique()
for job_category in job_categories:
# print(job_category)
# We create an agent with expertise in the job category
base_persona = "You are an experienced freelancer on online labor marketplaces."
expertise = f"You regularly perform jobs in the following category: { job_category }."
agent = Agent(name = job_category, traits = {"base_persona":base_persona, "expertise":expertise})
# agent.print()
# We take the job posts in the job category as scenarios for the survey
df_category = df[df["job_category"] == job_category]
scenarios = [Scenario({"job_category": row["job_category"], "job_post": row["job_post"]}) for _, row in df_category.iterrows()]
# print(scenarios)
# We administer the survey to the agent with our selected LLM
job_category_results = survey.by(scenarios).by(agent).by(model).run()
# job_category_results.print()
results[job_category] = job_category_results
return results
In [17]:
results = data_labeling(df, jobs_survey)
Accessing Results¶
In the previous step we created independent Results objects for our individual agents' survey results and stored them as a dict with keys = job categories for easy reference. (We also could have just created them separately, or as a list or some other convenient type.) In the next steps we should how to access results with built-in print and analytical methods.
In [18]:
len(results)
Out[18]:
3
In [19]:
results.keys()
Out[19]:
dict_keys(['Freelance Writing', 'Graphic Design', 'Web Development'])
In [20]:
type(results["Graphic Design"])
Out[20]:
edsl.results.Results.Results
Here we inspect the full results for "Graphic Design" job posts:
In [21]:
results["Graphic Design"]
Result 0
Result ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │ │ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │ │ │ Graphic Design.'}) │ │ scenario │ {'job_category': 'Graphic Design', 'job_post': "{'title': 'Freelance Graphic Designer', │ │ │ 'description': 'Seeking a creative and detail-oriented Freelance Graphic Designer to create │ │ │ compelling visuals for digital and print media. Responsibilities include collaborating with clients │ │ │ to understand their vision, producing original graphics and layouts, and ensuring brand consistency │ │ │ across all designs. Proficiency in Adobe Creative Suite and a strong portfolio are required.', │ │ │ 'requirements': 'Proven graphic designing experience, a strong portfolio, familiarity with design │ │ │ software and technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen eye for │ │ │ aesthetics and details, excellent communication skills, ability to work methodically and meet │ │ │ deadlines.'}"} │ │ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │ │ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │ │ iteration │ 0 │ │ answer │ {'specific_ls': '4', 'specific_ls_comment': 'The job post provides a general outline of what is │ │ │ expected from a freelance graphic designer, such as creating visuals for digital and print media, │ │ │ proficiency in Adobe Creative Suite, and having a strong portfolio. However, it lacks specificity │ │ │ regarding the type of clients, project scope, design specialties, or particular industries or │ │ │ markets served. This is fairly typical for a broad graphic design role, making it somewhat generic │ │ │ compared to more specialized or detailed graphic design job postings.', 'generic_ls': '8', │ │ │ 'generic_ls_comment': 'The job post is quite generic as it includes common requirements and │ │ │ responsibilities that are typical for most graphic design job listings. It lacks specific │ │ │ information about the type of projects, the industry focus, or any unique company culture or │ │ │ project-related specifics that would differentiate it from other listings.', 'specific_mc': │ │ │ 'Somewhat generic', 'specific_mc_comment': 'The job post provided is somewhat generic as it lists │ │ │ common responsibilities and requirements that are typical for a freelance graphic designer role. │ │ │ Most graphic design job posts will ask for proficiency in Adobe Creative Suite, a strong portfolio, │ │ │ and good communication skills. However, it does not delve into specifics such as the type of │ │ │ clients, the industry focus, or unique design styles or techniques that might be needed, which │ │ │ would make it more specific.'} │ │ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │ │ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │ │ │ Consider the following job post: {\'title\': \'Freelance Graphic Designer\', \'description\': │ │ │ \'Seeking a creative and detail-oriented Freelance Graphic Designer to create compelling visuals │ │ │ for digital and print media. Responsibilities include collaborating with clients to understand │ │ │ their vision, producing original graphics and layouts, and ensuring brand consistency across all │ │ │ designs. Proficiency in Adobe Creative Suite and a strong portfolio are required.\', │ │ │ \'requirements\': \'Proven graphic designing experience, a strong portfolio, familiarity with │ │ │ design software and technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen │ │ │ eye for aesthetics and details, excellent communication skills, ability to work methodically and │ │ │ meet deadlines.\'}.\n How generic or specific is the job post is compared with other job │ │ │ posts in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat generic\n\n2: │ │ │ Neither generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn a valid JSON │ │ │ formatted like this, selecting only the number of the option:\n{"answer": <put answer code here>, │ │ │ "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │ │ │ 'MultipleChoice'}, 'specific_mc_system_prompt': {'text': "You are answering questions as if you │ │ │ were a human. Do not break character. You are an agent with the following │ │ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │ │ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │ │ │ 'class_name': 'AgentInstruction'}, 'specific_ls_user_prompt': {'text': 'You are being asked the │ │ │ following question: \n Consider the following job category at an online labor marketplace: │ │ │ Graphic Design.\n Consider the following job post: {\'title\': \'Freelance Graphic │ │ │ Designer\', \'description\': \'Seeking a creative and detail-oriented Freelance Graphic Designer to │ │ │ create compelling visuals for digital and print media. Responsibilities include collaborating with │ │ │ clients to understand their vision, producing original graphics and layouts, and ensuring brand │ │ │ consistency across all designs. Proficiency in Adobe Creative Suite and a strong portfolio are │ │ │ required.\', \'requirements\': \'Proven graphic designing experience, a strong portfolio, │ │ │ familiarity with design software and technologies (such as InDesign, Illustrator, Dreamweaver, │ │ │ Photoshop), a keen eye for aesthetics and details, excellent communication skills, ability to work │ │ │ methodically and meet deadlines.\'}.\n On a scale from 0 to 10, rate how specific the job │ │ │ post is compared with other job posts in the same category\n (0 = Very generic, 10 = Very │ │ │ specific).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: │ │ │ 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting only the code of the │ │ │ option (codes start at 0):\n{"answer": <put answer code here>, "comment": "<put explanation │ │ │ here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │ │ │ 'specific_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │ │ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │ │ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │ │ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │ │ │ 'generic_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │ │ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │ │ │ following job post: {\'title\': \'Freelance Graphic Designer\', \'description\': \'Seeking a │ │ │ creative and detail-oriented Freelance Graphic Designer to create compelling visuals for digital │ │ │ and print media. Responsibilities include collaborating with clients to understand their vision, │ │ │ producing original graphics and layouts, and ensuring brand consistency across all designs. │ │ │ Proficiency in Adobe Creative Suite and a strong portfolio are required.\', \'requirements\': │ │ │ \'Proven graphic designing experience, a strong portfolio, familiarity with design software and │ │ │ technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen eye for aesthetics and │ │ │ details, excellent communication skills, ability to work methodically and meet deadlines.\'}.\n │ │ │ On a scale from 0 to 10, rate how generic the job post is compared to other job posts in the same │ │ │ category\n (0 = Very specific, 10 = Very generic).\nThe options are\n\n0: 0\n\n1: 1\n\n2: │ │ │ 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON │ │ │ formatted like this, selecting only the code of the option (codes start at 0):\n{"answer": <put │ │ │ answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', │ │ │ 'class_name': 'LinearScale'}, 'generic_ls_system_prompt': {'text': "You are answering questions as │ │ │ if you were a human. Do not break character. You are an agent with the following │ │ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │ │ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │ │ │ 'class_name': 'AgentInstruction'}} │ └───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Result 1
Result ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │ │ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │ │ │ Graphic Design.'}) │ │ scenario │ {'job_category': 'Graphic Design', 'job_post': "{'title': 'Remote UI/UX Designer', 'description': │ │ │ 'In search of a Remote UI/UX Designer to enhance user experience across our digital platforms. The │ │ │ ideal candidate will have experience in creating wireframes, storyboards, user flows, and process │ │ │ flows. The role involves designing graphic user interface elements, like menus, tabs, and widgets. │ │ │ Must be able to work independently and collaborate with the development team to implement │ │ │ designs.', 'requirements': 'Proven UI/UX design experience, portfolio of design projects, knowledge │ │ │ of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design software like │ │ │ Adobe Illustrator and Photoshop, team spirit, strong communication skills, and problem-solving │ │ │ ability.'}"} │ │ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │ │ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │ │ iteration │ 1 │ │ answer │ {'specific_ls': '7', 'specific_ls_comment': 'The job post is quite specific as it outlines the need │ │ │ for a UI/UX designer with experience in creating wireframes, storyboards, user flows, and process │ │ │ flows. It also specifies the requirement for knowledge in particular wireframe tools and design │ │ │ software, which is more detailed than a generic job description. However, it does not mention │ │ │ specific project types or industries, which could make it even more specific.', 'generic_ls': '7', │ │ │ 'generic_ls_comment': "The job post for a 'Remote UI/UX Designer' is somewhat generic, as it lists │ │ │ common responsibilities and requirements that are typical for such positions. Many UI/UX design job │ │ │ postings seek candidates with experience in wireframing, storyboarding, and creating user flows, as │ │ │ well as proficiency in tools like Adobe Illustrator and Photoshop. The mention of needing a team │ │ │ player with strong communication skills is also a standard expectation. However, it does not │ │ │ provide specific details about the company, the projects, or unique challenges that might set it │ │ │ apart from other similar listings.", 'specific_mc': 'Neither generic nor specific', │ │ │ 'specific_mc_comment': "The job post is neither highly generic nor highly specific. It outlines │ │ │ common expectations for a UI/UX designer role such as creating wireframes, storyboards, user flows, │ │ │ and designing graphic user interface elements. The requirements listed are also typical for this │ │ │ type of position, including experience, a portfolio, knowledge of wireframe tools, and design │ │ │ software skills. However, it does not provide highly specific details such as the company's │ │ │ industry, the exact nature of the digital platforms, or the particular design aesthetics they are │ │ │ looking for, which would make it more specific."} │ │ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │ │ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │ │ │ Consider the following job post: {\'title\': \'Remote UI/UX Designer\', \'description\': \'In │ │ │ search of a Remote UI/UX Designer to enhance user experience across our digital platforms. The │ │ │ ideal candidate will have experience in creating wireframes, storyboards, user flows, and process │ │ │ flows. The role involves designing graphic user interface elements, like menus, tabs, and widgets. │ │ │ Must be able to work independently and collaborate with the development team to implement │ │ │ designs.\', \'requirements\': \'Proven UI/UX design experience, portfolio of design projects, │ │ │ knowledge of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design │ │ │ software like Adobe Illustrator and Photoshop, team spirit, strong communication skills, and │ │ │ problem-solving ability.\'}.\n How generic or specific is the job post is compared with │ │ │ other job posts in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat │ │ │ generic\n\n2: Neither generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn │ │ │ a valid JSON formatted like this, selecting only the number of the option:\n{"answer": <put answer │ │ │ code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │ │ │ 'MultipleChoice'}, 'specific_mc_system_prompt': {'text': "You are answering questions as if you │ │ │ were a human. Do not break character. You are an agent with the following │ │ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │ │ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │ │ │ 'class_name': 'AgentInstruction'}, 'specific_ls_user_prompt': {'text': 'You are being asked the │ │ │ following question: \n Consider the following job category at an online labor marketplace: │ │ │ Graphic Design.\n Consider the following job post: {\'title\': \'Remote UI/UX Designer\', │ │ │ \'description\': \'In search of a Remote UI/UX Designer to enhance user experience across our │ │ │ digital platforms. The ideal candidate will have experience in creating wireframes, storyboards, │ │ │ user flows, and process flows. The role involves designing graphic user interface elements, like │ │ │ menus, tabs, and widgets. Must be able to work independently and collaborate with the development │ │ │ team to implement designs.\', \'requirements\': \'Proven UI/UX design experience, portfolio of │ │ │ design projects, knowledge of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date │ │ │ knowledge of design software like Adobe Illustrator and Photoshop, team spirit, strong │ │ │ communication skills, and problem-solving ability.\'}.\n On a scale from 0 to 10, rate how │ │ │ specific the job post is compared with other job posts in the same category\n (0 = Very │ │ │ generic, 10 = Very specific).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: │ │ │ 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting │ │ │ only the code of the option (codes start at 0):\n{"answer": <put answer code here>, "comment": │ │ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │ │ │ 'specific_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │ │ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │ │ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │ │ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │ │ │ 'generic_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │ │ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │ │ │ following job post: {\'title\': \'Remote UI/UX Designer\', \'description\': \'In search of a Remote │ │ │ UI/UX Designer to enhance user experience across our digital platforms. The ideal candidate will │ │ │ have experience in creating wireframes, storyboards, user flows, and process flows. The role │ │ │ involves designing graphic user interface elements, like menus, tabs, and widgets. Must be able to │ │ │ work independently and collaborate with the development team to implement designs.\', │ │ │ \'requirements\': \'Proven UI/UX design experience, portfolio of design projects, knowledge of │ │ │ wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design software like │ │ │ Adobe Illustrator and Photoshop, team spirit, strong communication skills, and problem-solving │ │ │ ability.\'}.\n On a scale from 0 to 10, rate how generic the job post is compared to other │ │ │ job posts in the same category\n (0 = Very specific, 10 = Very generic).\nThe options │ │ │ are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: │ │ │ 10\n\nReturn a valid JSON formatted like this, selecting only the code of the option (codes start │ │ │ at 0):\n{"answer": <put answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may │ │ │ be selected.', 'class_name': 'LinearScale'}, 'generic_ls_system_prompt': {'text': "You are │ │ │ answering questions as if you were a human. Do not break character. You are an agent with the │ │ │ following persona:\n{'base_persona': 'You are an experienced freelancer on online labor │ │ │ marketplaces.', 'expertise': 'You regularly perform jobs in the following category: Graphic │ │ │ Design.'}", 'class_name': 'AgentInstruction'}} │ └───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Result 2
Result ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │ │ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │ │ │ Graphic Design.'}) │ │ scenario │ {'job_category': 'Graphic Design', 'job_post': '{\'title\': \'Senior Graphic Designer\', │ │ │ \'description\': \'Looking for a Senior Graphic Designer to lead our creative team. │ │ │ Responsibilities include conceptualizing and implementing design solutions that meet marketing │ │ │ strategies from concept to completion. This role requires leadership skills, as it involves guiding │ │ │ a team of designers, providing direction and feedback, while also working closely with │ │ │ cross-functional teams to deliver high-quality design assets.\', \'requirements\': "Bachelor\'s │ │ │ degree in graphic design or a related field, minimum of 5 years\' experience in a professional │ │ │ design environment, strong portfolio showcasing high-end digital design skills, mastery of Adobe │ │ │ Creative Suite, excellent leadership, communication, and organizational skills, ability to pitch │ │ │ creative concepts to clients and stakeholders."}'} │ │ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │ │ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │ │ iteration │ 2 │ │ answer │ {'specific_ls': '7', 'specific_ls_comment': "The job post is quite specific, detailing the level of │ │ │ experience required, specific educational background, proficiency in Adobe Creative Suite, and │ │ │ leadership qualities needed. It also outlines the role's responsibilities and the expectation of │ │ │ working with a team and stakeholders. However, it might lack details about the company culture, │ │ │ specific projects, or the design aesthetics preferred, which would make it a '10'.", 'generic_ls': │ │ │ '7', 'generic_ls_comment': 'The job post is somewhat generic as it includes common requirements and │ │ │ responsibilities for a Senior Graphic Designer role. Many job posts in this category will ask for a │ │ │ degree, experience, a strong portfolio, and proficiency in Adobe Creative Suite, as well as │ │ │ leadership and communication skills. However, it lacks specific details about the company, the │ │ │ particular projects the designer will be working on, and the unique value or culture of the team │ │ │ they will be leading.', 'specific_mc': 'Somewhat specific', 'specific_mc_comment': 'The job post is │ │ │ somewhat specific as it outlines a clear set of responsibilities, required qualifications, and the │ │ │ leadership role associated with the position. It goes beyond a basic description by specifying the │ │ │ need for a senior designer to lead a team and the requirement of a strong portfolio and mastery of │ │ │ Adobe Creative Suite, which are tailored to a specific level of expertise and experience.'} │ │ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │ │ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │ │ │ Consider the following job post: {\'title\': \'Senior Graphic Designer\', \'description\': │ │ │ \'Looking for a Senior Graphic Designer to lead our creative team. Responsibilities include │ │ │ conceptualizing and implementing design solutions that meet marketing strategies from concept to │ │ │ completion. This role requires leadership skills, as it involves guiding a team of designers, │ │ │ providing direction and feedback, while also working closely with cross-functional teams to deliver │ │ │ high-quality design assets.\', \'requirements\': "Bachelor\'s degree in graphic design or a related │ │ │ field, minimum of 5 years\' experience in a professional design environment, strong portfolio │ │ │ showcasing high-end digital design skills, mastery of Adobe Creative Suite, excellent leadership, │ │ │ communication, and organizational skills, ability to pitch creative concepts to clients and │ │ │ stakeholders."}.\n How generic or specific is the job post is compared with other job posts │ │ │ in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat generic\n\n2: Neither │ │ │ generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn a valid JSON formatted │ │ │ like this, selecting only the number of the option:\n{"answer": <put answer code here>, "comment": │ │ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'MultipleChoice'}, │ │ │ 'specific_mc_system_prompt': {'text': "You are answering questions as if you were a human. Do not │ │ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │ │ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │ │ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │ │ │ 'specific_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │ │ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │ │ │ following job post: {\'title\': \'Senior Graphic Designer\', \'description\': \'Looking for a │ │ │ Senior Graphic Designer to lead our creative team. Responsibilities include conceptualizing and │ │ │ implementing design solutions that meet marketing strategies from concept to completion. This role │ │ │ requires leadership skills, as it involves guiding a team of designers, providing direction and │ │ │ feedback, while also working closely with cross-functional teams to deliver high-quality design │ │ │ assets.\', \'requirements\': "Bachelor\'s degree in graphic design or a related field, minimum of 5 │ │ │ years\' experience in a professional design environment, strong portfolio showcasing high-end │ │ │ digital design skills, mastery of Adobe Creative Suite, excellent leadership, communication, and │ │ │ organizational skills, ability to pitch creative concepts to clients and stakeholders."}.\n │ │ │ On a scale from 0 to 10, rate how specific the job post is compared with other job posts in the │ │ │ same category\n (0 = Very generic, 10 = Very specific).\nThe options are\n\n0: 0\n\n1: │ │ │ 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON │ │ │ formatted like this, selecting only the code of the option (codes start at 0):\n{"answer": <put │ │ │ answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', │ │ │ 'class_name': 'LinearScale'}, 'specific_ls_system_prompt': {'text': "You are answering questions as │ │ │ if you were a human. Do not break character. You are an agent with the following │ │ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │ │ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │ │ │ 'class_name': 'AgentInstruction'}, 'generic_ls_user_prompt': {'text': 'You are being asked the │ │ │ following question: \n Consider the following job category at an online labor marketplace: │ │ │ Graphic Design.\n Consider the following job post: {\'title\': \'Senior Graphic Designer\', │ │ │ \'description\': \'Looking for a Senior Graphic Designer to lead our creative team. │ │ │ Responsibilities include conceptualizing and implementing design solutions that meet marketing │ │ │ strategies from concept to completion. This role requires leadership skills, as it involves guiding │ │ │ a team of designers, providing direction and feedback, while also working closely with │ │ │ cross-functional teams to deliver high-quality design assets.\', \'requirements\': "Bachelor\'s │ │ │ degree in graphic design or a related field, minimum of 5 years\' experience in a professional │ │ │ design environment, strong portfolio showcasing high-end digital design skills, mastery of Adobe │ │ │ Creative Suite, excellent leadership, communication, and organizational skills, ability to pitch │ │ │ creative concepts to clients and stakeholders."}.\n On a scale from 0 to 10, rate how │ │ │ generic the job post is compared to other job posts in the same category\n (0 = Very │ │ │ specific, 10 = Very generic).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: │ │ │ 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting │ │ │ only the code of the option (codes start at 0):\n{"answer": <put answer code here>, "comment": │ │ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │ │ │ 'generic_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │ │ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │ │ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │ │ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}} │ └───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Out[21]:
Result 0
Result
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │
│ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │
│ │ Graphic Design.'}) │
│ scenario │ {'job_category': 'Graphic Design', 'job_post': "{'title': 'Freelance Graphic Designer', │
│ │ 'description': 'Seeking a creative and detail-oriented Freelance Graphic Designer to create │
│ │ compelling visuals for digital and print media. Responsibilities include collaborating with clients │
│ │ to understand their vision, producing original graphics and layouts, and ensuring brand consistency │
│ │ across all designs. Proficiency in Adobe Creative Suite and a strong portfolio are required.', │
│ │ 'requirements': 'Proven graphic designing experience, a strong portfolio, familiarity with design │
│ │ software and technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen eye for │
│ │ aesthetics and details, excellent communication skills, ability to work methodically and meet │
│ │ deadlines.'}"} │
│ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │
│ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │
│ iteration │ 0 │
│ answer │ {'specific_ls': '4', 'specific_ls_comment': 'The job post provides a general outline of what is │
│ │ expected from a freelance graphic designer, such as creating visuals for digital and print media, │
│ │ proficiency in Adobe Creative Suite, and having a strong portfolio. However, it lacks specificity │
│ │ regarding the type of clients, project scope, design specialties, or particular industries or │
│ │ markets served. This is fairly typical for a broad graphic design role, making it somewhat generic │
│ │ compared to more specialized or detailed graphic design job postings.', 'generic_ls': '8', │
│ │ 'generic_ls_comment': 'The job post is quite generic as it includes common requirements and │
│ │ responsibilities that are typical for most graphic design job listings. It lacks specific │
│ │ information about the type of projects, the industry focus, or any unique company culture or │
│ │ project-related specifics that would differentiate it from other listings.', 'specific_mc': │
│ │ 'Somewhat generic', 'specific_mc_comment': 'The job post provided is somewhat generic as it lists │
│ │ common responsibilities and requirements that are typical for a freelance graphic designer role. │
│ │ Most graphic design job posts will ask for proficiency in Adobe Creative Suite, a strong portfolio, │
│ │ and good communication skills. However, it does not delve into specifics such as the type of │
│ │ clients, the industry focus, or unique design styles or techniques that might be needed, which │
│ │ would make it more specific.'} │
│ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │
│ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │
│ │ Consider the following job post: {\'title\': \'Freelance Graphic Designer\', \'description\': │
│ │ \'Seeking a creative and detail-oriented Freelance Graphic Designer to create compelling visuals │
│ │ for digital and print media. Responsibilities include collaborating with clients to understand │
│ │ their vision, producing original graphics and layouts, and ensuring brand consistency across all │
│ │ designs. Proficiency in Adobe Creative Suite and a strong portfolio are required.\', │
│ │ \'requirements\': \'Proven graphic designing experience, a strong portfolio, familiarity with │
│ │ design software and technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen │
│ │ eye for aesthetics and details, excellent communication skills, ability to work methodically and │
│ │ meet deadlines.\'}.\n How generic or specific is the job post is compared with other job │
│ │ posts in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat generic\n\n2: │
│ │ Neither generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn a valid JSON │
│ │ formatted like this, selecting only the number of the option:\n{"answer": <put answer code here>, │
│ │ "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │
│ │ 'MultipleChoice'}, 'specific_mc_system_prompt': {'text': "You are answering questions as if you │
│ │ were a human. Do not break character. You are an agent with the following │
│ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │
│ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │
│ │ 'class_name': 'AgentInstruction'}, 'specific_ls_user_prompt': {'text': 'You are being asked the │
│ │ following question: \n Consider the following job category at an online labor marketplace: │
│ │ Graphic Design.\n Consider the following job post: {\'title\': \'Freelance Graphic │
│ │ Designer\', \'description\': \'Seeking a creative and detail-oriented Freelance Graphic Designer to │
│ │ create compelling visuals for digital and print media. Responsibilities include collaborating with │
│ │ clients to understand their vision, producing original graphics and layouts, and ensuring brand │
│ │ consistency across all designs. Proficiency in Adobe Creative Suite and a strong portfolio are │
│ │ required.\', \'requirements\': \'Proven graphic designing experience, a strong portfolio, │
│ │ familiarity with design software and technologies (such as InDesign, Illustrator, Dreamweaver, │
│ │ Photoshop), a keen eye for aesthetics and details, excellent communication skills, ability to work │
│ │ methodically and meet deadlines.\'}.\n On a scale from 0 to 10, rate how specific the job │
│ │ post is compared with other job posts in the same category\n (0 = Very generic, 10 = Very │
│ │ specific).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: │
│ │ 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting only the code of the │
│ │ option (codes start at 0):\n{"answer": <put answer code here>, "comment": "<put explanation │
│ │ here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │
│ │ 'specific_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │
│ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │
│ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │
│ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │
│ │ 'generic_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │
│ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │
│ │ following job post: {\'title\': \'Freelance Graphic Designer\', \'description\': \'Seeking a │
│ │ creative and detail-oriented Freelance Graphic Designer to create compelling visuals for digital │
│ │ and print media. Responsibilities include collaborating with clients to understand their vision, │
│ │ producing original graphics and layouts, and ensuring brand consistency across all designs. │
│ │ Proficiency in Adobe Creative Suite and a strong portfolio are required.\', \'requirements\': │
│ │ \'Proven graphic designing experience, a strong portfolio, familiarity with design software and │
│ │ technologies (such as InDesign, Illustrator, Dreamweaver, Photoshop), a keen eye for aesthetics and │
│ │ details, excellent communication skills, ability to work methodically and meet deadlines.\'}.\n │
│ │ On a scale from 0 to 10, rate how generic the job post is compared to other job posts in the same │
│ │ category\n (0 = Very specific, 10 = Very generic).\nThe options are\n\n0: 0\n\n1: 1\n\n2: │
│ │ 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON │
│ │ formatted like this, selecting only the code of the option (codes start at 0):\n{"answer": <put │
│ │ answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', │
│ │ 'class_name': 'LinearScale'}, 'generic_ls_system_prompt': {'text': "You are answering questions as │
│ │ if you were a human. Do not break character. You are an agent with the following │
│ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │
│ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │
│ │ 'class_name': 'AgentInstruction'}} │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Result 1
Result
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │
│ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │
│ │ Graphic Design.'}) │
│ scenario │ {'job_category': 'Graphic Design', 'job_post': "{'title': 'Remote UI/UX Designer', 'description': │
│ │ 'In search of a Remote UI/UX Designer to enhance user experience across our digital platforms. The │
│ │ ideal candidate will have experience in creating wireframes, storyboards, user flows, and process │
│ │ flows. The role involves designing graphic user interface elements, like menus, tabs, and widgets. │
│ │ Must be able to work independently and collaborate with the development team to implement │
│ │ designs.', 'requirements': 'Proven UI/UX design experience, portfolio of design projects, knowledge │
│ │ of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design software like │
│ │ Adobe Illustrator and Photoshop, team spirit, strong communication skills, and problem-solving │
│ │ ability.'}"} │
│ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │
│ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │
│ iteration │ 1 │
│ answer │ {'specific_ls': '7', 'specific_ls_comment': 'The job post is quite specific as it outlines the need │
│ │ for a UI/UX designer with experience in creating wireframes, storyboards, user flows, and process │
│ │ flows. It also specifies the requirement for knowledge in particular wireframe tools and design │
│ │ software, which is more detailed than a generic job description. However, it does not mention │
│ │ specific project types or industries, which could make it even more specific.', 'generic_ls': '7', │
│ │ 'generic_ls_comment': "The job post for a 'Remote UI/UX Designer' is somewhat generic, as it lists │
│ │ common responsibilities and requirements that are typical for such positions. Many UI/UX design job │
│ │ postings seek candidates with experience in wireframing, storyboarding, and creating user flows, as │
│ │ well as proficiency in tools like Adobe Illustrator and Photoshop. The mention of needing a team │
│ │ player with strong communication skills is also a standard expectation. However, it does not │
│ │ provide specific details about the company, the projects, or unique challenges that might set it │
│ │ apart from other similar listings.", 'specific_mc': 'Neither generic nor specific', │
│ │ 'specific_mc_comment': "The job post is neither highly generic nor highly specific. It outlines │
│ │ common expectations for a UI/UX designer role such as creating wireframes, storyboards, user flows, │
│ │ and designing graphic user interface elements. The requirements listed are also typical for this │
│ │ type of position, including experience, a portfolio, knowledge of wireframe tools, and design │
│ │ software skills. However, it does not provide highly specific details such as the company's │
│ │ industry, the exact nature of the digital platforms, or the particular design aesthetics they are │
│ │ looking for, which would make it more specific."} │
│ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │
│ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │
│ │ Consider the following job post: {\'title\': \'Remote UI/UX Designer\', \'description\': \'In │
│ │ search of a Remote UI/UX Designer to enhance user experience across our digital platforms. The │
│ │ ideal candidate will have experience in creating wireframes, storyboards, user flows, and process │
│ │ flows. The role involves designing graphic user interface elements, like menus, tabs, and widgets. │
│ │ Must be able to work independently and collaborate with the development team to implement │
│ │ designs.\', \'requirements\': \'Proven UI/UX design experience, portfolio of design projects, │
│ │ knowledge of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design │
│ │ software like Adobe Illustrator and Photoshop, team spirit, strong communication skills, and │
│ │ problem-solving ability.\'}.\n How generic or specific is the job post is compared with │
│ │ other job posts in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat │
│ │ generic\n\n2: Neither generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn │
│ │ a valid JSON formatted like this, selecting only the number of the option:\n{"answer": <put answer │
│ │ code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │
│ │ 'MultipleChoice'}, 'specific_mc_system_prompt': {'text': "You are answering questions as if you │
│ │ were a human. Do not break character. You are an agent with the following │
│ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │
│ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │
│ │ 'class_name': 'AgentInstruction'}, 'specific_ls_user_prompt': {'text': 'You are being asked the │
│ │ following question: \n Consider the following job category at an online labor marketplace: │
│ │ Graphic Design.\n Consider the following job post: {\'title\': \'Remote UI/UX Designer\', │
│ │ \'description\': \'In search of a Remote UI/UX Designer to enhance user experience across our │
│ │ digital platforms. The ideal candidate will have experience in creating wireframes, storyboards, │
│ │ user flows, and process flows. The role involves designing graphic user interface elements, like │
│ │ menus, tabs, and widgets. Must be able to work independently and collaborate with the development │
│ │ team to implement designs.\', \'requirements\': \'Proven UI/UX design experience, portfolio of │
│ │ design projects, knowledge of wireframe tools (e.g. Wireframe.cc and InVision), up-to-date │
│ │ knowledge of design software like Adobe Illustrator and Photoshop, team spirit, strong │
│ │ communication skills, and problem-solving ability.\'}.\n On a scale from 0 to 10, rate how │
│ │ specific the job post is compared with other job posts in the same category\n (0 = Very │
│ │ generic, 10 = Very specific).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: │
│ │ 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting │
│ │ only the code of the option (codes start at 0):\n{"answer": <put answer code here>, "comment": │
│ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │
│ │ 'specific_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │
│ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │
│ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │
│ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │
│ │ 'generic_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │
│ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │
│ │ following job post: {\'title\': \'Remote UI/UX Designer\', \'description\': \'In search of a Remote │
│ │ UI/UX Designer to enhance user experience across our digital platforms. The ideal candidate will │
│ │ have experience in creating wireframes, storyboards, user flows, and process flows. The role │
│ │ involves designing graphic user interface elements, like menus, tabs, and widgets. Must be able to │
│ │ work independently and collaborate with the development team to implement designs.\', │
│ │ \'requirements\': \'Proven UI/UX design experience, portfolio of design projects, knowledge of │
│ │ wireframe tools (e.g. Wireframe.cc and InVision), up-to-date knowledge of design software like │
│ │ Adobe Illustrator and Photoshop, team spirit, strong communication skills, and problem-solving │
│ │ ability.\'}.\n On a scale from 0 to 10, rate how generic the job post is compared to other │
│ │ job posts in the same category\n (0 = Very specific, 10 = Very generic).\nThe options │
│ │ are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: │
│ │ 10\n\nReturn a valid JSON formatted like this, selecting only the code of the option (codes start │
│ │ at 0):\n{"answer": <put answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may │
│ │ be selected.', 'class_name': 'LinearScale'}, 'generic_ls_system_prompt': {'text': "You are │
│ │ answering questions as if you were a human. Do not break character. You are an agent with the │
│ │ following persona:\n{'base_persona': 'You are an experienced freelancer on online labor │
│ │ marketplaces.', 'expertise': 'You regularly perform jobs in the following category: Graphic │
│ │ Design.'}", 'class_name': 'AgentInstruction'}} │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Result 2
Result
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent(name = 'Graphic Design', traits = {'base_persona': 'You are an experienced freelancer on │
│ │ online labor marketplaces.', 'expertise': 'You regularly perform jobs in the following category: │
│ │ Graphic Design.'}) │
│ scenario │ {'job_category': 'Graphic Design', 'job_post': '{\'title\': \'Senior Graphic Designer\', │
│ │ \'description\': \'Looking for a Senior Graphic Designer to lead our creative team. │
│ │ Responsibilities include conceptualizing and implementing design solutions that meet marketing │
│ │ strategies from concept to completion. This role requires leadership skills, as it involves guiding │
│ │ a team of designers, providing direction and feedback, while also working closely with │
│ │ cross-functional teams to deliver high-quality design assets.\', \'requirements\': "Bachelor\'s │
│ │ degree in graphic design or a related field, minimum of 5 years\' experience in a professional │
│ │ design environment, strong portfolio showcasing high-end digital design skills, mastery of Adobe │
│ │ Creative Suite, excellent leadership, communication, and organizational skills, ability to pitch │
│ │ creative concepts to clients and stakeholders."}'} │
│ model │ LanguageModelOpenAIFour(model = 'gpt-4-1106-preview', parameters={'temperature': 0.5, 'max_tokens': │
│ │ 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │
│ iteration │ 2 │
│ answer │ {'specific_ls': '7', 'specific_ls_comment': "The job post is quite specific, detailing the level of │
│ │ experience required, specific educational background, proficiency in Adobe Creative Suite, and │
│ │ leadership qualities needed. It also outlines the role's responsibilities and the expectation of │
│ │ working with a team and stakeholders. However, it might lack details about the company culture, │
│ │ specific projects, or the design aesthetics preferred, which would make it a '10'.", 'generic_ls': │
│ │ '7', 'generic_ls_comment': 'The job post is somewhat generic as it includes common requirements and │
│ │ responsibilities for a Senior Graphic Designer role. Many job posts in this category will ask for a │
│ │ degree, experience, a strong portfolio, and proficiency in Adobe Creative Suite, as well as │
│ │ leadership and communication skills. However, it lacks specific details about the company, the │
│ │ particular projects the designer will be working on, and the unique value or culture of the team │
│ │ they will be leading.', 'specific_mc': 'Somewhat specific', 'specific_mc_comment': 'The job post is │
│ │ somewhat specific as it outlines a clear set of responsibilities, required qualifications, and the │
│ │ leadership role associated with the position. It goes beyond a basic description by specifying the │
│ │ need for a senior designer to lead a team and the requirement of a strong portfolio and mastery of │
│ │ Adobe Creative Suite, which are tailored to a specific level of expertise and experience.'} │
│ prompt │ {'specific_mc_user_prompt': {'text': 'You are being asked the following question: \n │
│ │ Consider the following job category at an online labor marketplace: Graphic Design.\n │
│ │ Consider the following job post: {\'title\': \'Senior Graphic Designer\', \'description\': │
│ │ \'Looking for a Senior Graphic Designer to lead our creative team. Responsibilities include │
│ │ conceptualizing and implementing design solutions that meet marketing strategies from concept to │
│ │ completion. This role requires leadership skills, as it involves guiding a team of designers, │
│ │ providing direction and feedback, while also working closely with cross-functional teams to deliver │
│ │ high-quality design assets.\', \'requirements\': "Bachelor\'s degree in graphic design or a related │
│ │ field, minimum of 5 years\' experience in a professional design environment, strong portfolio │
│ │ showcasing high-end digital design skills, mastery of Adobe Creative Suite, excellent leadership, │
│ │ communication, and organizational skills, ability to pitch creative concepts to clients and │
│ │ stakeholders."}.\n How generic or specific is the job post is compared with other job posts │
│ │ in the same category?\nThe options are\n\n0: Highly generic\n\n1: Somewhat generic\n\n2: Neither │
│ │ generic nor specific\n\n3: Somewhat specific\n\n4: Highly specific\n\nReturn a valid JSON formatted │
│ │ like this, selecting only the number of the option:\n{"answer": <put answer code here>, "comment": │
│ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'MultipleChoice'}, │
│ │ 'specific_mc_system_prompt': {'text': "You are answering questions as if you were a human. Do not │
│ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │
│ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │
│ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}, │
│ │ 'specific_ls_user_prompt': {'text': 'You are being asked the following question: \n Consider │
│ │ the following job category at an online labor marketplace: Graphic Design.\n Consider the │
│ │ following job post: {\'title\': \'Senior Graphic Designer\', \'description\': \'Looking for a │
│ │ Senior Graphic Designer to lead our creative team. Responsibilities include conceptualizing and │
│ │ implementing design solutions that meet marketing strategies from concept to completion. This role │
│ │ requires leadership skills, as it involves guiding a team of designers, providing direction and │
│ │ feedback, while also working closely with cross-functional teams to deliver high-quality design │
│ │ assets.\', \'requirements\': "Bachelor\'s degree in graphic design or a related field, minimum of 5 │
│ │ years\' experience in a professional design environment, strong portfolio showcasing high-end │
│ │ digital design skills, mastery of Adobe Creative Suite, excellent leadership, communication, and │
│ │ organizational skills, ability to pitch creative concepts to clients and stakeholders."}.\n │
│ │ On a scale from 0 to 10, rate how specific the job post is compared with other job posts in the │
│ │ same category\n (0 = Very generic, 10 = Very specific).\nThe options are\n\n0: 0\n\n1: │
│ │ 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON │
│ │ formatted like this, selecting only the code of the option (codes start at 0):\n{"answer": <put │
│ │ answer code here>, "comment": "<put explanation here>"}\nOnly 1 option may be selected.', │
│ │ 'class_name': 'LinearScale'}, 'specific_ls_system_prompt': {'text': "You are answering questions as │
│ │ if you were a human. Do not break character. You are an agent with the following │
│ │ persona:\n{'base_persona': 'You are an experienced freelancer on online labor marketplaces.', │
│ │ 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}", │
│ │ 'class_name': 'AgentInstruction'}, 'generic_ls_user_prompt': {'text': 'You are being asked the │
│ │ following question: \n Consider the following job category at an online labor marketplace: │
│ │ Graphic Design.\n Consider the following job post: {\'title\': \'Senior Graphic Designer\', │
│ │ \'description\': \'Looking for a Senior Graphic Designer to lead our creative team. │
│ │ Responsibilities include conceptualizing and implementing design solutions that meet marketing │
│ │ strategies from concept to completion. This role requires leadership skills, as it involves guiding │
│ │ a team of designers, providing direction and feedback, while also working closely with │
│ │ cross-functional teams to deliver high-quality design assets.\', \'requirements\': "Bachelor\'s │
│ │ degree in graphic design or a related field, minimum of 5 years\' experience in a professional │
│ │ design environment, strong portfolio showcasing high-end digital design skills, mastery of Adobe │
│ │ Creative Suite, excellent leadership, communication, and organizational skills, ability to pitch │
│ │ creative concepts to clients and stakeholders."}.\n On a scale from 0 to 10, rate how │
│ │ generic the job post is compared to other job posts in the same category\n (0 = Very │
│ │ specific, 10 = Very generic).\nThe options are\n\n0: 0\n\n1: 1\n\n2: 2\n\n3: 3\n\n4: 4\n\n5: │
│ │ 5\n\n6: 6\n\n7: 7\n\n8: 8\n\n9: 9\n\n10: 10\n\nReturn a valid JSON formatted like this, selecting │
│ │ only the code of the option (codes start at 0):\n{"answer": <put answer code here>, "comment": │
│ │ "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': 'LinearScale'}, │
│ │ 'generic_ls_system_prompt': {'text': "You are answering questions as if you were a human. Do not │
│ │ break character. You are an agent with the following persona:\n{'base_persona': 'You are an │
│ │ experienced freelancer on online labor marketplaces.', 'expertise': 'You regularly perform jobs in │
│ │ the following category: Graphic Design.'}", 'class_name': 'AgentInstruction'}} │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
We can identify the column names to select the fields that we want to inspect:
In [22]:
results["Graphic Design"].to_pandas().columns
Out[22]:
Index(['agent.agent_name', 'agent.base_persona', 'agent.expertise',
'answer.generic_ls', 'answer.generic_ls_comment', 'answer.specific_ls',
'answer.specific_ls_comment', 'answer.specific_mc',
'answer.specific_mc_comment', 'model.frequency_penalty',
'model.max_tokens', 'model.model', 'model.presence_penalty',
'model.temperature', 'model.top_p', 'model.use_cache',
'prompt.generic_ls_system_prompt', 'prompt.generic_ls_user_prompt',
'prompt.specific_ls_system_prompt', 'prompt.specific_ls_user_prompt',
'prompt.specific_mc_system_prompt', 'prompt.specific_mc_user_prompt',
'scenario.job_category', 'scenario.job_post'],
dtype='object')
We can select individual fields in a variety of ways:
In [23]:
(results["Graphic Design"]
.select("job_post", "specific_ls", "generic_ls", "specific_mc")
.print()
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ answer ┃ answer ┃ answer ┃ ┃ .job_post ┃ .specific_ls ┃ .generic_ls ┃ .specific_mc ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ {'title': 'Freelance Graphic Designer', │ 4 │ 8 │ Somewhat generic │ │ 'description': 'Seeking a creative and │ │ │ │ │ detail-oriented Freelance Graphic Designer to │ │ │ │ │ create compelling visuals for digital and print │ │ │ │ │ media. Responsibilities include collaborating with │ │ │ │ │ clients to understand their vision, producing │ │ │ │ │ original graphics and layouts, and ensuring brand │ │ │ │ │ consistency across all designs. Proficiency in │ │ │ │ │ Adobe Creative Suite and a strong portfolio are │ │ │ │ │ required.', 'requirements': 'Proven graphic │ │ │ │ │ designing experience, a strong portfolio, │ │ │ │ │ familiarity with design software and technologies │ │ │ │ │ (such as InDesign, Illustrator, Dreamweaver, │ │ │ │ │ Photoshop), a keen eye for aesthetics and details, │ │ │ │ │ excellent communication skills, ability to work │ │ │ │ │ methodically and meet deadlines.'} │ │ │ │ ├─────────────────────────────────────────────────────┼──────────────┼─────────────┼──────────────────────────────┤ │ {'title': 'Remote UI/UX Designer', 'description': │ 7 │ 7 │ Neither generic nor specific │ │ 'In search of a Remote UI/UX Designer to enhance │ │ │ │ │ user experience across our digital platforms. The │ │ │ │ │ ideal candidate will have experience in creating │ │ │ │ │ wireframes, storyboards, user flows, and process │ │ │ │ │ flows. The role involves designing graphic user │ │ │ │ │ interface elements, like menus, tabs, and widgets. │ │ │ │ │ Must be able to work independently and collaborate │ │ │ │ │ with the development team to implement designs.', │ │ │ │ │ 'requirements': 'Proven UI/UX design experience, │ │ │ │ │ portfolio of design projects, knowledge of │ │ │ │ │ wireframe tools (e.g. Wireframe.cc and InVision), │ │ │ │ │ up-to-date knowledge of design software like Adobe │ │ │ │ │ Illustrator and Photoshop, team spirit, strong │ │ │ │ │ communication skills, and problem-solving │ │ │ │ │ ability.'} │ │ │ │ ├─────────────────────────────────────────────────────┼──────────────┼─────────────┼──────────────────────────────┤ │ {'title': 'Senior Graphic Designer', 'description': │ 7 │ 7 │ Somewhat specific │ │ 'Looking for a Senior Graphic Designer to lead our │ │ │ │ │ creative team. Responsibilities include │ │ │ │ │ conceptualizing and implementing design solutions │ │ │ │ │ that meet marketing strategies from concept to │ │ │ │ │ completion. This role requires leadership skills, │ │ │ │ │ as it involves guiding a team of designers, │ │ │ │ │ providing direction and feedback, while also │ │ │ │ │ working closely with cross-functional teams to │ │ │ │ │ deliver high-quality design assets.', │ │ │ │ │ 'requirements': "Bachelor's degree in graphic │ │ │ │ │ design or a related field, minimum of 5 years' │ │ │ │ │ experience in a professional design environment, │ │ │ │ │ strong portfolio showcasing high-end digital design │ │ │ │ │ skills, mastery of Adobe Creative Suite, excellent │ │ │ │ │ leadership, communication, and organizational │ │ │ │ │ skills, ability to pitch creative concepts to │ │ │ │ │ clients and stakeholders."} │ │ │ │ └─────────────────────────────────────────────────────┴──────────────┴─────────────┴──────────────────────────────┘
We can apply some labels to our table for readability. Note that each question field also automatically includes a <question>_comment field for any commentary by the LLM on the question:
In [24]:
(results["Web Development"]
.select("job_post", "specific_mc", "specific_mc_comment")
.print(pretty_labels = {
"scenario.job_post":"Job post description",
"answer.specific_mc":"How generic or specific? (Multiple choice)",
"answer.specific_mc_comment":"Comment"})
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ How generic or specific? (Multiple ┃ ┃ ┃ Job post description ┃ choice) ┃ Comment ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ {'jobTitle': 'Front-End Developer', │ Somewhat generic │ The job post provided falls into │ │ 'jobDescription': 'Seeking a │ │ the 'Somewhat generic' category. It │ │ creative Front-End Developer with │ │ covers the basic requirements and │ │ expertise in HTML, CSS, JavaScript, │ │ responsibilities one would expect │ │ and frameworks like React or Vue.js │ │ from a Front-End Developer role, │ │ to build and optimize user │ │ such as proficiency in HTML, CSS, │ │ interfaces for our web │ │ JavaScript, and popular frameworks │ │ applications. Responsibilities │ │ like React or Vue.js, as well as │ │ include translating design │ │ experience with responsive design │ │ wireframes into code, optimizing │ │ and optimization for speed and │ │ web pages for maximum speed and │ │ scalability. These are common │ │ scalability, and ensuring a │ │ expectations for such positions. │ │ seamless user experience across all │ │ However, it does not delve into │ │ devices and platforms.', │ │ project-specific technologies or │ │ 'qualifications': 'Proven │ │ business domains that might make it │ │ experience as a Front-End │ │ highly specific. It also doesn't │ │ Developer, strong understanding of │ │ stay at a high level of abstraction │ │ web markup and front-end coding │ │ that would render it highly │ │ languages, experience with │ │ generic. Therefore, it strikes a │ │ responsive and adaptive design, │ │ balance by providing enough detail │ │ good problem-solving skills, and │ │ to filter candidates with relevant │ │ familiarity with software like │ │ skills without being tailored to a │ │ Adobe Suite, Photoshop, and content │ │ niche requirement. │ │ management systems.', │ │ │ │ 'responsibilities': 'Develop new │ │ │ │ user-facing features, build │ │ │ │ reusable code and libraries for │ │ │ │ future use, ensure the technical │ │ │ │ feasibility of UI/UX designs, │ │ │ │ optimize application for maximum │ │ │ │ speed and scalability, collaborate │ │ │ │ with other team members and │ │ │ │ stakeholders.'} │ │ │ ├─────────────────────────────────────┼─────────────────────────────────────┼─────────────────────────────────────┤ │ {'jobTitle': 'Back-End Developer', │ Somewhat generic │ The job post is somewhat generic as │ │ 'jobDescription': 'We are looking │ │ it lists common responsibilities │ │ for a Back-End Developer to join │ │ and qualifications for a back-end │ │ our team to develop server-side │ │ developer role. It does not mention │ │ logic, definition and maintenance │ │ specific frameworks or technologies │ │ of the central database, and ensure │ │ other than programming languages, │ │ high performance and responsiveness │ │ and it does not provide details │ │ to requests from the front-end. You │ │ about the company, the project, or │ │ will also be responsible for │ │ unique requirements that might set │ │ integrating the front-end elements │ │ it apart from other similar job │ │ built by your coworkers into the │ │ postings. │ │ application.', 'qualifications': │ │ │ │ 'Strong knowledge of back-end │ │ │ │ programming languages like Java, │ │ │ │ Python, Ruby, or .NET, │ │ │ │ understanding of accessibility and │ │ │ │ security compliance, user │ │ │ │ authentication and authorization │ │ │ │ between multiple systems, servers, │ │ │ │ and environments, integration of │ │ │ │ multiple data sources and databases │ │ │ │ into one system, familiarity with │ │ │ │ development aiding tools.', │ │ │ │ 'responsibilities': 'Integration of │ │ │ │ user-facing elements developed by │ │ │ │ front-end developers with │ │ │ │ server-side logic, writing │ │ │ │ reusable, testable, and efficient │ │ │ │ code, design and implementation of │ │ │ │ low-latency, high-availability, and │ │ │ │ performant applications, │ │ │ │ implementation of security and data │ │ │ │ protection.'} │ │ │ ├─────────────────────────────────────┼─────────────────────────────────────┼─────────────────────────────────────┤ │ {'jobTitle': 'Full Stack │ Somewhat generic │ The job post is somewhat generic as │ │ Developer', 'jobDescription': │ │ it covers a broad range of │ │ 'Looking for a Full Stack Developer │ │ responsibilities and skills that │ │ to produce scalable software │ │ are commonly expected of a Full │ │ solutions. You’ll be part of a │ │ Stack Developer. It does not delve │ │ cross-functional team that’s │ │ into specific project details, │ │ responsible for the full software │ │ technologies beyond the most common │ │ development life cycle, from │ │ ones, or unique company practices │ │ conception to deployment. You │ │ that would make it highly specific. │ │ should be comfortable around both │ │ However, it's not highly generic │ │ front-end and back-end coding │ │ because it does list specific │ │ languages, development frameworks, │ │ languages and frameworks, as well │ │ and third-party libraries.', │ │ as a clear description of the job │ │ 'qualifications': 'Experience │ │ role. │ │ developing desktop and mobile │ │ │ │ applications, familiarity with │ │ │ │ common stacks, knowledge of │ │ │ │ multiple front-end languages and │ │ │ │ libraries (e.g. HTML/ CSS, │ │ │ │ JavaScript, XML, jQuery), knowledge │ │ │ │ of multiple back-end languages │ │ │ │ (e.g. C#, Java, Python) and │ │ │ │ JavaScript frameworks (e.g. │ │ │ │ Angular, React, Node.js).', │ │ │ │ 'responsibilities': 'Work with │ │ │ │ development teams and product │ │ │ │ managers to ideate software │ │ │ │ solutions, design client-side and │ │ │ │ server-side architecture, build the │ │ │ │ front-end of applications through │ │ │ │ appealing visual design, develop │ │ │ │ and manage well-functioning │ │ │ │ databases and applications, write │ │ │ │ effective APIs.'} │ │ │ └─────────────────────────────────────┴─────────────────────────────────────┴─────────────────────────────────────┘
We can also access results as a SQL table (called self) with the .sql() method, choosing between a "wide" horizontal view of all fields and a "long" vertical view, and optionally removing the column name prefixes 'agent', 'model', 'prompt', etc.:
In [26]:
results["Graphic Design"].sql("select * from self", shape="long")
Out[26]:
| id | data_type | key | value | |
|---|---|---|---|---|
| 0 | 0 | agent | base_persona | You are an experienced freelancer on online la... |
| 1 | 0 | agent | expertise | You regularly perform jobs in the following ca... |
| 2 | 0 | agent | agent_name | Graphic Design |
| 3 | 0 | scenario | job_category | Graphic Design |
| 4 | 0 | scenario | job_post | {'title': 'Freelance Graphic Designer', 'descr... |
| ... | ... | ... | ... | ... |
| 67 | 2 | prompt | specific_mc_system_prompt | {'text': "You are answering questions as if yo... |
| 68 | 2 | prompt | specific_ls_user_prompt | {'text': 'You are being asked the following qu... |
| 69 | 2 | prompt | specific_ls_system_prompt | {'text': "You are answering questions as if yo... |
| 70 | 2 | prompt | generic_ls_user_prompt | {'text': 'You are being asked the following qu... |
| 71 | 2 | prompt | generic_ls_system_prompt | {'text': "You are answering questions as if yo... |
72 rows × 4 columns
In [27]:
results["Graphic Design"].sql("select agent_name, job_post, specific_mc, specific_mc_comment from self", shape="wide", remove_prefix=True)
Out[27]:
| agent_name | job_post | specific_mc | specific_mc_comment | |
|---|---|---|---|---|
| 0 | Graphic Design | {'title': 'Freelance Graphic Designer', 'descr... | Somewhat generic | The job post provided is somewhat generic as i... |
| 1 | Graphic Design | {'title': 'Remote UI/UX Designer', 'descriptio... | Neither generic nor specific | The job post is neither highly generic nor hig... |
| 2 | Graphic Design | {'title': 'Senior Graphic Designer', 'descript... | Somewhat specific | The job post is somewhat specific as it outlin... |
In [28]:
results["Graphic Design"].word_cloud_plot("specific_mc_comment")
In [29]:
results["Graphic Design"].bar_chart("specific_ls")
Consider the following job category at an online labor marketplace: {{ job_category }}.
Consider the following job post: {{ job_post }}.
On a scale from 0 to 10, rate how specific the job post is compared with other job posts in the same category
(0 = Very generic, 10 = Very specific).
