|
How to use edsl: This notebook shows how to use Expected Parrot open source tools for conducting social science and market research with AI simulations. >> Get the latest version on PyPI << |
|
Introduction¶
This notebook is a demonstration of the Expected Parrot Domain-Specific Language, or edsl. The "domain" is the construction and administration of surveys to intelligences, both human and artificial. Surveys by AIs? Yes: there is growing body of evidence that we can use AIs as stand-ins for some kinds of empirical research:
Large Language Models as Simulated Economic Agents
Using GPT for Market Research
Out of one, many: Using Language Models to Simulate Human Subjects
Using Large Language Models to Simulate Multiple Humans
Why a domain-specific language?
Technical reasons
One of the challenges in working with AIs is that they often require customized, one-off programming. In using AIs to simulate surveys and experiments, there are several specific tasks:
Creating questions and surveys
Creating contexts that agents will consider when answering questions
Creating agents with relevant traits
Administering surveys to AIs with different models, parameters and subtle variations in prompts
Despite the commonality of these tasks, each experiment can easily have enough differences that some monolithic run_experiment() function would not work well, hence our creation of a DSL. Edsl also happens to be an embedded domain-specific language, as it is embedded in Python. This has the benefit that everything in edsl can fit within the larger Python ecosystem.
Community reasons
Although AI experiments and simulations hold immense potential, they can also easily be misused. Language models will readily make things up, and there are questions for which we have no good reason to think they can offer us anything. We think the best solution to this problem is letting thousands of flowers bloom and putting these tools in the hands of end users. If people share their code using standardized, open format, we hope that:
Results from these experiments will be easily replicable
Best practices will emerge and spread
People will suggest and add cool features to edsl itself
Key features & capabilities
Ability to create commonly used question types (multiple choice, checkbox, free text, linear scale, etc.) with extensive validation checks.
Ability to combine questions into surveys and experiments with arbitarily complex skip-logic.
Ability to run surveys against different agent backgrounds and LLMs, with different settings and parameters.
Ability to chain questions where the input to one depends on another, in a fluent interface.
Tools for data visualization and analysis.
Local caching of API calls in a database, to reduce costs and facilitate creation.
Use cases
Simulated surveys and experiments provide scaleable solutions in a wide range of applications:
Scalable qualitative research, such as data-labeling and content reviews
Generating feedback on content and ideas
Exploring outcomes and effects of a business decision, such as a price change
In-depth analysis of interactions between agent persona traits and responses
Benefits of AI simulations
AI simulations are a powerful new tool for learning, experimentation, problem-solving and decision-making. Some of the benefits include:
Cost & time savings
Consistency & completeness in survey results
Enhanced data utilization & analytics
Data privacy & confidentiality
Open source license
Edsl is open source and distributed with the MIT License by Expected Parrot (formerly Emeritus), a startup building tools for the future of work. Expected Parrot offers hosting, customization, access to super agent sets, tools for incorporating general population datasets and special capabilities for certain use cases. Get in touch to learn more.
Installing edsl¶
Go to PyPI and install or update to the latest version of edsl:
https://pypi.org/project/edsl/
Requirements:
Python 3.9 - 3.11
[For now] API key for OpenAI, Google or DeepInfra — Stay tuned, the edsl API is coming soon!
To install:
$ pip install edsl
To update:
$ pip install --upgrade edsl
To check your version:
$ pip show edsl
The first time you access edsl you will be prompted to (optionally) enter API keys for OpenAI, Google and DeepInfra. This step can be skipped by pressing enter at each prompt. You do not need an API key to construct surveys in edsl; however, an API key is required in order to simulate responses using an LLM.
Note: The edsl API is coming soon and will allow you to access all available LLMs with a single edsl key.
The prompt for API keys looks like this:
Available LLMs¶
LLMs currently available using edsl:
from edsl import Model
Model.available()
['gpt-3.5-turbo', 'gpt-4-1106-preview', 'gemini_pro', 'llama-2-13b-chat-hf', 'llama-2-70b-chat-hf', 'mixtral-8x7B-instruct-v0.1']
A quick example¶
Edsl allows us to get started right away with a simple question. Here we create a multiple choice question and simulate a response to it:
Do you enjoy snow shoveling?
from edsl.questions import QuestionMultipleChoice
q1 = QuestionMultipleChoice(
question_name = "snow_shoveling",
question_text = "Do you enjoy snow shoveling?",
question_options = ["Yes", "No", "I do not know"],
)
We use the run() method to send the question to the default LLM (GPT 4) and inspect the full result (we'll narrow down the columns in next steps):
result = q1.run()
result.print()
┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓ ┃ answer ┃ answer ┃ agent ┃ model ┃ model ┃ model ┃ model ┃ model ┃ model ┃ model ┃ prompt ┃ prompt ┃ ┃ .snow_… ┃ .snow… ┃ .agent… ┃ .model ┃ .max_t… ┃ .use_… ┃ .prese… ┃ .temp… ┃ .top_p ┃ .frequ… ┃ .snow… ┃ .snow_… ┃ ┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩ │ No │ No, I │ Agent_0 │ gpt-3… │ 1000 │ True │ 0 │ 0.5 │ 1 │ 0 │ {'tex… │ {'text… │ │ │ do not │ │ │ │ │ │ │ │ │ 'You │ 'You │ │ │ enjoy │ │ │ │ │ │ │ │ │ are │ are │ │ │ snow │ │ │ │ │ │ │ │ │ being │ answer… │ │ │ shove… │ │ │ │ │ │ │ │ │ asked │ questi… │ │ │ It can │ │ │ │ │ │ │ │ │ the │ as if │ │ │ be │ │ │ │ │ │ │ │ │ follo… │ you │ │ │ tiring │ │ │ │ │ │ │ │ │ quest… │ were a │ │ │ and │ │ │ │ │ │ │ │ │ Do you │ human. │ │ │ time-… │ │ │ │ │ │ │ │ │ enjoy │ Do not │ │ │ │ │ │ │ │ │ │ │ │ snow │ break │ │ │ │ │ │ │ │ │ │ │ │ shove… │ charac… │ │ │ │ │ │ │ │ │ │ │ │ optio… │ You are │ │ │ │ │ │ │ │ │ │ │ │ are\n… │ an │ │ │ │ │ │ │ │ │ │ │ │ Yes\n… │ agent │ │ │ │ │ │ │ │ │ │ │ │ No\n\… │ with │ │ │ │ │ │ │ │ │ │ │ │ I do │ the │ │ │ │ │ │ │ │ │ │ │ │ not │ follow… │ │ │ │ │ │ │ │ │ │ │ │ know\… │ person… │ │ │ │ │ │ │ │ │ │ │ │ a │ 'class… │ │ │ │ │ │ │ │ │ │ │ │ valid │ 'Agent… │ │ │ │ │ │ │ │ │ │ │ │ JSON │ │ │ │ │ │ │ │ │ │ │ │ │ forma… │ │ │ │ │ │ │ │ │ │ │ │ │ like │ │ │ │ │ │ │ │ │ │ │ │ │ this, │ │ │ │ │ │ │ │ │ │ │ │ │ selec… │ │ │ │ │ │ │ │ │ │ │ │ │ only │ │ │ │ │ │ │ │ │ │ │ │ │ the │ │ │ │ │ │ │ │ │ │ │ │ │ number │ │ │ │ │ │ │ │ │ │ │ │ │ of the │ │ │ │ │ │ │ │ │ │ │ │ │ optio… │ │ │ │ │ │ │ │ │ │ │ │ │ <put │ │ │ │ │ │ │ │ │ │ │ │ │ answer │ │ │ │ │ │ │ │ │ │ │ │ │ code │ │ │ │ │ │ │ │ │ │ │ │ │ here>, │ │ │ │ │ │ │ │ │ │ │ │ │ "comm… │ │ │ │ │ │ │ │ │ │ │ │ │ "<put │ │ │ │ │ │ │ │ │ │ │ │ │ expla… │ │ │ │ │ │ │ │ │ │ │ │ │ here>… │ │ │ │ │ │ │ │ │ │ │ │ │ 1 │ │ │ │ │ │ │ │ │ │ │ │ │ option │ │ │ │ │ │ │ │ │ │ │ │ │ may be │ │ │ │ │ │ │ │ │ │ │ │ │ selec… │ │ │ │ │ │ │ │ │ │ │ │ │ 'clas… │ │ │ │ │ │ │ │ │ │ │ │ │ 'Mult… │ │ └─────────┴────────┴─────────┴────────┴─────────┴────────┴─────────┴────────┴────────┴─────────┴────────┴─────────┘
The result includes information about the LLM and prompts that were used, and a field for any additional commentary by the LLM. We can select the fields that we want to print and apply some labels to our table:
(result
.select("snow_shoveling", "snow_shoveling_comment")
.print(pretty_labels={"answer.snow_shoveling":q1.question_text, "answer.snow_shoveling_comment":"Comment"})
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Do you enjoy snow shoveling? ┃ Comment ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ No │ No, I do not enjoy snow shoveling. It can be tiring and time-consuming. │ └──────────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
result.select("model.*").print()
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ ┃ model ┃ model ┃ model ┃ model ┃ model ┃ model ┃ model ┃ ┃ .model ┃ .max_tokens ┃ .use_cache ┃ .presence_penalty ┃ .temperature ┃ .top_p ┃ .frequency_penalty ┃ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ gpt-3.5-turbo │ 1000 │ True │ 0 │ 0.5 │ 1 │ 0 │ └───────────────┴─────────────┴────────────┴───────────────────┴──────────────┴────────┴────────────────────┘
result.select("prompt.*").print()
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ prompt ┃ prompt ┃ ┃ .snow_shoveling_user_prompt ┃ .snow_shoveling_system_prompt ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ {'text': 'You are being asked the following question: │ {'text': 'You are answering questions as if you were a │ │ Do you enjoy snow shoveling?\nThe options are\n\n0: │ human. Do not break character. You are an agent with │ │ Yes\n\n1: No\n\n2: I do not know\n\nReturn a valid │ the following persona:\n{}', 'class_name': │ │ JSON formatted like this, selecting only the number of │ 'AgentInstruction'} │ │ the option:\n{"answer": <put answer code here>, │ │ │ "comment": "<put explanation here>"}\nOnly 1 option │ │ │ may be selected.', 'class_name': │ │ │ 'MultipleChoiceTurbo'} │ │ └────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘
result
Result 0
Result ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Attribute ┃ Value ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ agent │ Agent(traits = {}) │ │ scenario │ {} │ │ model │ LanguageModelOpenAIThreeFiveTurbo(model = 'gpt-3.5-turbo', parameters={'temperature': 0.5, │ │ │ 'max_tokens': 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │ │ iteration │ 0 │ │ answer │ {'snow_shoveling': 'No', 'snow_shoveling_comment': 'No, I do not enjoy snow shoveling. It can be │ │ │ tiring and time-consuming.'} │ │ prompt │ {'snow_shoveling_user_prompt': {'text': 'You are being asked the following question: Do you enjoy │ │ │ snow shoveling?\nThe options are\n\n0: Yes\n\n1: No\n\n2: I do not know\n\nReturn a valid JSON │ │ │ formatted like this, selecting only the number of the option:\n{"answer": <put answer code here>, │ │ │ "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │ │ │ 'MultipleChoiceTurbo'}, 'snow_shoveling_system_prompt': {'text': 'You are answering questions as if │ │ │ you were a human. Do not break character. You are an agent with the following persona:\n{}', │ │ │ 'class_name': 'AgentInstruction'}} │ └───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
Result 0
Result
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent(traits = {}) │
│ scenario │ {} │
│ model │ LanguageModelOpenAIThreeFiveTurbo(model = 'gpt-3.5-turbo', parameters={'temperature': 0.5, │
│ │ 'max_tokens': 1000, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'use_cache': True}) │
│ iteration │ 0 │
│ answer │ {'snow_shoveling': 'No', 'snow_shoveling_comment': 'No, I do not enjoy snow shoveling. It can be │
│ │ tiring and time-consuming.'} │
│ prompt │ {'snow_shoveling_user_prompt': {'text': 'You are being asked the following question: Do you enjoy │
│ │ snow shoveling?\nThe options are\n\n0: Yes\n\n1: No\n\n2: I do not know\n\nReturn a valid JSON │
│ │ formatted like this, selecting only the number of the option:\n{"answer": <put answer code here>, │
│ │ "comment": "<put explanation here>"}\nOnly 1 option may be selected.', 'class_name': │
│ │ 'MultipleChoiceTurbo'}, 'snow_shoveling_system_prompt': {'text': 'You are answering questions as if │
│ │ you were a human. Do not break character. You are an agent with the following persona:\n{}', │
│ │ 'class_name': 'AgentInstruction'}} │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘
We can use the add_question() method to chain questions and administer them to an LLM at once. A Yes/No question has preset options:
from edsl.questions import QuestionYesNo
q2 = QuestionYesNo(
question_name = "own_shovel",
question_text = "Do you own a shovel?"
)
result = q1.add_question(q2).run()
(result
.select("snow_shoveling","own_shovel")
.print(pretty_labels={"answer.snow_shoveling":q1.question_text, "answer.own_shovel":q2.question_text})
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Do you enjoy snow shoveling? ┃ Do you own a shovel? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ No │ No │ └──────────────────────────────┴──────────────────────┘
We can add skip-logic to make one question dependent on the response to another question with the method add_stop_rule(), which applies a rule expression to determine whether to add the next question:
result = (q1
.add_question(q2)
.add_stop_rule("snow_shoveling", "snow_shoveling == 'No'")
.run()
)
(result
.select("answer.*")
.print()
)
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .own_shovel ┃ .own_shovel_comment ┃ .snow_shoveling ┃ .snow_shoveling_comment ┃ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ No │ No, I do not own a shovel. │ No │ No, I do not enjoy snow shoveling. It can be │ │ │ │ │ tiring and time-consuming. │ └─────────────┴────────────────────────────┴─────────────────┴────────────────────────────────────────────────────┘
Specifying the agent LLM¶
In the examples above we administered our questions to the default LLM (GPT 4) and provided no additional instructions about the LLM. Here we add a persona for the LLM to reference in responding to our questions, and also specify the LLM as GPT 4 for demonstration purposes (we can select a single model to use or multiple models at once):
You are a winter enthusiast.
from edsl.agents import Agent
persona = "You are a winter enthusiast."
agent = Agent(name="winter_lover", traits={"persona":persona})
Note that the agent name is optional and useful for data analysis. If we do not specify a name, a defaul name Agent_0 will be included when we generate results.
Now we use the method by() to administer a question to our personified agent (still using the default LLM):
result = q1.by(agent).run()
(result
.select("agent.*", "snow_shoveling")
.print(pretty_labels={"agent.persona":"Persona", "answer.snow_shoveling":q1.question_text, "answer.snow_shoveling_comment":"Comment"})
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ agent ┃ ┃ ┃ Persona ┃ .agent_name ┃ Do you enjoy snow shoveling? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ You are a winter enthusiast. │ winter_lover │ Yes │ └──────────────────────────────┴──────────────┴──────────────────────────────┘
Here we do the same but specify that the LLM is GPT 4:
m4 = Model('gpt-4-1106-preview')
result = q1.by(agent).by(m4).run()
result.select('agent.*','answer.*','model.model').print()
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ ┃ agent ┃ agent ┃ answer ┃ answer ┃ model ┃ ┃ .persona ┃ .agent_name ┃ .snow_shoveling ┃ .snow_shoveling_comment ┃ .model ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ You are a winter │ winter_lover │ Yes │ There's something │ gpt-4-1106-preview │ │ enthusiast. │ │ │ satisfying about clearing a │ │ │ │ │ │ path through the fresh snow │ │ │ │ │ │ and the quiet world it │ │ │ │ │ │ creates. Plus, it's a great │ │ │ │ │ │ workout! │ │ └─────────────────────────────┴──────────────┴─────────────────┴─────────────────────────────┴────────────────────┘
If we want to use multiple models we list them together:
m35 = Model('gpt-3.5-turbo', cache=False)
result = q1.by(agent).by(m35, m4).run()
result.select('agent.*','answer.*','model.model').print()
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ ┃ agent ┃ agent ┃ answer ┃ answer ┃ model ┃ ┃ .persona ┃ .agent_name ┃ .snow_shoveling ┃ .snow_shoveling_comment ┃ .model ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ You are a winter │ winter_lover │ Yes │ Yes, I enjoy snow │ gpt-3.5-turbo │ │ enthusiast. │ │ │ shoveling. It's a great way │ │ │ │ │ │ to stay active and enjoy │ │ │ │ │ │ the winter weather. │ │ ├─────────────────────────────┼──────────────┼─────────────────┼─────────────────────────────┼────────────────────┤ │ You are a winter │ winter_lover │ Yes │ There's something │ gpt-4-1106-preview │ │ enthusiast. │ │ │ satisfying about clearing a │ │ │ │ │ │ path through the fresh snow │ │ │ │ │ │ and the quiet world it │ │ │ │ │ │ creates. Plus, it's a great │ │ │ │ │ │ workout! │ │ └─────────────────────────────┴──────────────┴─────────────────┴─────────────────────────────┴────────────────────┘
Designing a survey¶
This section shows how to construct questions of different types, combine them into surveys and administer them to panels of diverse AI agents.
Setup¶
Importing the basic edsl tools:
from edsl.questions import QuestionMultipleChoice, QuestionCheckBox, QuestionFreeText, QuestionList, QuestionBudget, QuestionYesNo, QuestionLinearScale
from edsl import Agent, Survey, Scenario, Model
m4 = Model('gpt-4-1106-preview', cache=False)
Creating questions¶
Here we create questions of different types. The question_name field is optional; a default name will be applied if a name is not specified. The other fields shown for each question type are required. Each type can also take an optional boolean: allow_nonresponse = False
The following settings are preset:
MAX_ANSWER_LENGTH = 2000
MAX_EXPRESSION_CONSTRAINT_LENGTH = 1000
MAX_NUM_OPTIONS = 20
MIN_NUM_OPTIONS = 2
MAX_OPTION_LENGTH = 1000
MAX_QUESTION_LENGTH = 100000q_multiple_choice = QuestionMultipleChoice(
question_name="multiple_choice",
question_text="How often does it snow where you live?",
question_options=["Rarely","Occasionally","Often","Always"],
)
q_yes_no = QuestionYesNo(
question_name="yes_no",
question_text="Have you previously owned or used a snow shovel?",
)
q_linear_scale = QuestionLinearScale(
question_name="linear_scale",
question_text="""On a scale of 0 to 10, how important is owning a snow shovel to you?
(0 being not important at all, and 10 being extremely important)""",
question_options=[0,1,2,3,4,5,6,7,8,9,10],
)
q_checkbox = QuestionCheckBox(
question_name="checkbox",
question_text="Where do you prefer to purchase a snow shovel? (Select all that apply.)",
question_options=[
"Local hardware store",
"Big box retailer (e.g., Home Depot, Walmart)",
"Online (e.g., Amazon)",
"Specialty stores",
"I do not typically purchase snow shovels"
],
)
q_free_text = QuestionFreeText(
question_name="free_text",
question_text="Have you experienced any issues with snow shovels in the past?",
)
q_list = QuestionList(
question_name="list",
question_text="List your top considerations when deciding whether to purchase a snow shovel.",
)
q_budget = QuestionBudget(
question_name="budget",
question_text="What percentage of your total time snow shoveling is typically spent shoveling each of the following areas?",
question_options=[
"Sidewalks and walkways",
"Driveways",
"Cars"
],
budget_sum=100,
)
Creating surveys¶
We can combine questions into a survey to administer them altogether:
survey_snow_shovels = Survey(
questions = [
q_multiple_choice,
q_yes_no,
q_linear_scale,
q_checkbox,
q_free_text,
q_list,
q_budget
]
)
Constructing agents¶
Next we identify target audiences and create personas for AI agents that will "respond" to our survey. Here we create a single agent with 3 different traits:
agent_robin = Agent(traits={
"location":"Massachusetts",
"gender":"Female",
"age":"44 years old"
})
We can also create panels of agents using lists of traits at once:
locations = ["San Diego, California", "Portland, Maine"]
ages = ["child", "adult"]
agents_locations = [Agent(traits={"location":l}) for l in locations]
agents_ages = [Agent(traits={"age":a}) for a in ages]
agent_robin
Agent(traits = {'location': 'Massachusetts', 'gender': 'Female', 'age': '44 years old'})
agents_locations
[Agent(traits = {'location': 'San Diego, California'}),
Agent(traits = {'location': 'Portland, Maine'})]
agents_ages
[Agent(traits = {'age': 'child'}), Agent(traits = {'age': 'adult'})]
Simulating results¶
We can administer questions and surveys to agents individually or all at once. Here we administer a single question to an individual agent using a specified LLM:
result_robin = q_yes_no.by(agent_robin).by(m4).run()
(result_robin
.select("agent.*", "yes_no")
.print(pretty_labels={"agent.gender":"Gender", "agent.location":"Location", "agent.age":"Age", "answer.yes_no":q_yes_no.question_text})
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ agent ┃ ┃ ┃ ┃ Gender ┃ Location ┃ .agent_name ┃ Age ┃ Have you previously owned or used a snow shovel? ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Female │ Massachusetts │ Agent_3 │ 44 years old │ Yes │ └────────┴───────────────┴─────────────┴──────────────┴──────────────────────────────────────────────────┘
Here we administer our question to a panel of agents with all possible combinations of the traits that we specified:
result_locations_ages = q_yes_no.by(agents_locations).by(agents_ages).by(m4).run()
(result_locations_ages
.select("agent.*","yes_no")
.print(pretty_labels={"agent.location":"Location", "agent.age":"Age", "answer.yes_no":q_yes_no.question_text})
)
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ agent ┃ ┃ ┃ ┃ Location ┃ .agent_name ┃ Age ┃ Have you previously owned or used a snow shovel? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ San Diego, California │ Agent_4 │ child │ No │ ├───────────────────────┼─────────────┼───────┼──────────────────────────────────────────────────┤ │ San Diego, California │ Agent_5 │ adult │ No │ ├───────────────────────┼─────────────┼───────┼──────────────────────────────────────────────────┤ │ Portland, Maine │ Agent_6 │ child │ Yes │ ├───────────────────────┼─────────────┼───────┼──────────────────────────────────────────────────┤ │ Portland, Maine │ Agent_7 │ adult │ Yes │ └───────────────────────┴─────────────┴───────┴──────────────────────────────────────────────────┘
Here we administer our entire survey to an agent:
result_robin = survey_snow_shovels.by(agent_robin).by(m4).run()
(result_robin
.select("agent.*",
"multiple_choice",
"yes_no",
"linear_scale",
# "checkbox",
# "free_text",
# "list",
# "budget"
)
.print(pretty_labels={
"agent.location":"Location",
"agent.gender":"Gender",
"agent.age":"Age",
"answer.multiple_choice":q_multiple_choice.question_text,
"answer.yes_no":q_yes_no.question_text,
"answer.linear_scale":q_linear_scale.question_text,
# "answer.checkbox":q_checkbox.question_text,
# "answer.free_text":q_free_text.question_text,
# "answer.list":q_list.question_text,
# "answer.budget":q_budget.question_text
})
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃ ┃ ┃ ┃ On a scale of 0 ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ to 10, how ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ important is ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ owning a snow ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ shovel to you? ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ (0 being ┃ ┃ ┃ ┃ ┃ ┃ ┃ Have you ┃ not important at ┃ ┃ ┃ ┃ ┃ ┃ How often does it ┃ previously owned ┃ all, and 10 being ┃ ┃ ┃ ┃ agent ┃ ┃ snow where you ┃ or used a snow ┃ extremely ┃ ┃ Gender ┃ Location ┃ .agent_name ┃ Age ┃ live? ┃ shovel? ┃ important) ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ Female │ Massachusetts │ Agent_3 │ 44 years old │ Often │ Yes │ 9 │ └────────┴───────────────┴─────────────┴──────────────┴───────────────────┴───────────────────┴───────────────────┘
(result_robin
.select(#"agent.*",
# "multiple_choice",
# "yes_no",
# "linear_scale",
"checkbox",
"free_text",
"list",
"budget"
)
.print(pretty_labels={
# "agent.location":"Location",
# "agent.gender":"Gender",
# "agent.age":"Age",
# "answer.multiple_choice":q_multiple_choice.question_text,
# "answer.yes_no":q_yes_no.question_text,
# "answer.linear_scale":q_linear_scale.question_text,
"answer.checkbox":q_checkbox.question_text,
"answer.free_text":q_free_text.question_text,
"answer.list":q_list.question_text,
"answer.budget":q_budget.question_text
})
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ List your top ┃ What percentage of your ┃ ┃ Where do you prefer to ┃ ┃ considerations when ┃ total time snow shoveling ┃ ┃ purchase a snow shovel? ┃ Have you experienced any ┃ deciding whether to ┃ is typically spent ┃ ┃ (Select all that apply ┃ issues with snow shovels ┃ purchase a snow shovel ┃ shoveling each of the ┃ ┃ .) ┃ in the past? ┃ . ┃ following areas? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ ['Local hardware store', │ Yes, I've had a few │ ['ergonomic handle', │ [{'Sidewalks and │ │ 'Big box retailer (e.g., │ issues with snow shovels │ 'blade width', 'material │ walkways': 30}, │ │ Home Depot, Walmart)'] │ in the past. One common │ durability', 'weight', │ {'Driveways': 50}, │ │ │ problem is the handle │ 'length', 'price', 'brand │ {'Cars': 20}] │ │ │ bending or breaking when │ reputation', 'storage │ │ │ │ the snow is particularly │ space required'] │ │ │ │ heavy or when ice is │ │ │ │ │ mixed in. I've also │ │ │ │ │ experienced the shovel │ │ │ │ │ blade cracking or the │ │ │ │ │ edge wearing down after │ │ │ │ │ repeated use over a few │ │ │ │ │ seasons. Finding an │ │ │ │ │ ergonomic handle that's │ │ │ │ │ comfortable for longer │ │ │ │ │ periods of shoveling has │ │ │ │ │ been a challenge as well. │ │ │ │ │ And let's not forget the │ │ │ │ │ occasional back strain │ │ │ │ │ from lifting too much │ │ │ │ │ snow at once! │ │ │ └────────────────────────────┴───────────────────────────┴────────────────────────────┴───────────────────────────┘
We can also use qualitative responses to create new quantitative questions by extracting themes and recurring topics in those responses.
Here we take the issues with snow shovels in the past and condense them into a list that we can use as options for new questions:
issues = result_robin.select("free_text")[0]["answer.free_text"][0]
issues
"Yes, I've had a few issues with snow shovels in the past. One common problem is the handle bending or breaking when the snow is particularly heavy or when ice is mixed in. I've also experienced the shovel blade cracking or the edge wearing down after repeated use over a few seasons. Finding an ergonomic handle that's comfortable for longer periods of shoveling has been a challenge as well. And let's not forget the occasional back strain from lifting too much snow at once!"
q_issues_list = QuestionList(
question_name="issues_list",
question_text="Create a list of issues mentioned in this text: " + issues
)
r_issues = q_issues_list.by(m4).run()
issues = r_issues.select("issues_list").to_list()[0]
issues
['handle bending or breaking', 'heavy snow or ice', 'shovel blade cracking', 'edge wearing down', 'ergonomic handle', 'back strain']
q_issues = QuestionMultipleChoice(
question_name="top_issue",
question_text="Select your top issue in using a snow shovel.",
question_options=issues
)
r_issues_robin = q_issues.by(agent_robin).by(m4).run()
r_issues_robin.select("top_issue").print()
┏━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .top_issue ┃ ┡━━━━━━━━━━━━━┩ │ back strain │ └─────────────┘
Scenarios¶
We can also create multiple versions of a question that we administer to agents at once as a set of "scenarios" with different inputs:
q_blizzard = QuestionYesNo(
question_name="blizzard",
question_text="""A blizzard is coming.
Predicted snowfall accumulation is {{snowfall}}.
You do not own or have access to a shovel.
A shovel at a store nearby normally costs $25 but is now marked up {{markup}}.
Will you purchase it?""",
)
snowfalls = ["1-6in", "6-12in", "1-2ft", "2-4ft"]
markups = ["10%", "30%", "50%", "100%"]
scenarios = [Scenario({"snowfall":s, "markup":m}) for s in snowfalls for m in markups]
r_blizzard = q_blizzard.by(scenarios).by(m4).run()
(r_blizzard
.select("scenario.*", "blizzard")
.print(pretty_labels={
"scenario.snowfall":"Predicted snowfall",
"scenario.markup":"Snow shovel markup",
"answer.blizzard":"Purchase decision"
})
)
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Predicted snowfall ┃ Snow shovel markup ┃ Purchase decision ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ 1-6in │ 10% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-6in │ 30% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-6in │ 50% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-6in │ 100% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 6-12in │ 10% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 6-12in │ 30% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 6-12in │ 50% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 6-12in │ 100% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-2ft │ 10% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-2ft │ 30% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-2ft │ 50% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 1-2ft │ 100% │ No │ ├────────────────────┼────────────────────┼───────────────────┤ │ 2-4ft │ 10% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 2-4ft │ 30% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 2-4ft │ 50% │ Yes │ ├────────────────────┼────────────────────┼───────────────────┤ │ 2-4ft │ 100% │ No │ └────────────────────┴────────────────────┴───────────────────┘
We can filter the results based on the responses:
(r_blizzard.filter("markup == '50%' and snowfall in ['1-6in','6-12in']")
.select('scenario.*','answer.*')
.print()
)
┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ scenario ┃ answer ┃ answer ┃ ┃ .snowfall ┃ .markup ┃ .blizzard ┃ .blizzard_comment ┃ ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ 1-6in │ 50% │ No │ Given the snowfall accumulation prediction of 1-6 inches, I might consider │ │ │ │ │ alternative methods to clear the snow, such as using a broom or borrowing a │ │ │ │ │ shovel from a neighbor, rather than purchasing a shovel at a 50% markup. │ ├───────────┼──────────┼───────────┼──────────────────────────────────────────────────────────────────────────────┤ │ 6-12in │ 50% │ No │ I have decided not to purchase the shovel at a 50% markup because I find the │ │ │ │ │ price increase to be unreasonable and I may consider alternative solutions │ │ │ │ │ for dealing with the snow. │ └───────────┴──────────┴───────────┴──────────────────────────────────────────────────────────────────────────────┘
Seeding a question¶
Scenarios can also be used to seed questions, which can be useful in ensuring consistency in responses to related questions. We do this by using the response to a question as a parameter of a subsequent question:
from edsl.questions import QuestionFunctional
from edsl.questions.compose_questions import compose_questions
from edsl import Scenario
q_seasons_best = QuestionMultipleChoice(
question_name = "seasons_best",
question_text = "What do you like best about {{season}}?",
question_options = ["Sports", "Weather", "Holidays", "Other"]
)
q_reasons = QuestionFreeText(
question_name = "reasons",
question_text = "You were previously asked: '" + q_seasons_best.question_text
+ "' You responded: '{{seasons_best}}'. Explain the reasons for your choice."
)
q_seeded = compose_questions(q_seasons_best, q_reasons)
r_seeded = q_seeded.by(Scenario({"season":"winter"})).by(m4).run()
(r_seeded
.select("scenario.season", "scenario.seasons_best", "answer.*")
.print(pretty_labels={"scenario.season":"Season", "scenario.seasons_best":q_seasons_best.question_text, "answer.seasons_best_reasons":"Top 3 reasons"})
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ What do you like best about ┃ answer ┃ ┃ ┃ Season ┃ {{season}}? ┃ .seasons_best_reasons_comment ┃ Top 3 reasons ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ winter │ {'answer.seasons_best': │ Functional. │ {'answer': {'answer.reasons': ['I │ │ │ ['Holidays']} │ │ chose holidays as the best part of │ │ │ │ │ winter because I enjoy spending │ │ │ │ │ time with family and friends, │ │ │ │ │ exchanging gifts, and taking part │ │ │ │ │ in festive traditions that bring │ │ │ │ │ joy and warmth during the colder │ │ │ │ │ months.']}, 'comment': None} │ └────────┴───────────────────────────────────┴───────────────────────────────┴────────────────────────────────────┘
Generating new results¶
By default, survey results are cached to avoid unintended costs in rerunning identical surveys. If you want to generate new results for a survey, one option is to modify the selection and parameters of the LLM. In the next example we (very slightly) vary the temperature of the LLM to achieve this:
models = [Model('gpt-4-1106-preview', temperature=0.5 + e/10000) for e in range(10)]
r_models = q_linear_scale.by(agent_robin).by(models).run()
r_models.select('model.temperature','answer.linear_scale','answer.linear_scale_comment').print()
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ model ┃ answer ┃ answer ┃ ┃ .temperature ┃ .linear_scale ┃ .linear_scale_comment ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ 0.5 │ 9 │ Living in Massachusetts, where we can get significant snowfall, owning a snow │ │ │ │ shovel is very important for clearing walkways and driveways. It's not a perfect │ │ │ │ 10 because sometimes a snow blower or hiring a plowing service can be an │ │ │ │ alternative, but it's still quite essential. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5001 │ 9 │ Living in Massachusetts, where we get significant snowfall each winter, owning a │ │ │ │ snow shovel is almost essential for clearing walkways and driveways. It's not a │ │ │ │ 10 because there are alternatives like snow blowers or hiring snow removal │ │ │ │ services, but it's still very important. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5002 │ 9 │ Living in Massachusetts, where we get significant snowfall each winter, owning a │ │ │ │ snow shovel is almost essential for clearing driveways and sidewalks. I've rated │ │ │ │ it a 9 because while it's very important, there could be alternatives like snow │ │ │ │ blowers or hiring snow removal services. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5003 │ 10 │ Living in Massachusetts, where we get heavy snowfall each winter, owning a snow │ │ │ │ shovel is essential for clearing driveways and sidewalks. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5004 │ 9 │ Living in Massachusetts, where we experience heavy snowfall, owning a snow │ │ │ │ shovel is almost essential for clearing driveways and sidewalks, hence the high │ │ │ │ importance. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5005 │ 9 │ Living in Massachusetts, where we get heavy snowfall each winter, owning a snow │ │ │ │ shovel is almost essential for clearing driveways and sidewalks. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5006 │ 9 │ Living in Massachusetts, where we experience heavy snowfall in the winter, │ │ │ │ owning a snow shovel is almost essential for clearing driveways and sidewalks. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5007 │ 10 │ Living in Massachusetts, where we get significant snowfall during the winter, │ │ │ │ owning a snow shovel is essential for clearing driveways and sidewalks. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5008 │ 10 │ Living in Massachusetts, where we get heavy snowfall each winter, owning a snow │ │ │ │ shovel is essential for clearing driveways and sidewalks. │ ├──────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ 0.5009 │ 9 │ Living in Massachusetts, where we get a heavy snowfall each year, owning a snow │ │ │ │ shovel is almost essential for clearing driveways and sidewalks. It's not a │ │ │ │ perfect 10 because there are alternatives like snow blowers or hiring snow │ │ │ │ removal services, but it's very important. │ └──────────────┴───────────────┴──────────────────────────────────────────────────────────────────────────────────┘