AI Tools¶
lumen.ai.tools
¶
__all__ = ['Tool', 'FunctionTool', 'ToolUser', 'VectorLookupTool', 'VectorLookupToolUser', 'TableLookup', 'IterativeTableLookup', 'DocumentLookup', 'DbtslLookup']
module-attribute
¶
DbtslLookup
¶
Bases:
DbtslLookup tool that creates a vector store of all available dbt semantic layers and responds with relevant metrics for user queries.
dimension_fetch_timeout = param.Number(default=15, doc='\n Maximum time in seconds to wait for a dimension values fetch operation.')
class-attribute
instance-attribute
¶
max_concurrent = param.Integer(default=5, doc='\n Maximum number of concurrent metadata fetch operations.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0.1, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=4, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
output_schema = DbtslLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}, 'main': {'template': PROMPTS_DIR / 'DbtslLookup' / 'main.jinja2', 'response_model': ThinkingYesNo}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Looks up additional context by querying dbt semantic layers based on the user query with a vector store.\n Useful for quickly gathering information about dbt semantic layers and their metrics to plan the steps.\n Not useful for looking up what datasets are available. Likely useful for all queries.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches metrics based on the user query, populates the DbtslMetaset, and returns formatted context.
DocumentLookup
¶
Bases:
The DocumentLookup tool creates a vector store of all available documents and responds with a list of the most relevant documents given the user query. Always use this for more context.
FunctionTool
¶
Bases:
FunctionTool wraps arbitrary functions and makes them available as a tool for an LLM to call. It inspects the arguments of the function and generates a pydantic Model that the LLM will populate.
The function may also consume information in memory, e.g. the current table or pipeline by declaring the requires parameter.
The function may also return context to add to the current working memory
by returning a dictionary and declaring the provides parameter. Any
keys listed in the provides parameter will be copied into working memory.
formatter = param.Parameter(default='{function}({arguments}) returned: {output}', doc="\n Formats the return value for inclusion in the global context.\n Accepts the 'function', 'arguments' and 'output' as formatting variables.")
class-attribute
instance-attribute
¶
function = param.Callable(default=None, allow_refs=False, doc='\n The function to call.')
class-attribute
instance-attribute
¶
inputs
property
¶
prompts = param.Dict(default={'main': {'template': PROMPTS_DIR / 'FunctionTool' / 'main.jinja2'}})
class-attribute
instance-attribute
¶
provides = param.List(default=[], readonly=False, constant=True, doc='\n List of context values it provides to current working memory.')
class-attribute
instance-attribute
¶
render_output = param.Boolean(default=False, doc='\n Whether to render the tool output directly, even if it is not already a Lumen View or Panel Viewable.')
class-attribute
instance-attribute
¶
requires = param.List(default=[], readonly=False, constant=True, doc='\n List of context values it requires to be in memory.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
IterativeTableLookup
¶
Bases:
Extended version of TableLookup that performs an iterative table selection process. This tool uses an LLM to select tables in multiple passes, examining schemas in detail.
conditions = param.List(default=['Useful for answering data queries', 'Avoid for follow-up questions when existing data is sufficient', 'Use only when existing information is insufficient for the current query'])
class-attribute
instance-attribute
¶
max_selection_iterations = param.Integer(default=3, doc='\n Maximum number of iterations for the iterative table selection process.')
class-attribute
instance-attribute
¶
not_with = param.List(default=['TableLookup'])
class-attribute
instance-attribute
¶
output_schema = IterativeTableLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}, 'iterative_selection': {'template': PROMPTS_DIR / 'IterativeTableLookup' / 'iterative_selection.jinja2', 'response_model': make_iterative_selection_model}})
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Looks up the most relevant tables and provides SQL schemas of those tables.')
class-attribute
instance-attribute
¶
table_similarity_threshold = param.Number(default=0.5, doc='\n If any tables have a similarity score above this threshold,\n those tables will be automatically selected and there will not be an\n iterative table selection process.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches tables based on the user query and returns formatted context.
sync(context)
async
¶
Update metaset when visible_slugs changes. This ensures SQL operations only work with visible tables by regenerating the metaset using get_metaset.
TableLookup
¶
Bases:
TableLookup tool that creates a vector store of all available tables and responds with relevant tables for user queries.
conditions = param.List(default=['Best paired with ChatAgent for general conversation about data', 'Avoid if table discovery already performed for same request', 'Not useful for data related queries'])
class-attribute
instance-attribute
¶
exclusions = param.List(default=['dbtsl_metaset'])
class-attribute
instance-attribute
¶
include_columns = param.Boolean(default=True, doc='\n Whether to include column names and descriptions in the embeddings.')
class-attribute
instance-attribute
¶
include_metadata = param.Boolean(default=True, doc='\n Whether to include table descriptions in the embeddings and responses.')
class-attribute
instance-attribute
¶
include_misc = param.Boolean(default=False, doc='\n Whether to include miscellaneous metadata in the embeddings,\n besides table and column descriptions.')
class-attribute
instance-attribute
¶
input_schema = TableLookupInputs
class-attribute
instance-attribute
¶
max_concurrent = param.Integer(default=1, doc='\n Maximum number of concurrent metadata fetch operations.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=10, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
not_with = param.List(default=['IterativeTableLookup'])
class-attribute
instance-attribute
¶
output_schema = TableLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Discovers relevant tables using vector search, providing context for other agents.\n Not to be used for finding tables for further analysis (e.g. SQL), because it does\n not provide a schema.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches tables based on the user query and returns formatted context.
sync(context)
async
¶
Sync the vector store with updated context. Only processes new sources that haven't been added yet.
Tool
¶
Bases:
A Tool can be invoked by another Actor to provide additional context or respond to a question. Unlike an Agent they never interact with or on behalf of a user directly.
always_use = param.Boolean(default=False, doc='\n Whether to always use this tool, even if it is not explicitly\n required by the current context.')
class-attribute
instance-attribute
¶
conditions = param.List(default=['Always requires a supporting agent to interpret results'])
class-attribute
instance-attribute
¶
applies(context)
async
classmethod
¶
Additional checks to determine if the tool should be used.
prepare(context)
async
¶
Prepare the tool with the initial context. Called once when the tool is first initialized.
sync(context)
async
¶
Allows the tool to update when the provided context changes. Subclasses should override this to handle context updates.
ToolUser
¶
Bases:
ToolUser is a mixin class for actors that use tools.
tools = param.List(default=[], doc='\n List of tools to use.')
class-attribute
instance-attribute
¶
VectorLookupTool
¶
Bases:
Baseclass for tools that search a vector database for relevant chunks.
enable_query_refinement = param.Boolean(default=True, doc='\n Whether to enable query refinement for improving search results.')
class-attribute
instance-attribute
¶
max_refinement_iterations = param.Integer(default=3, bounds=(1, 10), doc='\n Maximum number of refinement iterations to perform.')
class-attribute
instance-attribute
¶
min_refinement_improvement = param.Number(default=0.05, bounds=(0, 1), doc='\n Minimum improvement in similarity score required to keep refining.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0.3, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=5, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
output_schema = VectorLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
refinement_similarity_threshold = param.Number(default=0.3, bounds=(0, 1), doc='\n Similarity threshold below which query refinement is triggered.')
class-attribute
instance-attribute
¶
vector_store = param.ClassSelector(class_=VectorStore, constant=True, doc='\n Vector store object which is queried to provide additional context\n before responding.')
class-attribute
instance-attribute
¶
prepare(context)
async
¶
respond(messages, context, **kwargs)
async
¶
Respond to a user query using the vector store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages
|
|
The user query and any additional context |
required |
**kwargs
|
|
Additional arguments for the response |
{}
|
Returns:
| Type | Description |
|---|---|
|
The response from the vector store |
VectorLookupToolUser
¶
Bases:
VectorLookupToolUser is a mixin class for actors that use vector lookup tools.
document_vector_store = param.ClassSelector(class_=VectorStore, default=None, doc='\n The vector store to use for document tools. If not provided, a new one will be created\n or inferred from the tools provided.')
class-attribute
instance-attribute
¶
vector_store = param.ClassSelector(class_=VectorStore, default=None, doc='\n The vector store to use for the tools. If not provided, a new one will be created\n or inferred from the tools provided.')
class-attribute
instance-attribute
¶
base
¶
FunctionTool
¶
Bases:
FunctionTool wraps arbitrary functions and makes them available as a tool for an LLM to call. It inspects the arguments of the function and generates a pydantic Model that the LLM will populate.
The function may also consume information in memory, e.g. the current table or pipeline by declaring the requires parameter.
The function may also return context to add to the current working memory
by returning a dictionary and declaring the provides parameter. Any
keys listed in the provides parameter will be copied into working memory.
formatter = param.Parameter(default='{function}({arguments}) returned: {output}', doc="\n Formats the return value for inclusion in the global context.\n Accepts the 'function', 'arguments' and 'output' as formatting variables.")
class-attribute
instance-attribute
¶
function = param.Callable(default=None, allow_refs=False, doc='\n The function to call.')
class-attribute
instance-attribute
¶
inputs
property
¶
prompts = param.Dict(default={'main': {'template': PROMPTS_DIR / 'FunctionTool' / 'main.jinja2'}})
class-attribute
instance-attribute
¶
provides = param.List(default=[], readonly=False, constant=True, doc='\n List of context values it provides to current working memory.')
class-attribute
instance-attribute
¶
render_output = param.Boolean(default=False, doc='\n Whether to render the tool output directly, even if it is not already a Lumen View or Panel Viewable.')
class-attribute
instance-attribute
¶
requires = param.List(default=[], readonly=False, constant=True, doc='\n List of context values it requires to be in memory.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Tool
¶
Bases:
A Tool can be invoked by another Actor to provide additional context or respond to a question. Unlike an Agent they never interact with or on behalf of a user directly.
always_use = param.Boolean(default=False, doc='\n Whether to always use this tool, even if it is not explicitly\n required by the current context.')
class-attribute
instance-attribute
¶
conditions = param.List(default=['Always requires a supporting agent to interpret results'])
class-attribute
instance-attribute
¶
applies(context)
async
classmethod
¶
Additional checks to determine if the tool should be used.
prepare(context)
async
¶
Prepare the tool with the initial context. Called once when the tool is first initialized.
sync(context)
async
¶
Allows the tool to update when the provided context changes. Subclasses should override this to handle context updates.
dbtsl_lookup
¶
DbtslLookup
¶
Bases:
DbtslLookup tool that creates a vector store of all available dbt semantic layers and responds with relevant metrics for user queries.
dimension_fetch_timeout = param.Number(default=15, doc='\n Maximum time in seconds to wait for a dimension values fetch operation.')
class-attribute
instance-attribute
¶
max_concurrent = param.Integer(default=5, doc='\n Maximum number of concurrent metadata fetch operations.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0.1, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=4, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
output_schema = DbtslLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}, 'main': {'template': PROMPTS_DIR / 'DbtslLookup' / 'main.jinja2', 'response_model': ThinkingYesNo}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Looks up additional context by querying dbt semantic layers based on the user query with a vector store.\n Useful for quickly gathering information about dbt semantic layers and their metrics to plan the steps.\n Not useful for looking up what datasets are available. Likely useful for all queries.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches metrics based on the user query, populates the DbtslMetaset, and returns formatted context.
document_lookup
¶
DocumentLookup
¶
Bases:
The DocumentLookup tool creates a vector store of all available documents and responds with a list of the most relevant documents given the user query. Always use this for more context.
table_lookup
¶
IterativeTableLookup
¶
Bases:
Extended version of TableLookup that performs an iterative table selection process. This tool uses an LLM to select tables in multiple passes, examining schemas in detail.
conditions = param.List(default=['Useful for answering data queries', 'Avoid for follow-up questions when existing data is sufficient', 'Use only when existing information is insufficient for the current query'])
class-attribute
instance-attribute
¶
max_selection_iterations = param.Integer(default=3, doc='\n Maximum number of iterations for the iterative table selection process.')
class-attribute
instance-attribute
¶
not_with = param.List(default=['TableLookup'])
class-attribute
instance-attribute
¶
output_schema = IterativeTableLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}, 'iterative_selection': {'template': PROMPTS_DIR / 'IterativeTableLookup' / 'iterative_selection.jinja2', 'response_model': make_iterative_selection_model}})
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Looks up the most relevant tables and provides SQL schemas of those tables.')
class-attribute
instance-attribute
¶
table_similarity_threshold = param.Number(default=0.5, doc='\n If any tables have a similarity score above this threshold,\n those tables will be automatically selected and there will not be an\n iterative table selection process.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches tables based on the user query and returns formatted context.
sync(context)
async
¶
Update metaset when visible_slugs changes. This ensures SQL operations only work with visible tables by regenerating the metaset using get_metaset.
TableLookup
¶
Bases:
TableLookup tool that creates a vector store of all available tables and responds with relevant tables for user queries.
conditions = param.List(default=['Best paired with ChatAgent for general conversation about data', 'Avoid if table discovery already performed for same request', 'Not useful for data related queries'])
class-attribute
instance-attribute
¶
exclusions = param.List(default=['dbtsl_metaset'])
class-attribute
instance-attribute
¶
include_columns = param.Boolean(default=True, doc='\n Whether to include column names and descriptions in the embeddings.')
class-attribute
instance-attribute
¶
include_metadata = param.Boolean(default=True, doc='\n Whether to include table descriptions in the embeddings and responses.')
class-attribute
instance-attribute
¶
include_misc = param.Boolean(default=False, doc='\n Whether to include miscellaneous metadata in the embeddings,\n besides table and column descriptions.')
class-attribute
instance-attribute
¶
input_schema = TableLookupInputs
class-attribute
instance-attribute
¶
max_concurrent = param.Integer(default=1, doc='\n Maximum number of concurrent metadata fetch operations.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=10, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
not_with = param.List(default=['IterativeTableLookup'])
class-attribute
instance-attribute
¶
output_schema = TableLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
purpose = param.String(default='\n Discovers relevant tables using vector search, providing context for other agents.\n Not to be used for finding tables for further analysis (e.g. SQL), because it does\n not provide a schema.')
class-attribute
instance-attribute
¶
respond(messages, context, **kwargs)
async
¶
Fetches tables based on the user query and returns formatted context.
sync(context)
async
¶
Sync the vector store with updated context. Only processes new sources that haven't been added yet.
make_iterative_selection_model(table_slugs)
¶
Creates a model for table selection in the coordinator. This is separate from the SQLAgent's table selection model to focus on context gathering rather than query execution.
vector_lookup
¶
VectorLookupTool
¶
Bases:
Baseclass for tools that search a vector database for relevant chunks.
enable_query_refinement = param.Boolean(default=True, doc='\n Whether to enable query refinement for improving search results.')
class-attribute
instance-attribute
¶
max_refinement_iterations = param.Integer(default=3, bounds=(1, 10), doc='\n Maximum number of refinement iterations to perform.')
class-attribute
instance-attribute
¶
min_refinement_improvement = param.Number(default=0.05, bounds=(0, 1), doc='\n Minimum improvement in similarity score required to keep refining.')
class-attribute
instance-attribute
¶
min_similarity = param.Number(default=0.3, doc='\n The minimum similarity to include a document.')
class-attribute
instance-attribute
¶
n = param.Integer(default=5, bounds=(1, None), doc='\n The number of document results to return.')
class-attribute
instance-attribute
¶
output_schema = VectorLookupOutputs
class-attribute
instance-attribute
¶
prompts = param.Dict(default={'refine_query': {'template': PROMPTS_DIR / 'VectorLookupTool' / 'refine_query.jinja2', 'response_model': make_refined_query_model}}, doc='Dictionary of available prompts for the tool.')
class-attribute
instance-attribute
¶
refinement_similarity_threshold = param.Number(default=0.3, bounds=(0, 1), doc='\n Similarity threshold below which query refinement is triggered.')
class-attribute
instance-attribute
¶
vector_store = param.ClassSelector(class_=VectorStore, constant=True, doc='\n Vector store object which is queried to provide additional context\n before responding.')
class-attribute
instance-attribute
¶
prepare(context)
async
¶
respond(messages, context, **kwargs)
async
¶
Respond to a user query using the vector store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages
|
|
The user query and any additional context |
required |
**kwargs
|
|
Additional arguments for the response |
{}
|
Returns:
| Type | Description |
|---|---|
|
The response from the vector store |
VectorLookupToolUser
¶
Bases:
VectorLookupToolUser is a mixin class for actors that use vector lookup tools.
document_vector_store = param.ClassSelector(class_=VectorStore, default=None, doc='\n The vector store to use for document tools. If not provided, a new one will be created\n or inferred from the tools provided.')
class-attribute
instance-attribute
¶
vector_store = param.ClassSelector(class_=VectorStore, default=None, doc='\n The vector store to use for the tools. If not provided, a new one will be created\n or inferred from the tools provided.')
class-attribute
instance-attribute
¶
make_refined_query_model(item_type_name='items')
¶
Creates a model for refining search queries in vector lookup tools.