Sources¶

Bases: Source

The AE5Source queries an Anaconda Enterprise 5 instance for statistics.

Specifically it provides tables with information nodes, deployments, sessions, jobs and resource profiles.

Bases: Source

The BaseSQLSource implements the additional API required by a SQL based data source.

Bases: Source

DerivedSource applies filtering and transforms to tables from other sources.

A DerivedSource references tables on other sources and optionally allows applying filters and transforms to the returned data which is then made available as a new (derived) table.

The DerivedSource has two modes:

Table Mode

When an explicit tables specification is provided full control over the exact tables to filter and transform is available. This is referred to as the 'table' mode.

In 'table' mode the tables can reference any table on any source using the reference syntax and declare filters and transforms to apply to that specific table, e.g. a table specification might look like this::

{
  'derived_table': {
    'source': 'original_source',
    'table': 'original_table'
    'filters': [
      ...
    ],
    'transforms': [
      ...
    ]
  }
}

Mirror mode

When a source is declared all tables on that Source are mirrored and filtered and transformed according to the supplied filters and transforms. This is referred to as 'mirror' mode.

In mirror mode the DerivedSource may reference an existing source directly, e.g.::

{
    'type': 'derived',
    'source': 'original_source',
    'filters': [...],
    'transforms': [...],
}

Bases: Source

FileSource loads CSV, Excel and Parquet files using pandas and dask read_* functions.

The FileSource can declare a list or dictionary of local or remote files which are then loaded using either pandas.read_* or dask.dataframe.read_* functions depending on whether use_dask is enabled.

Bases: Source

InMemorySource can be used to work with in-memory data.

Bases: FileSource

The JSONSource is very similar to the FileSource but loads json files.

Both local and remote JSON files can be fetched by declaring them as a list or dictionaries of tables.

Bases: Source

JoinedSource performs a join on tables from one or more sources.

A JoinedSource applies a join on two or more sources returning new table(s) with data from all sources. It iterates over the tables specification and merges the specified tables from the declared sources on the supplied index.

In this way multiple tables from multiple sources can be merged. Individual tables from sources that should not be joined may also be surfaced by declaring a single source and table in the specification.

As a simple example we may have sources A and B, which contain tables 'foo' and 'bar' respectively. We now want to merge these tables on column 'a' in Table A with column 'b' in Table B::

{'new_table': [
  {'source': 'A', 'table': 'foo', 'index': 'a'},
  {'source': 'B', 'table': 'bar', 'index': 'b'}
]}

The joined source will now publish the "new_table" with all columns from tables "foo" and "bar" except for the index column from table "bar", which was merged with the index column "a" from table "foo".

Bases: Source

" PanelSessionSource queries the session_info endpoint of a Panel application.

Panel applications with --rest-session-info enabled can be queried about session statistics. This source makes this data available to Lumen for monitoring.

Bases: Source

RESTSource allows querying REST endpoints conforming to the Lumen REST specification.

The url must offer two endpoints, the /data endpoint must return data in a records format while the /schema endpoint must return a valid Lumen JSON schema.

Bases: MultiTypeComponent

Source components provide allow querying all kinds of data.

A Source can return one or more tables queried using the .get_tables method, a description of the data returned by each table in the form of a JSON schema accessible via the .get_schema method and lastly a .get method that allows filtering the data.

The Source base class also implements both in-memory and disk caching which can be enabled if a cache_dir is provided. Data cached to disk is stored as parquet files.

Bases: Source

WebsiteSource queries whether a website responds with a 400 status code.

Bases: BaseSQLSource

BigQuerySource provides access to a Google BigQuery project.

Bases: BaseSQLSource

DuckDBSource provides a simple wrapper around the DuckDB SQL connector.

To specify tables to be queried provide a list or dictionary of tables. A SQL expression to fetch the data from the table will then be generated using the sql_expr, e.g. if we specify a local table flights.db the default sql_expr SELECT * FROM {table} will expand that to SELECT * FROM flights.db. If you want to specify a full SQL expression as a table you must change the sql_expr to '{table}' ensuring no further templating is applied.

Note that certain functionality in DuckDB requires modules to be loaded before making a query. These can be specified using the initializers parameter, providing the ability to define DuckDb statements to be run when initializing the connection.

Bases: Source

Bases: IntakeBaseSource

An IntakeSource loads data from an Intake catalog.

Intake is a lightweight set of tools for loading and sharing data in data science projects using convenient catalog specifications.

The IntakeSource can be given a dictionary catalog specification OR a URI pointing to a catalog.yaml file on disk.

Bases: IntakeBaseSQLSource

Base class with common parameters for Dremio sources.

Bases: IntakeBaseDremioSource

IntakeDremioSQLSource allows querying a subset of Dremio catalog tables and views via custom SQL expressions.

When provided with the uri of the Dremio server and credentials to authenticate with the Dremio instance, unlike IntakeDremioSource, only the tables specified in the tables parameter can be used.

Requires the intake-dremio package to be installed.

Bases: IntakeBaseDremioSource

IntakeDremioSource allows querying Dremio catalog tables and views.

When provided with the uri of the Dremio server and credentials to authenticate with the Dremio instance all available tables can be queried via this Source.

Requires the intake-dremio package to be installed.

Bases: BaseSQLSource, IntakeBaseSource

Bases: IntakeBaseSQLSource, IntakeSource

IntakeSQLSource extends the IntakeSource with support for SQL data.

In addition to the standard intake support for reading catalogs the IntakeSQLSource computes the schema by querying the database instead of loading all the data into memory and allows for SQLTransform to be applied when querying the SQL database.

Bases: Source

PrometheusSource allows querying Prometheus PromQL endpoints.

The PrometheusSource is configured to return timeseries about CPU, memory and network usage as well as restarts for a list of Kubernetes pods specified by ID.

Bases: BaseSQLSource

SnowflakeSource uses the snowflake-python-connector library to load data from Snowflake.

Bases: BaseSQLSource

SQLAlchemySource uses SQLAlchemy to connect to various SQL databases.

Supports both synchronous and asynchronous database drivers through SQLAlchemy's engine system. Can connect to PostgreSQL, MySQL, SQLite, Oracle, MSSQL, and more.