GitHub/ReAgent
1
0
Fork 0
mirror of https://github.com/facebookresearch/ReAgent.git synced 2026-06-16 12:44:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
gh-pages
ReAgent/api/reagent.models.html
T
Zhengxing Chen f2d2a25aff Update docs 11/20/2021
2021-11-20 20:47:01 -08:00

2855 lines
375 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>reagent.models package &mdash; ReAgent 1.0 documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="reagent.prediction package" href="reagent.prediction.html" />
<link rel="prev" title="reagent.optimizer package" href="reagent.optimizer.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home"> ReAgent
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../usage.html">Usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rasp_tutorial.html">RASP (Not Actively Maintained)</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Advanced Topics</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../continuous_integration.html">Continuous Integration</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Package Reference</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="reagent.core.html">Core</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.data.html">Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.gym.html">Gym</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.evaluation.html">Evaluation</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.lite.html">Lite</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.mab.html">MAB</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.model_managers.html">Model Managers</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.model_utils.html">Model Utils</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.net_builder.html">Net Builders</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.optimizer.html">Optimizers</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Models</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#submodules">Submodules</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.actor">reagent.models.actor module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.base">reagent.models.base module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.bcq">reagent.models.bcq module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.categorical_dqn">reagent.models.categorical_dqn module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.cem_planner">reagent.models.cem_planner module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.containers">reagent.models.containers module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.convolutional_network">reagent.models.convolutional_network module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.critic">reagent.models.critic module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.dqn">reagent.models.dqn module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.dueling_q_network">reagent.models.dueling_q_network module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.embedding_bag_concat">reagent.models.embedding_bag_concat module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.fully_connected_network">reagent.models.fully_connected_network module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.linear_regression">reagent.models.linear_regression module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.mdn_rnn">reagent.models.mdn_rnn module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.mlp_scorer">reagent.models.mlp_scorer module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.model_feature_config_provider">reagent.models.model_feature_config_provider module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.no_soft_update_embedding">reagent.models.no_soft_update_embedding module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.seq2reward_model">reagent.models.seq2reward_model module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.seq2slate">reagent.models.seq2slate module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.seq2slate_reward">reagent.models.seq2slate_reward module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.synthetic_reward">reagent.models.synthetic_reward module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models.world_model">reagent.models.world_model module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#module-reagent.models">Module contents</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="reagent.prediction.html">Prediction</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.preprocessing.html">Preprocessing</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.training.html">Training</a></li>
<li class="toctree-l1"><a class="reference internal" href="reagent.workflow.html">Workflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="modules.html">All Modules</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Others</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="https://github.com/facebookresearch/ReAgent">Github</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">ReAgent</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
<li>reagent.models package</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/api/reagent.models.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="reagent-models-package">
<h1>reagent.models package<a class="headerlink" href="#reagent-models-package" title="Permalink to this headline"></a></h1>
<section id="submodules">
<h2>Submodules<a class="headerlink" href="#submodules" title="Permalink to this headline"></a></h2>
</section>
<section id="module-reagent.models.actor">
<span id="reagent-models-actor-module"></span><h2>reagent.models.actor module<a class="headerlink" href="#module-reagent.models.actor" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.actor.</span></span><span class="sig-name descname"><span class="pre">DirichletFullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor.EPSILON">
<span class="sig-name descname"><span class="pre">EPSILON</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">1e-06</span></em><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor.EPSILON" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor.get_log_prob">
<span class="sig-name descname"><span class="pre">get_log_prob</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor.get_log_prob" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.actor.DirichletFullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.actor.DirichletFullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.actor.FullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.actor.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'tanh'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">exploration_variance</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.FullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.FullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="reagent.core.html#reagent.core.types.ActorOutput" title="reagent.core.types.ActorOutput"><span class="pre">reagent.core.types.ActorOutput</span></a></span></span><a class="headerlink" href="#reagent.models.actor.FullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.FullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.FullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.actor.FullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.actor.FullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.actor.GaussianFullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.actor.</span></span><span class="sig-name descname"><span class="pre">GaussianFullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scale</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.05</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_l2_normalization</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.GaussianFullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.GaussianFullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.GaussianFullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.GaussianFullyConnectedActor.get_log_prob">
<span class="sig-name descname"><span class="pre">get_log_prob</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">squashed_action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.GaussianFullyConnectedActor.get_log_prob" title="Permalink to this definition"></a></dt>
<dd><p>Action is expected to be squashed with tanh</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.GaussianFullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.GaussianFullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.actor.GaussianFullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.actor.GaussianFullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.actor.StochasticActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.actor.</span></span><span class="sig-name descname"><span class="pre">StochasticActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">scorer</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sampler</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.StochasticActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.StochasticActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.StochasticActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.StochasticActor.get_distributed_data_parallel_model">
<span class="sig-name descname"><span class="pre">get_distributed_data_parallel_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.StochasticActor.get_distributed_data_parallel_model" title="Permalink to this definition"></a></dt>
<dd><p>Return DistributedDataParallel version of this model</p>
<p>This needs to be implemented explicitly because:
1) Model with EmbeddingBag module is not compatible with vanilla DistributedDataParallel
2) Exporting logic needs structured data. DistributedDataParallel doesn’t work with structured data.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.actor.StochasticActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.actor.StochasticActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.actor.StochasticActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.actor.StochasticActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.base">
<span id="reagent-models-base-module"></span><h2>reagent.models.base module<a class="headerlink" href="#module-reagent.models.base" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.base.</span></span><span class="sig-name descname"><span class="pre">ModelBase</span></span><a class="headerlink" href="#reagent.models.base.ModelBase" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>A base class to support exporting through ONNX</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.cpu_model">
<span class="sig-name descname"><span class="pre">cpu_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.base.ModelBase.cpu_model" title="Permalink to this definition"></a></dt>
<dd><p>Override this in DistributedDataParallel models</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.feature_config">
<span class="sig-name descname"><span class="pre">feature_config</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#reagent.models.base.ModelBase.feature_config" title="Permalink to this definition"></a></dt>
<dd><p>If the model needs additional preprocessing, e.g., using sequence features,
returns the config here.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.get_distributed_data_parallel_model">
<span class="sig-name descname"><span class="pre">get_distributed_data_parallel_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.base.ModelBase.get_distributed_data_parallel_model" title="Permalink to this definition"></a></dt>
<dd><p>Return DistributedDataParallel version of this model</p>
<p>This needs to be implemented explicitly because:
1) Model with EmbeddingBag module is not compatible with vanilla DistributedDataParallel
2) Exporting logic needs structured data. DistributedDataParallel doesn’t work with structured data.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.get_target_network">
<span class="sig-name descname"><span class="pre">get_target_network</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.base.ModelBase.get_target_network" title="Permalink to this definition"></a></dt>
<dd><p>Return a copy of this network to be used as target network</p>
<p>Subclass should override this if the target network should share parameters
with the network to be trained.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Any</span></span></span><a class="headerlink" href="#reagent.models.base.ModelBase.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.base.ModelBase.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.base.ModelBase.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.bcq">
<span id="reagent-models-bcq-module"></span><h2>reagent.models.bcq module<a class="headerlink" href="#module-reagent.models.bcq" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.bcq.BatchConstrainedDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.bcq.</span></span><span class="sig-name descname"><span class="pre">BatchConstrainedDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">q_network</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">imitator_network</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">bcq_drop_threshold</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.bcq.BatchConstrainedDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.bcq.BatchConstrainedDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.bcq.BatchConstrainedDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.bcq.BatchConstrainedDQN.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.bcq.BatchConstrainedDQN.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.bcq.BatchConstrainedDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.bcq.BatchConstrainedDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.categorical_dqn">
<span id="reagent-models-categorical-dqn-module"></span><h2>reagent.models.categorical_dqn module<a class="headerlink" href="#module-reagent.models.categorical_dqn" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.categorical_dqn.CategoricalDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.categorical_dqn.</span></span><span class="sig-name descname"><span class="pre">CategoricalDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">distributional_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">qmin</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">qmax</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.categorical_dqn.CategoricalDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.categorical_dqn.CategoricalDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.categorical_dqn.CategoricalDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.categorical_dqn.CategoricalDQN.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.categorical_dqn.CategoricalDQN.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.categorical_dqn.CategoricalDQN.log_dist">
<span class="sig-name descname"><span class="pre">log_dist</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.categorical_dqn.CategoricalDQN.log_dist" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.categorical_dqn.CategoricalDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.categorical_dqn.CategoricalDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.cem_planner">
<span id="reagent-models-cem-planner-module"></span><h2>reagent.models.cem_planner module<a class="headerlink" href="#module-reagent.models.cem_planner" title="Permalink to this headline"></a></h2>
<p>A network which implements a cross entropy method-based planner</p>
<p>The planner plans the best next action based on simulation data generated by
an ensemble of world models.</p>
<p>The idea is inspired by: <a class="reference external" href="https://arxiv.org/abs/1805.12114">https://arxiv.org/abs/1805.12114</a></p>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.cem_planner.</span></span><span class="sig-name descname"><span class="pre">CEMPlannerNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mem_net_list</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#reagent.models.world_model.MemoryNetwork" title="reagent.models.world_model.MemoryNetwork"><span class="pre">reagent.models.world_model.MemoryNetwork</span></a><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cem_num_iterations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cem_population_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ensemble_population_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_elites</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">plan_horizon_length</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">discrete_action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">terminal_effective</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">gamma</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">alpha</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.25</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epsilon</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.001</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_upper_bounds</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">numpy.ndarray</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_lower_bounds</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">numpy.ndarray</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.acc_rewards_of_all_solutions">
<span class="sig-name descname"><span class="pre">acc_rewards_of_all_solutions</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">solutions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">float</span></span></span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.acc_rewards_of_all_solutions" title="Permalink to this definition"></a></dt>
<dd><p>Calculate accumulated rewards of solutions.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>state</strong> – the input which contains the starting state</p></li>
<li><p><strong>solutions</strong> – its shape is (cem_pop_size, plan_horizon_length, action_dim)</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>a vector of size cem_pop_size, which is the reward of each solution</p>
</dd>
</dl>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.acc_rewards_of_one_solution">
<span class="sig-name descname"><span class="pre">acc_rewards_of_one_solution</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">init_state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">solution</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">solution_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.acc_rewards_of_one_solution" title="Permalink to this definition"></a></dt>
<dd><p>ensemble_pop_size trajectories will be sampled to evaluate a
CEM solution. Each trajectory is generated by one world model</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>init_state</strong> – its shape is (state_dim, )</p></li>
<li><p><strong>solution</strong> – its shape is (plan_horizon_length, action_dim)</p></li>
<li><p><strong>solution_idx</strong> – the index of the solution</p></li>
</ul>
</dd>
<dt class="field-even">Return reward</dt>
<dd class="field-even"><p>Reward of each of ensemble_pop_size trajectories</p>
</dd>
</dl>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.constrained_variance">
<span class="sig-name descname"><span class="pre">constrained_variance</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mean</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">var</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.constrained_variance" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.continuous_planning">
<span class="sig-name descname"><span class="pre">continuous_planning</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.continuous_planning" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.discrete_planning">
<span class="sig-name descname"><span class="pre">discrete_planning</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">numpy.ndarray</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.discrete_planning" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.sample_reward_next_state_terminal">
<span class="sig-name descname"><span class="pre">sample_reward_next_state_terminal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">mem_net</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.world_model.MemoryNetwork" title="reagent.models.world_model.MemoryNetwork"><span class="pre">reagent.models.world_model.MemoryNetwork</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.sample_reward_next_state_terminal" title="Permalink to this definition"></a></dt>
<dd><p>Sample one-step dynamics based on the provided world model</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.cem_planner.CEMPlannerNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.cem_planner.CEMPlannerNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.containers">
<span id="reagent-models-containers-module"></span><h2>reagent.models.containers module<a class="headerlink" href="#module-reagent.models.containers" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.containers.Sequential">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.containers.</span></span><span class="sig-name descname"><span class="pre">Sequential</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.containers.Sequential" title="Permalink to this definition"></a></dt>
<dt class="sig sig-object py">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.containers.</span></span><span class="sig-name descname"><span class="pre">Sequential</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.OrderedDict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">torch.nn.modules.module.Module</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.container.Sequential</span></code>, <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Used this instead of torch.nn.Sequential to automate model tracing</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.containers.Sequential.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.containers.Sequential.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.convolutional_network">
<span id="reagent-models-convolutional-network-module"></span><h2>reagent.models.convolutional_network module<a class="headerlink" href="#module-reagent.models.convolutional_network" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.convolutional_network.ConvolutionalNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.convolutional_network.</span></span><span class="sig-name descname"><span class="pre">ConvolutionalNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">cnn_parameters</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layers</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.convolutional_network.ConvolutionalNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.convolutional_network.ConvolutionalNetwork.conv_forward">
<span class="sig-name descname"><span class="pre">conv_forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.convolutional_network.ConvolutionalNetwork.conv_forward" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.convolutional_network.ConvolutionalNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.FloatTensor</span></span></span><a class="headerlink" href="#reagent.models.convolutional_network.ConvolutionalNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass for generic convnet DNNs. Assumes activation names
are valid pytorch activation names.
:param input image tensor</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.convolutional_network.ConvolutionalNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.convolutional_network.ConvolutionalNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.critic">
<span id="reagent-models-critic-module"></span><h2>reagent.models.critic module<a class="headerlink" href="#module-reagent.models.critic" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.critic.FullyConnectedCritic">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.critic.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedCritic</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.critic.FullyConnectedCritic" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.critic.FullyConnectedCritic.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.critic.FullyConnectedCritic.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.critic.FullyConnectedCritic.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.critic.FullyConnectedCritic.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.critic.FullyConnectedCritic.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.critic.FullyConnectedCritic.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.dqn">
<span id="reagent-models-dqn-module"></span><h2>reagent.models.dqn module<a class="headerlink" href="#module-reagent.models.dqn" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.dqn.FullyConnectedDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.dqn.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'linear'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout_ratio</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">normalized_output</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dqn.FullyConnectedDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected" title="reagent.models.fully_connected_network.FloatFeatureFullyConnected"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.fully_connected_network.FloatFeatureFullyConnected</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dqn.FullyConnectedDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">possible_actions_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.dqn.FullyConnectedDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.dqn.FullyConnectedDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.dqn.FullyConnectedDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.dueling_q_network">
<span id="reagent-models-dueling-q-network-module"></span><h2>reagent.models.dueling_q_network module<a class="headerlink" href="#module-reagent.models.dueling_q_network" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.DuelingQNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.dueling_q_network.</span></span><span class="sig-name descname"><span class="pre">DuelingQNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">shared_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">advantage_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">value_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.DuelingQNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.DuelingQNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">possible_actions_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.dueling_q_network.DuelingQNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.DuelingQNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.DuelingQNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.DuelingQNetwork.make_fully_connected">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">make_fully_connected</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.DuelingQNetwork.make_fully_connected" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.DuelingQNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.dueling_q_network.DuelingQNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.ParametricDuelingQNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.dueling_q_network.</span></span><span class="sig-name descname"><span class="pre">ParametricDuelingQNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">shared_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">advantage_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">value_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.ParametricDuelingQNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.ParametricDuelingQNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.dueling_q_network.ParametricDuelingQNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.ParametricDuelingQNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.ParametricDuelingQNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.ParametricDuelingQNetwork.make_fully_connected">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">make_fully_connected</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.dueling_q_network.ParametricDuelingQNetwork.make_fully_connected" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.dueling_q_network.ParametricDuelingQNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.dueling_q_network.ParametricDuelingQNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.embedding_bag_concat">
<span id="reagent-models-embedding-bag-concat-module"></span><h2>reagent.models.embedding_bag_concat module<a class="headerlink" href="#module-reagent.models.embedding_bag_concat" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.embedding_bag_concat.EmbeddingBagConcat">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.embedding_bag_concat.</span></span><span class="sig-name descname"><span class="pre">EmbeddingBagConcat</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">model_feature_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">embedding_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.embedding_bag_concat.EmbeddingBagConcat" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Concatenating embedding with float features before passing the input
to DQN</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.embedding_bag_concat.EmbeddingBagConcat.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.embedding_bag_concat.EmbeddingBagConcat.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.embedding_bag_concat.EmbeddingBagConcat.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.embedding_bag_concat.EmbeddingBagConcat.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py property">
<dt class="sig sig-object py" id="reagent.models.embedding_bag_concat.EmbeddingBagConcat.output_dim">
<em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">output_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.embedding_bag_concat.EmbeddingBagConcat.output_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.embedding_bag_concat.EmbeddingBagConcat.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.embedding_bag_concat.EmbeddingBagConcat.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.fully_connected_network">
<span id="reagent-models-fully-connected-network-module"></span><h2>reagent.models.fully_connected_network module<a class="headerlink" href="#module-reagent.models.fully_connected_network" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FloatFeatureFullyConnected">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.fully_connected_network.</span></span><span class="sig-name descname"><span class="pre">FloatFeatureFullyConnected</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'linear'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout_ratio</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">normalized_output</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>A fully connected network that takes FloatFeatures input
and supports distributional prediction.</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FloatFeatureFullyConnected.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FloatFeatureFullyConnected.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FloatFeatureFullyConnected.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FullyConnectedNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.fully_connected_network.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">min_std</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout_ratio</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">normalize_output</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">orthogonal_init</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.FullyConnectedNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FullyConnectedNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.fully_connected_network.FullyConnectedNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass for generic feed-forward DNNs. Assumes activation names
are valid pytorch activation names.
:param input tensor</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FullyConnectedNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.FullyConnectedNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.FullyConnectedNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.fully_connected_network.FullyConnectedNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.SlateBatchNorm1d">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.fully_connected_network.</span></span><span class="sig-name descname"><span class="pre">SlateBatchNorm1d</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.SlateBatchNorm1d" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Same as nn.BatchNorm1d is input has shape (batch_size, feat_dim).
But if input has shape (batch_size, num_candidates, item_feats), like in LearnedVM,
we transpose it, since that’s what nn.BatchNorm1d computes Batch Normalization over
1st dimension, while we want to compute it over item_feats.</p>
<p>NOTE: this is different from nn.BatchNorm2d which is for CNNs, and expects 4D inputs</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.SlateBatchNorm1d.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.fully_connected_network.SlateBatchNorm1d.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.SlateBatchNorm1d.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.fully_connected_network.SlateBatchNorm1d.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="reagent.models.fully_connected_network.gaussian_fill_w_gain">
<span class="sig-prename descclassname"><span class="pre">reagent.models.fully_connected_network.</span></span><span class="sig-name descname"><span class="pre">gaussian_fill_w_gain</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">gain</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_in</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">min_std</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.0</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="headerlink" href="#reagent.models.fully_connected_network.gaussian_fill_w_gain" title="Permalink to this definition"></a></dt>
<dd><p>Gaussian initialization with gain.</p>
</dd></dl>
</section>
<section id="module-reagent.models.linear_regression">
<span id="reagent-models-linear-regression-module"></span><h2>reagent.models.linear_regression module<a class="headerlink" href="#module-reagent.models.linear_regression" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.linear_regression.LinearRegressionUCB">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.linear_regression.</span></span><span class="sig-name descname"><span class="pre">LinearRegressionUCB</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">l2_reg_lambda</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">predict_ucb</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ucb_alpha</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1.0</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.linear_regression.LinearRegressionUCB" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>A linear regression model for LinUCB.
Note that instead of being trained by a PyTorch optimizer, we explicitly</p>
<blockquote>
<div><p>update attributes A and b (according to the LinUCB formulas implemented in
reagent.training.cb.linucb_trainer.LinUCBTrainer).</p>
</div></blockquote>
<dl class="simple">
<dt>Since computing the regression coefficients inverse matrix inversion (expensive op), we</dt><dd><p>save time by only computing the coefficients when necessary (when doing inference).</p>
</dd>
</dl>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>input_dim</strong> – Dimension of input data</p></li>
<li><p><strong>l2_reg_lambda</strong> – The weight on L2 regularization</p></li>
<li><p><strong>predict_ucb</strong> – If True, the model outputs an Upper Confidence Bound (UCB).
If False, the model outputs the point estimate</p></li>
<li><p><strong>ucb_alpha</strong> – The coefficient on the standard deviation in UCB formula.
Only used if predict_ucb=True.</p></li>
</ul>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.linear_regression.LinearRegressionUCB.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">inp</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ucb_alpha</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.linear_regression.LinearRegressionUCB.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward can return the mean or a UCB. If returning UCB, the CI width is stddev*ucb_alpha
If ucb_alpha is not passed in, a fixed alpha from init is used</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.linear_regression.LinearRegressionUCB.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.linear_regression.LinearRegressionUCB.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.linear_regression.LinearRegressionUCB.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.linear_regression.LinearRegressionUCB.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="reagent.models.linear_regression.batch_quadratic_form">
<span class="sig-prename descclassname"><span class="pre">reagent.models.linear_regression.</span></span><span class="sig-name descname"><span class="pre">batch_quadratic_form</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">A</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.linear_regression.batch_quadratic_form" title="Permalink to this definition"></a></dt>
<dd><p>Compute the quadratic form x^T * A * x for a batched input x.
Inspired by <a class="reference external" href="https://stackoverflow.com/questions/18541851/calculate-vt-a-v-for-a-matrix-of-vectors-v">https://stackoverflow.com/questions/18541851/calculate-vt-a-v-for-a-matrix-of-vectors-v</a>
This is a vectorized implementation of out[i] = x[i].t() &#64; A &#64; x[i]
x shape: (B, N)
A shape: (N, N)
output shape: (B)</p>
</dd></dl>
</section>
<section id="module-reagent.models.mdn_rnn">
<span id="reagent-models-mdn-rnn-module"></span><h2>reagent.models.mdn_rnn module<a class="headerlink" href="#module-reagent.models.mdn_rnn" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.mdn_rnn.</span></span><span class="sig-name descname"><span class="pre">MDNRNN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hiddens</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hidden_layers</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_gaussians</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Mixture Density Network - Recurrent Neural Network</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">actions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">states</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hidden</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass of MDN-RNN</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>actions</strong> – (SEQ_LEN, BATCH_SIZE, ACTION_DIM) torch tensor</p></li>
<li><p><strong>states</strong> – (SEQ_LEN, BATCH_SIZE, STATE_DIM) torch tensor</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>parameters of the GMM prediction for the next state,</p>
</dd>
</dl>
<p>gaussian prediction of the reward and logit prediction of
non-terminality. And the RNN’s outputs.</p>
<blockquote>
<div><ul class="simple">
<li><p>mus: (SEQ_LEN, BATCH_SIZE, NUM_GAUSSIANS, STATE_DIM) torch tensor</p></li>
<li><p>sigmas: (SEQ_LEN, BATCH_SIZE, NUM_GAUSSIANS, STATE_DIM) torch tensor</p></li>
<li><p>logpi: (SEQ_LEN, BATCH_SIZE, NUM_GAUSSIANS) torch tensor</p></li>
<li><p>reward: (SEQ_LEN, BATCH_SIZE) torch tensor</p></li>
<li><p>not_terminal: (SEQ_LEN, BATCH_SIZE) torch tensor</p></li>
<li><dl class="simple">
<dt>last_step_hidden_and_cell: TUPLE(</dt><dd><p>(NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE),
(NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE)</p>
</dd>
</dl>
</li>
</ul>
<p>) torch tensor
- all_steps_hidden: (SEQ_LEN, BATCH_SIZE, HIDDEN_SIZE) torch tensor</p>
</div></blockquote>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNN.get_initial_hidden_state">
<span class="sig-name descname"><span class="pre">get_initial_hidden_state</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">batch_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNN.get_initial_hidden_state" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemoryPool">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.mdn_rnn.</span></span><span class="sig-name descname"><span class="pre">MDNRNNMemoryPool</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">max_replay_memory_size</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemoryPool" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemoryPool.deque_sample">
<span class="sig-name descname"><span class="pre">deque_sample</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">indices</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemoryPool.deque_sample" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemoryPool.insert_into_memory">
<span class="sig-name descname"><span class="pre">insert_into_memory</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">next_state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">reward</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">not_terminal</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemoryPool.insert_into_memory" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py property">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemoryPool.memory_size">
<em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">memory_size</span></span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemoryPool.memory_size" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemoryPool.sample_memories">
<span class="sig-name descname"><span class="pre">sample_memories</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">batch_size</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_gpu</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="reagent.core.html#reagent.core.types.MemoryNetworkInput" title="reagent.core.types.MemoryNetworkInput"><span class="pre">reagent.core.types.MemoryNetworkInput</span></a></span></span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemoryPool.sample_memories" title="Permalink to this definition"></a></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>batch_size</strong> – number of samples to return</p></li>
<li><p><strong>use_gpu</strong> – whether to put samples on gpu</p></li>
</ul>
</dd>
</dl>
<p>State’s shape is SEQ_LEN x BATCH_SIZE x STATE_DIM, for example.
By default, MDN-RNN consumes data with SEQ_LEN as the first dimension.</p>
</dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.mdn_rnn.</span></span><span class="sig-name descname"><span class="pre">MDNRNNMemorySample</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">next_state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">reward</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">not_terminal</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">NamedTuple</span></code></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample.action">
<span class="sig-name descname"><span class="pre">action</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">numpy.ndarray</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample.action" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 1</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample.next_state">
<span class="sig-name descname"><span class="pre">next_state</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">numpy.ndarray</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample.next_state" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 2</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample.not_terminal">
<span class="sig-name descname"><span class="pre">not_terminal</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">float</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample.not_terminal" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 4</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample.reward">
<span class="sig-name descname"><span class="pre">reward</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">float</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample.reward" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 3</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.MDNRNNMemorySample.state">
<span class="sig-name descname"><span class="pre">state</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">numpy.ndarray</span></em><a class="headerlink" href="#reagent.models.mdn_rnn.MDNRNNMemorySample.state" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 0</p>
</dd></dl>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.gmm_loss">
<span class="sig-prename descclassname"><span class="pre">reagent.models.mdn_rnn.</span></span><span class="sig-name descname"><span class="pre">gmm_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">batch</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">mus</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sigmas</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logpi</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">reduce</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.gmm_loss" title="Permalink to this definition"></a></dt>
<dd><p>Computes the gmm loss.</p>
<p>Compute minus the log probability of batch under the GMM model described
by mus, sigmas, pi. Precisely, with bs1, bs2, … the sizes of the batch
dimensions (several batch dimension are useful when you have both a batch
axis and a time step axis), gs the number of mixtures and fs the number of
features.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>batch</strong> – (bs1, bs2, <a href="#id1"><span class="problematic" id="id2">*</span></a>, fs) torch tensor</p></li>
<li><p><strong>mus</strong> – (bs1, bs2, <a href="#id3"><span class="problematic" id="id4">*</span></a>, gs, fs) torch tensor</p></li>
<li><p><strong>sigmas</strong> – (bs1, bs2, <a href="#id5"><span class="problematic" id="id6">*</span></a>, gs, fs) torch tensor</p></li>
<li><p><strong>logpi</strong> – (bs1, bs2, <a href="#id7"><span class="problematic" id="id8">*</span></a>, gs) torch tensor</p></li>
<li><p><strong>reduce</strong> – if not reduce, the mean in the following formula is omitted</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p></p>
</dd>
</dl>
<dl class="simple">
<dt>loss(batch) = - mean_{i1=0..bs1, i2=0..bs2, …} log(</dt><dd><dl class="simple">
<dt>sum_{k=1..gs} pi[i1, i2, …, k] * N(</dt><dd><p>batch[i1, i2, …, :] | mus[i1, i2, …, k, :], sigmas[i1, i2, …, k, :]))</p>
</dd>
</dl>
</dd>
</dl>
<p>NOTE: The loss is not reduced along the feature dimension (i.e. it should
scale linearily with fs).</p>
<p>Adapted from: <a class="reference external" href="https://github.com/ctallec/world-models">https://github.com/ctallec/world-models</a></p>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="reagent.models.mdn_rnn.transpose">
<span class="sig-prename descclassname"><span class="pre">reagent.models.mdn_rnn.</span></span><span class="sig-name descname"><span class="pre">transpose</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mdn_rnn.transpose" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</section>
<section id="module-reagent.models.mlp_scorer">
<span id="reagent-models-mlp-scorer-module"></span><h2>reagent.models.mlp_scorer module<a class="headerlink" href="#module-reagent.models.mlp_scorer" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.mlp_scorer.MLPScorer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.mlp_scorer.</span></span><span class="sig-name descname"><span class="pre">MLPScorer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mlp</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_user_feat</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mlp_scorer.MLPScorer" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Log-space in and out</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mlp_scorer.MLPScorer.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">obs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mlp_scorer.MLPScorer.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.mlp_scorer.MLPScorer.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.mlp_scorer.MLPScorer.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.mlp_scorer.MLPScorer.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.mlp_scorer.MLPScorer.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.model_feature_config_provider">
<span id="reagent-models-model-feature-config-provider-module"></span><h2>reagent.models.model_feature_config_provider module<a class="headerlink" href="#module-reagent.models.model_feature_config_provider" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.model_feature_config_provider.</span></span><span class="sig-name descname"><span class="pre">ModelFeatureConfigProvider</span></span><a class="headerlink" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY">
<span class="sig-name descname"><span class="pre">REGISTRY</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">{'raw':</span> <span class="pre">&lt;class</span> <span class="pre">'reagent.models.model_feature_config_provider.RawModelFeatureConfigProvider'&gt;}</span></em><a class="headerlink" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY_FROZEN">
<span class="sig-name descname"><span class="pre">REGISTRY_FROZEN</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">True</span></em><a class="headerlink" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY_FROZEN" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY_NAME">
<span class="sig-name descname"><span class="pre">REGISTRY_NAME</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'ModelFeatureConfigProvider'</span></em><a class="headerlink" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.REGISTRY_NAME" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.get_model_feature_config">
<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">get_model_feature_config</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a></span></span><a class="headerlink" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider.get_model_feature_config" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.RawModelFeatureConfigProvider">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.model_feature_config_provider.</span></span><span class="sig-name descname"><span class="pre">RawModelFeatureConfigProvider</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">float_feature_infos:</span> <span class="pre">List[reagent.core.types.FloatFeatureInfo]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">id_mapping_config:</span> <span class="pre">Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">reagent.core.types.IdMappingUnion]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">id_list_feature_configs:</span> <span class="pre">List[reagent.core.types.IdListFeatureConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">id_score_list_feature_configs:</span> <span class="pre">List[reagent.core.types.IdScoreListFeatureConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.model_feature_config_provider.RawModelFeatureConfigProvider" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.model_feature_config_provider.ModelFeatureConfigProvider" title="reagent.models.model_feature_config_provider.ModelFeatureConfigProvider"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.model_feature_config_provider.ModelFeatureConfigProvider</span></code></a>, <a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.core.types.ModelFeatureConfig</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.model_feature_config_provider.RawModelFeatureConfigProvider.get_model_feature_config">
<span class="sig-name descname"><span class="pre">get_model_feature_config</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a></span></span><a class="headerlink" href="#reagent.models.model_feature_config_provider.RawModelFeatureConfigProvider.get_model_feature_config" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.no_soft_update_embedding">
<span id="reagent-models-no-soft-update-embedding-module"></span><h2>reagent.models.no_soft_update_embedding module<a class="headerlink" href="#module-reagent.models.no_soft_update_embedding" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.no_soft_update_embedding.</span></span><span class="sig-name descname"><span class="pre">NoSoftUpdateEmbedding</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">num_embeddings</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">embedding_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">padding_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">norm_type</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">2.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scale_grad_by_freq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sparse</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">_weight</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">device</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dtype</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.sparse.Embedding</span></code></p>
<p>Use this instead of vanilla Embedding module to avoid soft-updating the embedding
table in the target network.</p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.embedding_dim">
<span class="sig-name descname"><span class="pre">embedding_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.embedding_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.max_norm">
<span class="sig-name descname"><span class="pre">max_norm</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.max_norm" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.norm_type">
<span class="sig-name descname"><span class="pre">norm_type</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">float</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.norm_type" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.num_embeddings">
<span class="sig-name descname"><span class="pre">num_embeddings</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.num_embeddings" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.padding_idx">
<span class="sig-name descname"><span class="pre">padding_idx</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.padding_idx" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.scale_grad_by_freq">
<span class="sig-name descname"><span class="pre">scale_grad_by_freq</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.scale_grad_by_freq" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.sparse">
<span class="sig-name descname"><span class="pre">sparse</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.sparse" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.weight">
<span class="sig-name descname"><span class="pre">weight</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">torch.Tensor</span></em><a class="headerlink" href="#reagent.models.no_soft_update_embedding.NoSoftUpdateEmbedding.weight" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.seq2reward_model">
<span id="reagent-models-seq2reward-model-module"></span><h2>reagent.models.seq2reward_model module<a class="headerlink" href="#module-reagent.models.seq2reward_model" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2reward_model.Seq2RewardNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2reward_model.</span></span><span class="sig-name descname"><span class="pre">Seq2RewardNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hiddens</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hidden_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2reward_model.Seq2RewardNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2reward_model.Seq2RewardNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">valid_reward_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2reward_model.Seq2RewardNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass of Seq2Reward</p>
<p>Takes in the current state and use it as init hidden
The input sequence are pure actions only
Output the predicted reward after each time step</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>actions</strong> – (SEQ_LEN, BATCH_SIZE, ACTION_DIM) torch tensor</p></li>
<li><p><strong>states</strong> – (SEQ_LEN, BATCH_SIZE, STATE_DIM) torch tensor</p></li>
<li><p><strong>valid_reward_len</strong> – (BATCH_SIZE,) torch tensor</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>predicated accumulated rewards at last step for the given sequence
- acc_reward: (BATCH_SIZE, 1) torch tensor</p>
</dd>
</dl>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2reward_model.Seq2RewardNetwork.get_initial_hidden_state">
<span class="sig-name descname"><span class="pre">get_initial_hidden_state</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2reward_model.Seq2RewardNetwork.get_initial_hidden_state" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2reward_model.Seq2RewardNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2reward_model.Seq2RewardNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2reward_model.Seq2RewardNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2reward_model.Seq2RewardNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.seq2slate">
<span id="reagent-models-seq2slate-module"></span><h2>reagent.models.seq2slate module<a class="headerlink" href="#module-reagent.models.seq2slate" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.BaselineNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">BaselineNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.BaselineNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.BaselineNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.PreprocessedRankingInput" title="reagent.core.types.PreprocessedRankingInput"><span class="pre">reagent.core.types.PreprocessedRankingInput</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.BaselineNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.BaselineNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.BaselineNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Decoder">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Decoder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layer</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Decoder" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Generic num_layers layer decoder with masking.</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Decoder.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">memory</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_src_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_tgt_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Decoder.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Decoder.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.Decoder.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLastLayerPytorch">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">DecoderLastLayerPytorch</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">d_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">nhead</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward=2048</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout=0.1</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activation=&lt;function</span> <span class="pre">relu&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layer_norm_eps=1e-05</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch_first=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">norm_first=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">device=None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dtype=None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderLastLayerPytorch" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.transformer.TransformerDecoderLayer</span></code></p>
<p>The last layer of Decoder.
Modified from PyTorch official code: instead of attention embedding,
return attention weights which can be directly used to sample items</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLastLayerPytorch.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">tgt</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">memory</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">memory_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderLastLayerPytorch.forward" title="Permalink to this definition"></a></dt>
<dd><p>Pass the inputs (and mask) through the decoder layer.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tgt</strong> – the sequence to the decoder layer (required).</p></li>
<li><p><strong>memory</strong> – the sequence from the last layer of the encoder (required).</p></li>
<li><p><strong>tgt_mask</strong> – the mask for the tgt sequence (optional).</p></li>
<li><p><strong>memory_mask</strong> – the mask for the memory sequence (optional).</p></li>
<li><p><strong>tgt_key_padding_mask</strong> – the mask for the tgt keys per batch (optional).</p></li>
<li><p><strong>memory_key_padding_mask</strong> – the mask for the memory keys per batch (optional).</p></li>
</ul>
</dd>
</dl>
<dl class="simple">
<dt>Shape:</dt><dd><p>see the docs in Transformer class.</p>
</dd>
</dl>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLastLayerPytorch.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.DecoderLastLayerPytorch.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLayer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">DecoderLayer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">size</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">self_attn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_attn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">feed_forward</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderLayer" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Decoder is made of self-attn, src-attn, and feed forward</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLayer.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">m</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_src_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_tgt_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderLayer.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderLayer.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.DecoderLayer.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderPyTorch">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">DecoderPyTorch</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_heads</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderPyTorch" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Transformer-based decoder based on PyTorch official implementation</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderPyTorch.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">tgt_embed</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">memory</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_src_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_tgt_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.DecoderPyTorch.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.DecoderPyTorch.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.DecoderPyTorch.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Embedder">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Embedder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_in</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_out</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Embedder" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Embedder.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Embedder.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Embedder.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.Embedder.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Encoder">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Encoder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layer</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Encoder" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Core encoder is a stack of num_layers layers</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Encoder.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Encoder.forward" title="Permalink to this definition"></a></dt>
<dd><p>Pass the input (and mask) through each layer in turn.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Encoder.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.Encoder.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderLayer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">EncoderLayer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">self_attn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">feed_forward</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.EncoderLayer" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Encoder is made up of self-attn and feed forward</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderLayer.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">src_embed</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.EncoderLayer.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderLayer.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.EncoderLayer.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderPyTorch">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">EncoderPyTorch</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_heads</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.EncoderPyTorch" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Transformer-based encoder based on PyTorch official implementation</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderPyTorch.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">src</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.EncoderPyTorch.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.EncoderPyTorch.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.EncoderPyTorch.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Generator">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Generator</span></span><a class="headerlink" href="#reagent.models.seq2slate.Generator" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>Candidate generation</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Generator.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">probs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">greedy</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Generator.forward" title="Permalink to this definition"></a></dt>
<dd><p>Decode one-step</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>probs</strong> – probability distributions of decoder.
Shape: batch_size, tgt_seq_len, candidate_size</p></li>
<li><p><strong>greedy</strong> – whether to greedily pick or sample the next symbol</p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Generator.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.Generator.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.MultiHeadedAttention">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">MultiHeadedAttention</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">num_heads</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.MultiHeadedAttention" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.MultiHeadedAttention.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">query</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">key</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">value</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">mask</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.MultiHeadedAttention.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.MultiHeadedAttention.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.MultiHeadedAttention.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionalEncoding">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">PositionalEncoding</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.PositionalEncoding" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>A special, non-learnable positional encoding for handling variable (possibly longer)
lengths of inputs. We simply add an ordinal number as an additional dimension for
the input embeddings, and then project them back to the original number of dimensions</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionalEncoding.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.PositionalEncoding.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionalEncoding.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.PositionalEncoding.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionwiseFeedForward">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">PositionwiseFeedForward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.PositionwiseFeedForward" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionwiseFeedForward.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.PositionwiseFeedForward.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.PositionwiseFeedForward.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.PositionwiseFeedForward.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_arch</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.model_utils.html#reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch" title="reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch"><span class="pre">reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">temperature</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.candidate_dim">
<span class="sig-name descname"><span class="pre">candidate_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.candidate_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.dim_model">
<span class="sig-name descname"><span class="pre">dim_model</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.dim_model" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.PreprocessedRankingInput" title="reagent.core.types.PreprocessedRankingInput"><span class="pre">reagent.core.types.PreprocessedRankingInput</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">mode</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.model_utils.html#reagent.model_utils.seq2slate_utils.Seq2SlateMode" title="reagent.model_utils.seq2slate_utils.Seq2SlateMode"><span class="pre">reagent.model_utils.seq2slate_utils.Seq2SlateMode</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">greedy</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">bool</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.get_distributed_data_parallel_model">
<span class="sig-name descname"><span class="pre">get_distributed_data_parallel_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.get_distributed_data_parallel_model" title="Permalink to this definition"></a></dt>
<dd><p>Return DistributedDataParallel version of this model</p>
<p>This needs to be implemented explicitly because:
1) Model with EmbeddingBag module is not compatible with vanilla DistributedDataParallel
2) Exporting logic needs structured data. DistributedDataParallel doesn’t work with structured data.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.max_src_seq_len">
<span class="sig-name descname"><span class="pre">max_src_seq_len</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.max_src_seq_len" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.max_tgt_seq_len">
<span class="sig-name descname"><span class="pre">max_tgt_seq_len</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.max_tgt_seq_len" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.num_stacked_layers">
<span class="sig-name descname"><span class="pre">num_stacked_layers</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.num_stacked_layers" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.output_arch">
<span class="sig-name descname"><span class="pre">output_arch</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><a class="reference internal" href="reagent.model_utils.html#reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch" title="reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch"><span class="pre">reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch</span></a></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.output_arch" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.state_dim">
<span class="sig-name descname"><span class="pre">state_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.state_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateNet.temperature">
<span class="sig-name descname"><span class="pre">temperature</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">float</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateNet.temperature" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateTransformerModel</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_heads</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_arch</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.model_utils.html#reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch" title="reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch"><span class="pre">reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">temperature</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state_embed_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>A Seq2Slate network with Transformer. The network is essentially an
encoder-decoder structure. The encoder inputs a sequence of candidate feature
vectors and a state feature vector, and the decoder outputs an ordered
list of candidate indices. The output order is learned through REINFORCE
algorithm to optimize sequence-wise reward.</p>
<p>One application example is to rank candidate feeds to a specific user such
that the final list of feeds as a whole optimizes the user’s engagement.</p>
<p>Seq2Slate paper: <a class="reference external" href="https://arxiv.org/abs/1810.02019">https://arxiv.org/abs/1810.02019</a>
Transformer paper: <a class="reference external" href="https://arxiv.org/abs/1706.03762">https://arxiv.org/abs/1706.03762</a></p>
<p>The model archtecture can also adapt to some variations.
(1) The decoder can be autoregressive
(2) The decoder can take encoder scores and perform iterative softmax (aka frechet sort)
(3) No decoder and the output order is solely based on encoder scores</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel.decode">
<span class="sig-name descname"><span class="pre">decode</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">memory</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_idx</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_seq</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel.decode" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel.encode">
<span class="sig-name descname"><span class="pre">encode</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel.encode" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel.encoder_output_to_scores">
<span class="sig-name descname"><span class="pre">encoder_output_to_scores</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput" title="reagent.models.seq2slate.Seq2SlateTransformerOutput"><span class="pre">reagent.models.seq2slate.Seq2SlateTransformerOutput</span></a></span></span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel.encoder_output_to_scores" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mode</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">greedy</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">bool</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput" title="reagent.models.seq2slate.Seq2SlateTransformerOutput"><span class="pre">reagent.models.seq2slate.Seq2SlateTransformerOutput</span></a></span></span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel.forward" title="Permalink to this definition"></a></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>input</strong> – model input</p></li>
<li><p><strong>mode</strong> – <p>a string indicating which mode to perform.
“rank”: return ranked actions and their generative probabilities.
“per_seq_log_probs”: return generative log probabilities of given</p>
<blockquote>
<div><p>tgt sequences (used for REINFORCE training)</p>
</div></blockquote>
<dl class="simple">
<dt>”per_symbol_log_probs”: return generative log probabilties of each</dt><dd><p>symbol in given tgt sequences (used in TEACHER FORCING training)</p>
</dd>
</dl>
</p></li>
<li><p><strong>tgt_seq_len</strong> – the length of output sequence to be decoded. Only used
in rank mode</p></li>
<li><p><strong>greedy</strong> – whether to sample based on softmax distribution or greedily
when decoding. Only used in rank mode</p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerModel.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerModel.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateTransformerNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_arch</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.model_utils.html#reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch" title="reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch"><span class="pre">reagent.model_utils.seq2slate_utils.Seq2SlateOutputArch</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">temperature</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_heads</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state_embed_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.seq2slate.Seq2SlateNet" title="reagent.models.seq2slate.Seq2SlateNet"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.seq2slate.Seq2SlateNet</span></code></a></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerNet.dim_feedforward">
<span class="sig-name descname"><span class="pre">dim_feedforward</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerNet.dim_feedforward" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerNet.num_heads">
<span class="sig-name descname"><span class="pre">num_heads</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerNet.num_heads" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerNet.state_embed_dim">
<span class="sig-name descname"><span class="pre">state_embed_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></em><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerNet.state_embed_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateTransformerOutput</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ranked_per_symbol_probs</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ranked_per_seq_probs</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ranked_tgt_out_idx</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">per_symbol_log_probs</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">per_seq_log_probs</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">encoder_scores</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">NamedTuple</span></code></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.encoder_scores">
<span class="sig-name descname"><span class="pre">encoder_scores</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.encoder_scores" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 5</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.per_seq_log_probs">
<span class="sig-name descname"><span class="pre">per_seq_log_probs</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.per_seq_log_probs" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 4</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.per_symbol_log_probs">
<span class="sig-name descname"><span class="pre">per_symbol_log_probs</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.per_symbol_log_probs" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 3</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_per_seq_probs">
<span class="sig-name descname"><span class="pre">ranked_per_seq_probs</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_per_seq_probs" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 1</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_per_symbol_probs">
<span class="sig-name descname"><span class="pre">ranked_per_symbol_probs</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_per_symbol_probs" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 0</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_tgt_out_idx">
<span class="sig-name descname"><span class="pre">ranked_tgt_out_idx</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.seq2slate.Seq2SlateTransformerOutput.ranked_tgt_out_idx" title="Permalink to this definition"></a></dt>
<dd><p>Alias for field number 2</p>
</dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate.SublayerConnection">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate.</span></span><span class="sig-name descname"><span class="pre">SublayerConnection</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dim_model</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.SublayerConnection" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>A residual connection followed by a layer norm.</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate.SublayerConnection.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sublayer</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate.SublayerConnection.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate.SublayerConnection.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate.SublayerConnection.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.seq2slate_reward">
<span id="reagent-models-seq2slate-reward-module"></span><h2>reagent.models.seq2slate_reward module<a class="headerlink" href="#module-reagent.models.seq2slate_reward" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateGRURewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate_reward.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateGRURewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateGRURewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase" title="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.seq2slate_reward.Seq2SlateRewardNetBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.embed">
<span class="sig-name descname"><span class="pre">embed</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_seq</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.embed" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.PreprocessedRankingInput" title="reagent.core.types.PreprocessedRankingInput"><span class="pre">reagent.core.types.PreprocessedRankingInput</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateGRURewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate_reward.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateRewardNetBase</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate_reward.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateRewardNetEnsemble</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">models</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_src_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetEnsemble.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate_reward.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateRewardNetJITWrapper</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase" title="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase"><span class="pre">reagent.models.seq2slate_reward.Seq2SlateRewardNetBase</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_seq</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_src_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_out_idx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">use_gpu</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetJITWrapper.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.seq2slate_reward.</span></span><span class="sig-name descname"><span class="pre">Seq2SlateTransformerRewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">candidate_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_stacked_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_heads</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_src_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_tgt_seq_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.seq2slate_reward.Seq2SlateRewardNetBase" title="reagent.models.seq2slate_reward.Seq2SlateRewardNetBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.seq2slate_reward.Seq2SlateRewardNetBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.decode">
<span class="sig-name descname"><span class="pre">decode</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">memory</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_src_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_in_seq</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_tgt_mask</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tgt_seq_len</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.decode" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.encode">
<span class="sig-name descname"><span class="pre">encode</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_seq</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_mask</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.encode" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.PreprocessedRankingInput" title="reagent.core.types.PreprocessedRankingInput"><span class="pre">reagent.core.types.PreprocessedRankingInput</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.seq2slate_reward.Seq2SlateTransformerRewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models.synthetic_reward">
<span id="reagent-models-synthetic-reward-module"></span><h2>reagent.models.synthetic_reward module<a class="headerlink" href="#module-reagent.models.synthetic_reward" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.Concat">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">Concat</span></span><a class="headerlink" href="#reagent.models.synthetic_reward.Concat" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.Concat.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.Concat.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.Concat.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.Concat.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramConvolutionalNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">NGramConvolutionalNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">last_layer_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">conv_net_params</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.parameters.ConvNetParameters" title="reagent.core.parameters.ConvNetParameters"><span class="pre">reagent.core.parameters.ConvNetParameters</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.NGramConvolutionalNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramConvolutionalNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.synthetic_reward.NGramConvolutionalNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass NGram conv net.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>shape</strong> (<em>input</em>) – seq_len, batch_size, feature_dim</p>
</dd>
</dl>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramConvolutionalNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.NGramConvolutionalNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramFullyConnectedNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">NGramFullyConnectedNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">last_layer_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.NGramFullyConnectedNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramFullyConnectedNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.synthetic_reward.NGramFullyConnectedNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass NGram conv net.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>shape</strong> (<em>input</em>) – seq_len, batch_size, feature_dim</p>
</dd>
</dl>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.NGramFullyConnectedNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.NGramFullyConnectedNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PETransformerEncoderLayer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">PETransformerEncoderLayer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">d_model</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">nhead</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">2048</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activation</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'relu'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layer_norm_eps</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1e-05</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_len</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_ff</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pos_weight</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.5</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch_first</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">device</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dtype</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.PETransformerEncoderLayer" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>PETransformerEncoderLayer is made up of Positional Encoding (PE), residual connections, self-attn and feedforward network.
Major differences between this implementation and the pytorch official torch.nn.TransformerEncoderLayer are:
1. Augment input data with positional encoding. hat{x} = x + PE{x}
2. Two paralle residual blocks are applied to the raw input data (x) and encoded input data (hat{x}), respectively, i.e. z = Residual(x), hat{z} = Residual(hat{x})
3. Treat z as the Value input, and hat{z} as the Query and Key input to feed a self-attention block.</p>
<dl>
<dt>Main Args:</dt><dd><p>d_model: the number of expected features in the input (required).
nhead: the number of heads in the multiheadattention models (required).
dim_feedforward: the dimension of the feedforward network model (default=2048).
activation: the activation function of intermediate layer, relu or gelu (default=relu).
layer_norm_eps: the eps value in layer normalization components (default=1e-5).
batch_first: If <code class="docutils literal notranslate"><span class="pre">True</span></code>, then the input and output tensors are provided</p>
<blockquote>
<div><p>as (batch, seq, feature). Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>.</p>
</div></blockquote>
<p>max_len: argument passed to the Positional Encoding module, see more details in the PositionalEncoding class.</p>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PETransformerEncoderLayer.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">src</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_mask</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">src_key_padding_mask</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.PETransformerEncoderLayer.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PETransformerEncoderLayer.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.PETransformerEncoderLayer.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PositionalEncoding">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">PositionalEncoding</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">feature_dim</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">128</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_len</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.PositionalEncoding" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PositionalEncoding.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.PositionalEncoding.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.PositionalEncoding.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.PositionalEncoding.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.ResidualBlock">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">ResidualBlock</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">d_model</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">64</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">128</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.ResidualBlock" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.ResidualBlock.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.ResidualBlock.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.ResidualBlock.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.ResidualBlock.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SequenceSyntheticRewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">SequenceSyntheticRewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">lstm_hidden_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">lstm_num_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">lstm_bidirectional</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">last_layer_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SequenceSyntheticRewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SequenceSyntheticRewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SequenceSyntheticRewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SequenceSyntheticRewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.SequenceSyntheticRewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SequentialMultiArguments">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">SequentialMultiArguments</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SequentialMultiArguments" title="Permalink to this definition"></a></dt>
<dt class="sig sig-object py">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">SequentialMultiArguments</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.OrderedDict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">torch.nn.modules.module.Module</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.container.Sequential</span></code></p>
<p>Sequential which can take more than 1 argument in forward function</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SequentialMultiArguments.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">inputs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SequentialMultiArguments.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SingleStepSyntheticRewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">SingleStepSyntheticRewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">last_layer_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SingleStepSyntheticRewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SingleStepSyntheticRewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SingleStepSyntheticRewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SingleStepSyntheticRewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.SingleStepSyntheticRewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SyntheticRewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">SyntheticRewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">net</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SyntheticRewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>This base class provides basic operations to consume inputs and call a synthetic reward net</p>
<p>A synthetic reward net (self.net) assumes the input contains only torch.Tensors.
Expected input shape:</p>
<blockquote>
<div><p>state: seq_len, batch_size, state_dim
action: seq_len, batch_size, action_dim</p>
</div></blockquote>
<dl class="simple">
<dt>Expected output shape:</dt><dd><p>reward: batch_size, seq_len</p>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SyntheticRewardNet.export_mlp">
<span class="sig-name descname"><span class="pre">export_mlp</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SyntheticRewardNet.export_mlp" title="Permalink to this definition"></a></dt>
<dd><p>Export an pytorch nn to feed to predictor wrapper.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SyntheticRewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">training_batch</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.MemoryNetworkInput" title="reagent.core.types.MemoryNetworkInput"><span class="pre">reagent.core.types.MemoryNetworkInput</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.SyntheticRewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.SyntheticRewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.SyntheticRewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.TransformerSyntheticRewardNet">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">TransformerSyntheticRewardNet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">d_model</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">nhead</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_encoder_layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dim_feedforward</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">128</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'relu'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">last_layer_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'leaky_relu'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layer_norm_eps</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1e-05</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">10</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.TransformerSyntheticRewardNet" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.TransformerSyntheticRewardNet.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.TransformerSyntheticRewardNet.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.TransformerSyntheticRewardNet.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.synthetic_reward.TransformerSyntheticRewardNet.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="reagent.models.synthetic_reward.ngram">
<span class="sig-prename descclassname"><span class="pre">reagent.models.synthetic_reward.</span></span><span class="sig-name descname"><span class="pre">ngram</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context_size</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ngram_padding</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.synthetic_reward.ngram" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</section>
<section id="module-reagent.models.world_model">
<span id="reagent-models-world-model-module"></span><h2>reagent.models.world_model module<a class="headerlink" href="#module-reagent.models.world_model" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.world_model.MemoryNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.world_model.</span></span><span class="sig-name descname"><span class="pre">MemoryNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hiddens</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hidden_layers</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_gaussians</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.world_model.MemoryNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.world_model.MemoryNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.world_model.MemoryNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.world_model.MemoryNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.world_model.MemoryNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.world_model.MemoryNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.world_model.MemoryNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="module-reagent.models">
<span id="module-contents"></span><h2>Module contents<a class="headerlink" href="#module-reagent.models" title="Permalink to this headline"></a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.BatchConstrainedDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">BatchConstrainedDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">q_network</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">imitator_network</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">bcq_drop_threshold</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.BatchConstrainedDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.BatchConstrainedDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.BatchConstrainedDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.BatchConstrainedDQN.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.BatchConstrainedDQN.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.BatchConstrainedDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.BatchConstrainedDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.CategoricalDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">CategoricalDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">distributional_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">qmin</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">qmax</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.CategoricalDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.CategoricalDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.CategoricalDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.CategoricalDQN.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.CategoricalDQN.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.CategoricalDQN.log_dist">
<span class="sig-name descname"><span class="pre">log_dist</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.CategoricalDQN.log_dist" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.CategoricalDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.CategoricalDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">DirichletFullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor.EPSILON">
<span class="sig-name descname"><span class="pre">EPSILON</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">1e-06</span></em><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor.EPSILON" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor.get_log_prob">
<span class="sig-name descname"><span class="pre">get_log_prob</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor.get_log_prob" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.DirichletFullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.DirichletFullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.DuelingQNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">DuelingQNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">shared_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">advantage_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">value_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DuelingQNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DuelingQNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">possible_actions_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.DuelingQNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DuelingQNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DuelingQNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.DuelingQNetwork.make_fully_connected">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">make_fully_connected</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.DuelingQNetwork.make_fully_connected" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.DuelingQNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.DuelingQNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">EmbeddingBagConcat</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">model_feature_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">embedding_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.EmbeddingBagConcat" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Concatenating embedding with float features before passing the input
to DQN</p>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat.feat2table">
<span class="sig-name descname"><span class="pre">feat2table</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#reagent.models.EmbeddingBagConcat.feat2table" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.EmbeddingBagConcat.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.EmbeddingBagConcat.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py property">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat.output_dim">
<em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">output_dim</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span></em><a class="headerlink" href="#reagent.models.EmbeddingBagConcat.output_dim" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.EmbeddingBagConcat.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.EmbeddingBagConcat.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'tanh'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">exploration_variance</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><a class="reference internal" href="reagent.core.html#reagent.core.types.ActorOutput" title="reagent.core.types.ActorOutput"><span class="pre">reagent.core.types.ActorOutput</span></a></span></span><a class="headerlink" href="#reagent.models.FullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.FullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedCritic">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedCritic</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">1</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedCritic" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedCritic.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedCritic.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedCritic.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedCritic.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedCritic.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.FullyConnectedCritic.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedDQN">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedDQN</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_activation</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'linear'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_atoms</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout_ratio</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">normalized_output</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedDQN" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.fully_connected_network.FloatFeatureFullyConnected" title="reagent.models.fully_connected_network.FloatFeatureFullyConnected"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.fully_connected_network.FloatFeatureFullyConnected</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedDQN.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">possible_actions_mask</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.FullyConnectedDQN.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedDQN.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.FullyConnectedDQN.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">FullyConnectedNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">min_std</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dropout_ratio</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">normalize_output</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">orthogonal_init</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.FullyConnectedNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass for generic feed-forward DNNs. Assumes activation names
are valid pytorch activation names.
:param input tensor</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.FullyConnectedNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.FullyConnectedNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.FullyConnectedNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.GaussianFullyConnectedActor">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">GaussianFullyConnectedActor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sizes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scale</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">0.05</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_layer_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_l2_normalization</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.GaussianFullyConnectedActor" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.GaussianFullyConnectedActor.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.GaussianFullyConnectedActor.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.GaussianFullyConnectedActor.get_log_prob">
<span class="sig-name descname"><span class="pre">get_log_prob</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">squashed_action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.Tensor</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.GaussianFullyConnectedActor.get_log_prob" title="Permalink to this definition"></a></dt>
<dd><p>Action is expected to be squashed with tanh</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.GaussianFullyConnectedActor.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.GaussianFullyConnectedActor.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.GaussianFullyConnectedActor.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.GaussianFullyConnectedActor.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.MLPScorer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">MLPScorer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mlp</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_user_feat</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.MLPScorer" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Log-space in and out</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.MLPScorer.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">obs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.MLPScorer.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.MLPScorer.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.MLPScorer.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.MLPScorer.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.MLPScorer.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.ModelBase">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">ModelBase</span></span><a class="headerlink" href="#reagent.models.ModelBase" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.module.Module</span></code></p>
<p>A base class to support exporting through ONNX</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ModelBase.cpu_model">
<span class="sig-name descname"><span class="pre">cpu_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ModelBase.cpu_model" title="Permalink to this definition"></a></dt>
<dd><p>Override this in DistributedDataParallel models</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ModelBase.feature_config">
<span class="sig-name descname"><span class="pre">feature_config</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="reagent.core.html#reagent.core.types.ModelFeatureConfig" title="reagent.core.types.ModelFeatureConfig"><span class="pre">reagent.core.types.ModelFeatureConfig</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#reagent.models.ModelBase.feature_config" title="Permalink to this definition"></a></dt>
<dd><p>If the model needs additional preprocessing, e.g., using sequence features,
returns the config here.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ModelBase.get_distributed_data_parallel_model">
<span class="sig-name descname"><span class="pre">get_distributed_data_parallel_model</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ModelBase.get_distributed_data_parallel_model" title="Permalink to this definition"></a></dt>
<dd><p>Return DistributedDataParallel version of this model</p>
<p>This needs to be implemented explicitly because:
1) Model with EmbeddingBag module is not compatible with vanilla DistributedDataParallel
2) Exporting logic needs structured data. DistributedDataParallel doesn’t work with structured data.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ModelBase.get_target_network">
<span class="sig-name descname"><span class="pre">get_target_network</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ModelBase.get_target_network" title="Permalink to this definition"></a></dt>
<dd><p>Return a copy of this network to be used as target network</p>
<p>Subclass should override this if the target network should share parameters
with the network to be trained.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ModelBase.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Any</span></span></span><a class="headerlink" href="#reagent.models.ModelBase.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.ModelBase.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.ModelBase.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.ParametricDuelingQNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">ParametricDuelingQNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">shared_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">advantage_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">value_network</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><span class="pre">reagent.models.base.ModelBase</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ParametricDuelingQNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ParametricDuelingQNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">torch.Tensor</span></span></span><a class="headerlink" href="#reagent.models.ParametricDuelingQNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Defines the computation performed at every call.</p>
<p>Should be overridden by all subclasses.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Although the recipe for forward pass needs to be defined within
this function, one should call the <code class="xref py py-class docutils literal notranslate"><span class="pre">Module</span></code> instance afterwards
instead of this since the former takes care of running the
registered hooks while the latter silently ignores them.</p>
</div>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ParametricDuelingQNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ParametricDuelingQNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.ParametricDuelingQNetwork.make_fully_connected">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">make_fully_connected</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">activations</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_batch_norm</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.ParametricDuelingQNetwork.make_fully_connected" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.ParametricDuelingQNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.ParametricDuelingQNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.Seq2RewardNetwork">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">Seq2RewardNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">action_dim</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hiddens</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_hidden_layers</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Seq2RewardNetwork" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.Seq2RewardNetwork.forward">
<span class="sig-name descname"><span class="pre">forward</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">action</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="reagent.core.html#reagent.core.types.FeatureData" title="reagent.core.types.FeatureData"><span class="pre">reagent.core.types.FeatureData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">valid_reward_len</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">torch.Tensor</span><span class="p"><span class="pre">]</span></span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Seq2RewardNetwork.forward" title="Permalink to this definition"></a></dt>
<dd><p>Forward pass of Seq2Reward</p>
<p>Takes in the current state and use it as init hidden
The input sequence are pure actions only
Output the predicted reward after each time step</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>actions</strong> – (SEQ_LEN, BATCH_SIZE, ACTION_DIM) torch tensor</p></li>
<li><p><strong>states</strong> – (SEQ_LEN, BATCH_SIZE, STATE_DIM) torch tensor</p></li>
<li><p><strong>valid_reward_len</strong> – (BATCH_SIZE,) torch tensor</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>predicated accumulated rewards at last step for the given sequence
- acc_reward: (BATCH_SIZE, 1) torch tensor</p>
</dd>
</dl>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.Seq2RewardNetwork.get_initial_hidden_state">
<span class="sig-name descname"><span class="pre">get_initial_hidden_state</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">state</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Seq2RewardNetwork.get_initial_hidden_state" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.Seq2RewardNetwork.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Seq2RewardNetwork.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="reagent.models.Seq2RewardNetwork.training">
<span class="sig-name descname"><span class="pre">training</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#reagent.models.Seq2RewardNetwork.training" title="Permalink to this definition"></a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="py class">
<dt class="sig sig-object py" id="reagent.models.Sequential">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">Sequential</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">torch.nn.modules.module.Module</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Sequential" title="Permalink to this definition"></a></dt>
<dt class="sig sig-object py">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">reagent.models.</span></span><span class="sig-name descname"><span class="pre">Sequential</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.OrderedDict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">torch.nn.modules.module.Module</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.modules.container.Sequential</span></code>, <a class="reference internal" href="#reagent.models.base.ModelBase" title="reagent.models.base.ModelBase"><code class="xref py py-class docutils literal notranslate"><span class="pre">reagent.models.base.ModelBase</span></code></a></p>
<p>Used this instead of torch.nn.Sequential to automate model tracing</p>
<dl class="py method">
<dt class="sig sig-object py" id="reagent.models.Sequential.input_prototype">
<span class="sig-name descname"><span class="pre">input_prototype</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#reagent.models.Sequential.input_prototype" title="Permalink to this definition"></a></dt>
<dd><p>This function provides the input for ONNX graph tracing.</p>
<p>The return value should be what expected by <cite>forward()</cite>.</p>
</dd></dl>
</dd></dl>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="reagent.optimizer.html" class="btn btn-neutral float-left" title="reagent.optimizer package" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="reagent.prediction.html" class="btn btn-neutral float-right" title="reagent.prediction package" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2022, Meta Platforms, Inc.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink