diff --git a/_images/reagent_banner.png b/_images/reagent_banner.png new file mode 100644 index 00000000..6510242e Binary files /dev/null and b/_images/reagent_banner.png differ diff --git a/_sources/api/ml.rl.evaluation.rst.txt b/_sources/api/ml.rl.evaluation.rst.txt new file mode 100644 index 00000000..24492e79 --- /dev/null +++ b/_sources/api/ml.rl.evaluation.rst.txt @@ -0,0 +1,78 @@ +ml.rl.evaluation package +======================== + +Submodules +---------- + +ml.rl.evaluation.cpe module +--------------------------- + +.. automodule:: ml.rl.evaluation.cpe + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.doubly\_robust\_estimator module +------------------------------------------------- + +.. automodule:: ml.rl.evaluation.doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.evaluation\_data\_page module +---------------------------------------------- + +.. automodule:: ml.rl.evaluation.evaluation_data_page + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.evaluator module +--------------------------------- + +.. automodule:: ml.rl.evaluation.evaluator + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.ranking\_evaluator module +------------------------------------------ + +.. automodule:: ml.rl.evaluation.ranking_evaluator + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.sequential\_doubly\_robust\_estimator module +------------------------------------------------------------- + +.. automodule:: ml.rl.evaluation.sequential_doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.weighted\_sequential\_doubly\_robust\_estimator module +----------------------------------------------------------------------- + +.. automodule:: ml.rl.evaluation.weighted_sequential_doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +ml.rl.evaluation.world\_model\_evaluator module +----------------------------------------------- + +.. automodule:: ml.rl.evaluation.world_model_evaluator + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.evaluation + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.models.rst.txt b/_sources/api/ml.rl.models.rst.txt new file mode 100644 index 00000000..4d1804ca --- /dev/null +++ b/_sources/api/ml.rl.models.rst.txt @@ -0,0 +1,150 @@ +ml.rl.models package +==================== + +Submodules +---------- + +ml.rl.models.actor module +------------------------- + +.. automodule:: ml.rl.models.actor + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.base module +------------------------ + +.. automodule:: ml.rl.models.base + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.bcq module +----------------------- + +.. automodule:: ml.rl.models.bcq + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.categorical\_dqn module +------------------------------------ + +.. automodule:: ml.rl.models.categorical_dqn + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.cem\_planner module +-------------------------------- + +.. automodule:: ml.rl.models.cem_planner + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.convolutional\_network module +------------------------------------------ + +.. automodule:: ml.rl.models.convolutional_network + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.dqn module +----------------------- + +.. automodule:: ml.rl.models.dqn + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.dueling\_q\_network module +--------------------------------------- + +.. automodule:: ml.rl.models.dueling_q_network + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.dueling\_quantile\_dqn module +------------------------------------------ + +.. automodule:: ml.rl.models.dueling_quantile_dqn + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.example\_sequence\_model module +-------------------------------------------- + +.. automodule:: ml.rl.models.example_sequence_model + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.fully\_connected\_network module +--------------------------------------------- + +.. automodule:: ml.rl.models.fully_connected_network + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.mdn\_rnn module +---------------------------- + +.. automodule:: ml.rl.models.mdn_rnn + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.no\_soft\_update\_embedding module +----------------------------------------------- + +.. automodule:: ml.rl.models.no_soft_update_embedding + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.parametric\_dqn module +----------------------------------- + +.. automodule:: ml.rl.models.critic + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.quantile\_dqn module +--------------------------------- + +.. automodule:: ml.rl.models.quantile_dqn + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.seq2slate module +----------------------------- + +.. automodule:: ml.rl.models.seq2slate + :members: + :undoc-members: + :show-inheritance: + +ml.rl.models.world\_model module +-------------------------------- + +.. automodule:: ml.rl.models.world_model + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.models + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.prediction.rst.txt b/_sources/api/ml.rl.prediction.rst.txt new file mode 100644 index 00000000..bae5ed6d --- /dev/null +++ b/_sources/api/ml.rl.prediction.rst.txt @@ -0,0 +1,30 @@ +ml.rl.prediction package +======================== + +Submodules +---------- + +ml.rl.prediction.dqn\_torch\_predictor module +--------------------------------------------- + +.. automodule:: ml.rl.prediction.dqn_torch_predictor + :members: + :undoc-members: + :show-inheritance: + +ml.rl.prediction.predictor\_wrapper module +------------------------------------------ + +.. automodule:: ml.rl.prediction.predictor_wrapper + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.prediction + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.readers.rst.txt b/_sources/api/ml.rl.readers.rst.txt new file mode 100644 index 00000000..f8ddeaf6 --- /dev/null +++ b/_sources/api/ml.rl.readers.rst.txt @@ -0,0 +1,46 @@ +ml.rl.readers package +===================== + +Submodules +---------- + +ml.rl.readers.base module +------------------------- + +.. automodule:: ml.rl.readers.base + :members: + :undoc-members: + :show-inheritance: + +ml.rl.readers.data\_streamer module +----------------------------------- + +.. automodule:: ml.rl.readers.data_streamer + :members: + :undoc-members: + :show-inheritance: + +ml.rl.readers.json\_dataset\_reader module +------------------------------------------ + +.. automodule:: ml.rl.readers.json_dataset_reader + :members: + :undoc-members: + :show-inheritance: + +ml.rl.readers.nparray\_reader module +------------------------------------ + +.. automodule:: ml.rl.readers.nparray_reader + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.readers + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.rst.txt b/_sources/api/ml.rl.rst.txt index a874ec76..c4010bd4 100644 --- a/_sources/api/ml.rl.rst.txt +++ b/_sources/api/ml.rl.rst.txt @@ -6,7 +6,11 @@ Subpackages .. toctree:: + ml.rl.evaluation + ml.rl.models + ml.rl.prediction ml.rl.preprocessing + ml.rl.readers ml.rl.simulators ml.rl.training ml.rl.workflow diff --git a/_sources/api/ml.rl.training.gradient_free.rst.txt b/_sources/api/ml.rl.training.gradient_free.rst.txt new file mode 100644 index 00000000..0629b4b8 --- /dev/null +++ b/_sources/api/ml.rl.training.gradient_free.rst.txt @@ -0,0 +1,30 @@ +ml.rl.training.gradient\_free package +===================================== + +Submodules +---------- + +ml.rl.training.gradient\_free.es\_worker module +----------------------------------------------- + +.. automodule:: ml.rl.training.gradient_free.es_worker + :members: + :undoc-members: + :show-inheritance: + +ml.rl.training.gradient\_free.evolution\_pool module +---------------------------------------------------- + +.. automodule:: ml.rl.training.gradient_free.evolution_pool + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.training.gradient_free + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.training.ranking.rst.txt b/_sources/api/ml.rl.training.ranking.rst.txt new file mode 100644 index 00000000..5477af1b --- /dev/null +++ b/_sources/api/ml.rl.training.ranking.rst.txt @@ -0,0 +1,30 @@ +ml.rl.training.ranking package +============================== + +Submodules +---------- + +ml.rl.training.ranking.ranking\_trainer module +---------------------------------------------- + +.. automodule:: ml.rl.training.ranking.ranking_trainer + :members: + :undoc-members: + :show-inheritance: + +ml.rl.training.ranking.seq2slate\_trainer module +------------------------------------------------ + +.. automodule:: ml.rl.training.ranking.seq2slate_trainer + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: ml.rl.training.ranking + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/ml.rl.training.rst.txt b/_sources/api/ml.rl.training.rst.txt index 89e15b50..57785f36 100644 --- a/_sources/api/ml.rl.training.rst.txt +++ b/_sources/api/ml.rl.training.rst.txt @@ -6,19 +6,13 @@ Subpackages .. toctree:: + ml.rl.training.gradient_free + ml.rl.training.ranking ml.rl.training.world_model Submodules ---------- -ml.rl.training.actor\_predictor module --------------------------------------- - -.. automodule:: ml.rl.training.actor_predictor - :members: - :undoc-members: - :show-inheritance: - ml.rl.training.c51\_trainer module ---------------------------------- @@ -91,14 +85,6 @@ ml.rl.training.on\_policy\_predictor module :undoc-members: :show-inheritance: -ml.rl.training.parametric\_dqn\_predictor module ------------------------------------------------- - -.. automodule:: ml.rl.training.parametric_dqn_predictor - :members: - :undoc-members: - :show-inheritance: - ml.rl.training.parametric\_dqn\_trainer module ---------------------------------------------- @@ -123,14 +109,6 @@ ml.rl.training.rl\_dataset module :undoc-members: :show-inheritance: -ml.rl.training.rl\_exporter module ----------------------------------- - -.. automodule:: ml.rl.training.rl_exporter - :members: - :undoc-members: - :show-inheritance: - ml.rl.training.rl\_trainer\_pytorch module ------------------------------------------ diff --git a/_sources/api/modules.rst.txt b/_sources/api/modules.rst.txt index 25b2afbc..3b064fef 100644 --- a/_sources/api/modules.rst.txt +++ b/_sources/api/modules.rst.txt @@ -1,7 +1,7 @@ -ml -== +reagent +======= .. toctree:: :maxdepth: 4 - ml + reagent diff --git a/_sources/api/reagent.evaluation.rst.txt b/_sources/api/reagent.evaluation.rst.txt new file mode 100644 index 00000000..981585bf --- /dev/null +++ b/_sources/api/reagent.evaluation.rst.txt @@ -0,0 +1,110 @@ +reagent.evaluation package +========================== + +Submodules +---------- + +reagent.evaluation.cpe module +----------------------------- + +.. automodule:: reagent.evaluation.cpe + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.doubly\_robust\_estimator module +--------------------------------------------------- + +.. automodule:: reagent.evaluation.doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.evaluation\_data\_page module +------------------------------------------------ + +.. automodule:: reagent.evaluation.evaluation_data_page + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.evaluator module +----------------------------------- + +.. automodule:: reagent.evaluation.evaluator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.ope\_adapter module +-------------------------------------- + +.. automodule:: reagent.evaluation.ope_adapter + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.ranking\_listwise\_evaluator module +------------------------------------------------------ + +.. automodule:: reagent.evaluation.ranking_listwise_evaluator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.ranking\_policy\_gradient\_evaluator module +-------------------------------------------------------------- + +.. automodule:: reagent.evaluation.ranking_policy_gradient_evaluator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.reward\_net\_evaluator module +------------------------------------------------ + +.. automodule:: reagent.evaluation.reward_net_evaluator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.seq2reward\_evaluator module +----------------------------------------------- + +.. automodule:: reagent.evaluation.seq2reward_evaluator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.sequential\_doubly\_robust\_estimator module +--------------------------------------------------------------- + +.. automodule:: reagent.evaluation.sequential_doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.weighted\_sequential\_doubly\_robust\_estimator module +------------------------------------------------------------------------- + +.. automodule:: reagent.evaluation.weighted_sequential_doubly_robust_estimator + :members: + :undoc-members: + :show-inheritance: + +reagent.evaluation.world\_model\_evaluator module +------------------------------------------------- + +.. automodule:: reagent.evaluation.world_model_evaluator + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.evaluation + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.agents.rst.txt b/_sources/api/reagent.gym.agents.rst.txt new file mode 100644 index 00000000..c40a9a56 --- /dev/null +++ b/_sources/api/reagent.gym.agents.rst.txt @@ -0,0 +1,30 @@ +reagent.gym.agents package +========================== + +Submodules +---------- + +reagent.gym.agents.agent module +------------------------------- + +.. automodule:: reagent.gym.agents.agent + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.agents.post\_step module +------------------------------------ + +.. automodule:: reagent.gym.agents.post_step + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.agents + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.envs.rst.txt b/_sources/api/reagent.gym.envs.rst.txt new file mode 100644 index 00000000..210e33b3 --- /dev/null +++ b/_sources/api/reagent.gym.envs.rst.txt @@ -0,0 +1,54 @@ +reagent.gym.envs package +======================== + +Submodules +---------- + +reagent.gym.envs.changing\_arms module +-------------------------------------- + +.. automodule:: reagent.gym.envs.changing_arms + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.envs.env\_factory module +------------------------------------ + +.. automodule:: reagent.gym.envs.env_factory + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.envs.recsim module +------------------------------ + +.. automodule:: reagent.gym.envs.recsim + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.envs.simple\_minigrid module +---------------------------------------- + +.. automodule:: reagent.gym.envs.simple_minigrid + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.envs.utils module +----------------------------- + +.. automodule:: reagent.gym.envs.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.envs + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.policies.rst.txt b/_sources/api/reagent.gym.policies.rst.txt new file mode 100644 index 00000000..f3d59962 --- /dev/null +++ b/_sources/api/reagent.gym.policies.rst.txt @@ -0,0 +1,46 @@ +reagent.gym.policies package +============================ + +Subpackages +----------- + +.. toctree:: + + reagent.gym.policies.samplers + reagent.gym.policies.scorers + +Submodules +---------- + +reagent.gym.policies.policy module +---------------------------------- + +.. automodule:: reagent.gym.policies.policy + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.predictor\_policies module +----------------------------------------------- + +.. automodule:: reagent.gym.policies.predictor_policies + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.random\_policies module +-------------------------------------------- + +.. automodule:: reagent.gym.policies.random_policies + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.policies + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.policies.samplers.rst.txt b/_sources/api/reagent.gym.policies.samplers.rst.txt new file mode 100644 index 00000000..a2c973eb --- /dev/null +++ b/_sources/api/reagent.gym.policies.samplers.rst.txt @@ -0,0 +1,38 @@ +reagent.gym.policies.samplers package +===================================== + +Submodules +---------- + +reagent.gym.policies.samplers.continuous\_sampler module +-------------------------------------------------------- + +.. automodule:: reagent.gym.policies.samplers.continuous_sampler + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.samplers.discrete\_sampler module +------------------------------------------------------ + +.. automodule:: reagent.gym.policies.samplers.discrete_sampler + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.samplers.top\_k\_sampler module +---------------------------------------------------- + +.. automodule:: reagent.gym.policies.samplers.top_k_sampler + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.policies.samplers + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.policies.scorers.rst.txt b/_sources/api/reagent.gym.policies.scorers.rst.txt new file mode 100644 index 00000000..0474cc8f --- /dev/null +++ b/_sources/api/reagent.gym.policies.scorers.rst.txt @@ -0,0 +1,38 @@ +reagent.gym.policies.scorers package +==================================== + +Submodules +---------- + +reagent.gym.policies.scorers.continuous\_scorer module +------------------------------------------------------ + +.. automodule:: reagent.gym.policies.scorers.continuous_scorer + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.scorers.discrete\_scorer module +---------------------------------------------------- + +.. automodule:: reagent.gym.policies.scorers.discrete_scorer + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.policies.scorers.slate\_q\_scorer module +---------------------------------------------------- + +.. automodule:: reagent.gym.policies.scorers.slate_q_scorer + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.policies.scorers + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.preprocessors.rst.txt b/_sources/api/reagent.gym.preprocessors.rst.txt new file mode 100644 index 00000000..0cef8ba2 --- /dev/null +++ b/_sources/api/reagent.gym.preprocessors.rst.txt @@ -0,0 +1,46 @@ +reagent.gym.preprocessors package +================================= + +Submodules +---------- + +reagent.gym.preprocessors.default\_preprocessors module +------------------------------------------------------- + +.. automodule:: reagent.gym.preprocessors.default_preprocessors + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.preprocessors.default\_serving\_preprocessors module +---------------------------------------------------------------- + +.. automodule:: reagent.gym.preprocessors.default_serving_preprocessors + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.preprocessors.replay\_buffer\_inserters module +---------------------------------------------------------- + +.. automodule:: reagent.gym.preprocessors.replay_buffer_inserters + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.preprocessors.trainer\_preprocessor module +------------------------------------------------------ + +.. automodule:: reagent.gym.preprocessors.trainer_preprocessor + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.preprocessors + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.rst.txt b/_sources/api/reagent.gym.rst.txt new file mode 100644 index 00000000..a95233a2 --- /dev/null +++ b/_sources/api/reagent.gym.rst.txt @@ -0,0 +1,42 @@ +reagent.gym package +=================== + +Subpackages +----------- + +.. toctree:: + + reagent.gym.agents + reagent.gym.envs + reagent.gym.policies + reagent.gym.preprocessors + reagent.gym.runners + reagent.gym.tests + +Submodules +---------- + +reagent.gym.types module +------------------------ + +.. automodule:: reagent.gym.types + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.utils module +------------------------ + +.. automodule:: reagent.gym.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.runners.rst.txt b/_sources/api/reagent.gym.runners.rst.txt new file mode 100644 index 00000000..867aa575 --- /dev/null +++ b/_sources/api/reagent.gym.runners.rst.txt @@ -0,0 +1,22 @@ +reagent.gym.runners package +=========================== + +Submodules +---------- + +reagent.gym.runners.gymrunner module +------------------------------------ + +.. automodule:: reagent.gym.runners.gymrunner + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.runners + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.tests.preprocessors.rst.txt b/_sources/api/reagent.gym.tests.preprocessors.rst.txt new file mode 100644 index 00000000..130f2ec6 --- /dev/null +++ b/_sources/api/reagent.gym.tests.preprocessors.rst.txt @@ -0,0 +1,30 @@ +reagent.gym.tests.preprocessors package +======================================= + +Submodules +---------- + +reagent.gym.tests.preprocessors.test\_default\_preprocessors module +------------------------------------------------------------------- + +.. automodule:: reagent.gym.tests.preprocessors.test_default_preprocessors + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.preprocessors.test\_replay\_buffer\_inserters module +---------------------------------------------------------------------- + +.. automodule:: reagent.gym.tests.preprocessors.test_replay_buffer_inserters + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.tests.preprocessors + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.gym.tests.rst.txt b/_sources/api/reagent.gym.tests.rst.txt new file mode 100644 index 00000000..7c78c690 --- /dev/null +++ b/_sources/api/reagent.gym.tests.rst.txt @@ -0,0 +1,69 @@ +reagent.gym.tests package +========================= + +Subpackages +----------- + +.. toctree:: + + reagent.gym.tests.preprocessors + +Submodules +---------- + +reagent.gym.tests.test\_gym module +---------------------------------- + +.. automodule:: reagent.gym.tests.test_gym + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.test\_gym\_offline module +------------------------------------------- + +.. automodule:: reagent.gym.tests.test_gym_offline + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.test\_linear\_dynamics module +----------------------------------------------- + +.. automodule:: reagent.gym.tests.test_linear_dynamics + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.test\_pomdp module +------------------------------------ + +.. automodule:: reagent.gym.tests.test_pomdp + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.test\_seq2reward\_model module +------------------------------------------------ + +.. automodule:: reagent.gym.tests.test_seq2reward_model + :members: + :undoc-members: + :show-inheritance: + +reagent.gym.tests.test\_world\_model module +------------------------------------------- + +.. automodule:: reagent.gym.tests.test_world_model + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.gym.tests + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.models.rst.txt b/_sources/api/reagent.models.rst.txt new file mode 100644 index 00000000..34405509 --- /dev/null +++ b/_sources/api/reagent.models.rst.txt @@ -0,0 +1,166 @@ +reagent.models package +====================== + +Submodules +---------- + +reagent.models.actor module +--------------------------- + +.. automodule:: reagent.models.actor + :members: + :undoc-members: + :show-inheritance: + +reagent.models.base module +-------------------------- + +.. automodule:: reagent.models.base + :members: + :undoc-members: + :show-inheritance: + +reagent.models.bcq module +------------------------- + +.. automodule:: reagent.models.bcq + :members: + :undoc-members: + :show-inheritance: + +reagent.models.categorical\_dqn module +-------------------------------------- + +.. automodule:: reagent.models.categorical_dqn + :members: + :undoc-members: + :show-inheritance: + +reagent.models.cem\_planner module +---------------------------------- + +.. automodule:: reagent.models.cem_planner + :members: + :undoc-members: + :show-inheritance: + +reagent.models.containers module +-------------------------------- + +.. automodule:: reagent.models.containers + :members: + :undoc-members: + :show-inheritance: + +reagent.models.convolutional\_network module +-------------------------------------------- + +.. automodule:: reagent.models.convolutional_network + :members: + :undoc-members: + :show-inheritance: + +reagent.models.critic module +---------------------------- + +.. automodule:: reagent.models.critic + :members: + :undoc-members: + :show-inheritance: + +reagent.models.dqn module +------------------------- + +.. automodule:: reagent.models.dqn + :members: + :undoc-members: + :show-inheritance: + +reagent.models.dueling\_q\_network module +----------------------------------------- + +.. automodule:: reagent.models.dueling_q_network + :members: + :undoc-members: + :show-inheritance: + +reagent.models.embedding\_bag\_concat module +-------------------------------------------- + +.. automodule:: reagent.models.embedding_bag_concat + :members: + :undoc-members: + :show-inheritance: + +reagent.models.fully\_connected\_network module +----------------------------------------------- + +.. automodule:: reagent.models.fully_connected_network + :members: + :undoc-members: + :show-inheritance: + +reagent.models.mdn\_rnn module +------------------------------ + +.. automodule:: reagent.models.mdn_rnn + :members: + :undoc-members: + :show-inheritance: + +reagent.models.model\_feature\_config\_provider module +------------------------------------------------------ + +.. automodule:: reagent.models.model_feature_config_provider + :members: + :undoc-members: + :show-inheritance: + +reagent.models.no\_soft\_update\_embedding module +------------------------------------------------- + +.. automodule:: reagent.models.no_soft_update_embedding + :members: + :undoc-members: + :show-inheritance: + +reagent.models.seq2reward\_model module +--------------------------------------- + +.. automodule:: reagent.models.seq2reward_model + :members: + :undoc-members: + :show-inheritance: + +reagent.models.seq2slate module +------------------------------- + +.. automodule:: reagent.models.seq2slate + :members: + :undoc-members: + :show-inheritance: + +reagent.models.seq2slate\_reward module +--------------------------------------- + +.. automodule:: reagent.models.seq2slate_reward + :members: + :undoc-members: + :show-inheritance: + +reagent.models.world\_model module +---------------------------------- + +.. automodule:: reagent.models.world_model + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.models + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.datasets.rst.txt b/_sources/api/reagent.ope.datasets.rst.txt new file mode 100644 index 00000000..8b66aaa1 --- /dev/null +++ b/_sources/api/reagent.ope.datasets.rst.txt @@ -0,0 +1,22 @@ +reagent.ope.datasets package +============================ + +Submodules +---------- + +reagent.ope.datasets.logged\_dataset module +------------------------------------------- + +.. automodule:: reagent.ope.datasets.logged_dataset + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope.datasets + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.estimators.rst.txt b/_sources/api/reagent.ope.estimators.rst.txt new file mode 100644 index 00000000..43d418d5 --- /dev/null +++ b/_sources/api/reagent.ope.estimators.rst.txt @@ -0,0 +1,54 @@ +reagent.ope.estimators package +============================== + +Submodules +---------- + +reagent.ope.estimators.contextual\_bandits\_estimators module +------------------------------------------------------------- + +.. automodule:: reagent.ope.estimators.contextual_bandits_estimators + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.estimators.estimator module +--------------------------------------- + +.. automodule:: reagent.ope.estimators.estimator + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.estimators.sequential\_estimators module +---------------------------------------------------- + +.. automodule:: reagent.ope.estimators.sequential_estimators + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.estimators.slate\_estimators module +----------------------------------------------- + +.. automodule:: reagent.ope.estimators.slate_estimators + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.estimators.types module +----------------------------------- + +.. automodule:: reagent.ope.estimators.types + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope.estimators + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.rst.txt b/_sources/api/reagent.ope.rst.txt new file mode 100644 index 00000000..236ca35e --- /dev/null +++ b/_sources/api/reagent.ope.rst.txt @@ -0,0 +1,32 @@ +reagent.ope package +=================== + +Subpackages +----------- + +.. toctree:: + + reagent.ope.datasets + reagent.ope.estimators + reagent.ope.test + reagent.ope.trainers + +Submodules +---------- + +reagent.ope.utils module +------------------------ + +.. automodule:: reagent.ope.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.test.rst.txt b/_sources/api/reagent.ope.test.rst.txt new file mode 100644 index 00000000..fa0da7fc --- /dev/null +++ b/_sources/api/reagent.ope.test.rst.txt @@ -0,0 +1,61 @@ +reagent.ope.test package +======================== + +Subpackages +----------- + +.. toctree:: + + reagent.ope.test.unit_tests + +Submodules +---------- + +reagent.ope.test.envs module +---------------------------- + +.. automodule:: reagent.ope.test.envs + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.gridworld module +--------------------------------- + +.. automodule:: reagent.ope.test.gridworld + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.mslr\_slate module +----------------------------------- + +.. automodule:: reagent.ope.test.mslr_slate + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.multiclass\_bandits module +------------------------------------------- + +.. automodule:: reagent.ope.test.multiclass_bandits + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.yandex\_web\_search module +------------------------------------------- + +.. automodule:: reagent.ope.test.yandex_web_search + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope.test + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.test.unit_tests.rst.txt b/_sources/api/reagent.ope.test.unit_tests.rst.txt new file mode 100644 index 00000000..d13d3245 --- /dev/null +++ b/_sources/api/reagent.ope.test.unit_tests.rst.txt @@ -0,0 +1,38 @@ +reagent.ope.test.unit\_tests package +==================================== + +Submodules +---------- + +reagent.ope.test.unit\_tests.test\_slate\_estimators module +----------------------------------------------------------- + +.. automodule:: reagent.ope.test.unit_tests.test_slate_estimators + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.unit\_tests.test\_types module +----------------------------------------------- + +.. automodule:: reagent.ope.test.unit_tests.test_types + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.test.unit\_tests.test\_utils module +----------------------------------------------- + +.. automodule:: reagent.ope.test.unit_tests.test_utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope.test.unit_tests + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.ope.trainers.rst.txt b/_sources/api/reagent.ope.trainers.rst.txt new file mode 100644 index 00000000..bd189b02 --- /dev/null +++ b/_sources/api/reagent.ope.trainers.rst.txt @@ -0,0 +1,30 @@ +reagent.ope.trainers package +============================ + +Submodules +---------- + +reagent.ope.trainers.linear\_trainers module +-------------------------------------------- + +.. automodule:: reagent.ope.trainers.linear_trainers + :members: + :undoc-members: + :show-inheritance: + +reagent.ope.trainers.rl\_tabular\_trainers module +------------------------------------------------- + +.. automodule:: reagent.ope.trainers.rl_tabular_trainers + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.ope.trainers + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.optimizer.rst.txt b/_sources/api/reagent.optimizer.rst.txt new file mode 100644 index 00000000..9f7cdb48 --- /dev/null +++ b/_sources/api/reagent.optimizer.rst.txt @@ -0,0 +1,70 @@ +reagent.optimizer package +========================= + +Submodules +---------- + +reagent.optimizer.optimizer module +---------------------------------- + +.. automodule:: reagent.optimizer.optimizer + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.scheduler module +---------------------------------- + +.. automodule:: reagent.optimizer.scheduler + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.scheduler\_union module +----------------------------------------- + +.. automodule:: reagent.optimizer.scheduler_union + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.uninferrable\_optimizers module +------------------------------------------------- + +.. automodule:: reagent.optimizer.uninferrable_optimizers + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.uninferrable\_schedulers module +------------------------------------------------- + +.. automodule:: reagent.optimizer.uninferrable_schedulers + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.union module +------------------------------ + +.. automodule:: reagent.optimizer.union + :members: + :undoc-members: + :show-inheritance: + +reagent.optimizer.utils module +------------------------------ + +.. automodule:: reagent.optimizer.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.optimizer + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.prediction.rst.txt b/_sources/api/reagent.prediction.rst.txt new file mode 100644 index 00000000..67a09264 --- /dev/null +++ b/_sources/api/reagent.prediction.rst.txt @@ -0,0 +1,22 @@ +reagent.prediction package +========================== + +Submodules +---------- + +reagent.prediction.predictor\_wrapper module +-------------------------------------------- + +.. automodule:: reagent.prediction.predictor_wrapper + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.prediction + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.preprocessing.rst.txt b/_sources/api/reagent.preprocessing.rst.txt new file mode 100644 index 00000000..e9373cec --- /dev/null +++ b/_sources/api/reagent.preprocessing.rst.txt @@ -0,0 +1,86 @@ +reagent.preprocessing package +============================= + +Submodules +---------- + +reagent.preprocessing.batch\_preprocessor module +------------------------------------------------ + +.. automodule:: reagent.preprocessing.batch_preprocessor + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.identify\_types module +-------------------------------------------- + +.. automodule:: reagent.preprocessing.identify_types + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.normalization module +------------------------------------------ + +.. automodule:: reagent.preprocessing.normalization + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.postprocessor module +------------------------------------------ + +.. automodule:: reagent.preprocessing.postprocessor + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.preprocessor module +----------------------------------------- + +.. automodule:: reagent.preprocessing.preprocessor + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.sparse\_preprocessor module +------------------------------------------------- + +.. automodule:: reagent.preprocessing.sparse_preprocessor + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.sparse\_to\_dense module +---------------------------------------------- + +.. automodule:: reagent.preprocessing.sparse_to_dense + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.transforms module +--------------------------------------- + +.. automodule:: reagent.preprocessing.transforms + :members: + :undoc-members: + :show-inheritance: + +reagent.preprocessing.types module +---------------------------------- + +.. automodule:: reagent.preprocessing.types + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.preprocessing + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.replay_memory.rst.txt b/_sources/api/reagent.replay_memory.rst.txt new file mode 100644 index 00000000..3b983f15 --- /dev/null +++ b/_sources/api/reagent.replay_memory.rst.txt @@ -0,0 +1,46 @@ +reagent.replay\_memory package +============================== + +Submodules +---------- + +reagent.replay\_memory.circular\_replay\_buffer module +------------------------------------------------------ + +.. automodule:: reagent.replay_memory.circular_replay_buffer + :members: + :undoc-members: + :show-inheritance: + +reagent.replay\_memory.prioritized\_replay\_buffer module +--------------------------------------------------------- + +.. automodule:: reagent.replay_memory.prioritized_replay_buffer + :members: + :undoc-members: + :show-inheritance: + +reagent.replay\_memory.sum\_tree module +--------------------------------------- + +.. automodule:: reagent.replay_memory.sum_tree + :members: + :undoc-members: + :show-inheritance: + +reagent.replay\_memory.utils module +----------------------------------- + +.. automodule:: reagent.replay_memory.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.replay_memory + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.rst.txt b/_sources/api/reagent.rst.txt new file mode 100644 index 00000000..d186ee0f --- /dev/null +++ b/_sources/api/reagent.rst.txt @@ -0,0 +1,94 @@ +reagent package +=============== + +Subpackages +----------- + +.. toctree:: + + reagent.evaluation + reagent.gym + reagent.models + reagent.ope + reagent.optimizer + reagent.prediction + reagent.preprocessing + reagent.replay_memory + reagent.training + reagent.workflow + +Submodules +---------- + +reagent.base\_dataclass module +------------------------------ + +.. automodule:: reagent.base_dataclass + :members: + :undoc-members: + :show-inheritance: + +reagent.debug\_on\_error module +------------------------------- + +.. automodule:: reagent.debug_on_error + :members: + :undoc-members: + :show-inheritance: + +reagent.json\_serialize module +------------------------------ + +.. automodule:: reagent.json_serialize + :members: + :undoc-members: + :show-inheritance: + +reagent.parameters module +------------------------- + +.. automodule:: reagent.parameters + :members: + :undoc-members: + :show-inheritance: + +reagent.parameters\_seq2slate module +------------------------------------ + +.. automodule:: reagent.parameters_seq2slate + :members: + :undoc-members: + :show-inheritance: + +reagent.tensorboardX module +--------------------------- + +.. automodule:: reagent.tensorboardX + :members: + :undoc-members: + :show-inheritance: + +reagent.torch\_utils module +--------------------------- + +.. automodule:: reagent.torch_utils + :members: + :undoc-members: + :show-inheritance: + +reagent.types module +-------------------- + +.. automodule:: reagent.types + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.training.gradient_free.rst.txt b/_sources/api/reagent.training.gradient_free.rst.txt new file mode 100644 index 00000000..69b38a63 --- /dev/null +++ b/_sources/api/reagent.training.gradient_free.rst.txt @@ -0,0 +1,30 @@ +reagent.training.gradient\_free package +======================================= + +Submodules +---------- + +reagent.training.gradient\_free.es\_worker module +------------------------------------------------- + +.. automodule:: reagent.training.gradient_free.es_worker + :members: + :undoc-members: + :show-inheritance: + +reagent.training.gradient\_free.evolution\_pool module +------------------------------------------------------ + +.. automodule:: reagent.training.gradient_free.evolution_pool + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.training.gradient_free + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.training.ranking.rst.txt b/_sources/api/reagent.training.ranking.rst.txt new file mode 100644 index 00000000..910dde54 --- /dev/null +++ b/_sources/api/reagent.training.ranking.rst.txt @@ -0,0 +1,54 @@ +reagent.training.ranking package +================================ + +Submodules +---------- + +reagent.training.ranking.seq2slate\_attn\_trainer module +-------------------------------------------------------- + +.. automodule:: reagent.training.ranking.seq2slate_attn_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.ranking.seq2slate\_dr\_trainer module +------------------------------------------------------ + +.. automodule:: reagent.training.ranking.seq2slate_dr_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.ranking.seq2slate\_sim\_trainer module +------------------------------------------------------- + +.. automodule:: reagent.training.ranking.seq2slate_sim_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.ranking.seq2slate\_tf\_trainer module +------------------------------------------------------ + +.. automodule:: reagent.training.ranking.seq2slate_tf_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.ranking.seq2slate\_trainer module +-------------------------------------------------- + +.. automodule:: reagent.training.ranking.seq2slate_trainer + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.training.ranking + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.training.rst.txt b/_sources/api/reagent.training.rst.txt new file mode 100644 index 00000000..79e2d8e2 --- /dev/null +++ b/_sources/api/reagent.training.rst.txt @@ -0,0 +1,159 @@ +reagent.training package +======================== + +Subpackages +----------- + +.. toctree:: + + reagent.training.gradient_free + reagent.training.ranking + reagent.training.world_model + +Submodules +---------- + +reagent.training.c51\_trainer module +------------------------------------ + +.. automodule:: reagent.training.c51_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.cem\_trainer module +------------------------------------ + +.. automodule:: reagent.training.cem_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.dqn\_trainer module +------------------------------------ + +.. automodule:: reagent.training.dqn_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.dqn\_trainer\_base module +------------------------------------------ + +.. automodule:: reagent.training.dqn_trainer_base + :members: + :undoc-members: + :show-inheritance: + +reagent.training.imitator\_training module +------------------------------------------ + +.. automodule:: reagent.training.imitator_training + :members: + :undoc-members: + :show-inheritance: + +reagent.training.loss\_reporter module +-------------------------------------- + +.. automodule:: reagent.training.loss_reporter + :members: + :undoc-members: + :show-inheritance: + +reagent.training.parameters module +---------------------------------- + +.. automodule:: reagent.training.parameters + :members: + :undoc-members: + :show-inheritance: + +reagent.training.parametric\_dqn\_trainer module +------------------------------------------------ + +.. automodule:: reagent.training.parametric_dqn_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.qrdqn\_trainer module +-------------------------------------- + +.. automodule:: reagent.training.qrdqn_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.reinforce module +--------------------------------- + +.. automodule:: reagent.training.reinforce + :members: + :undoc-members: + :show-inheritance: + +reagent.training.reward\_network\_trainer module +------------------------------------------------ + +.. automodule:: reagent.training.reward_network_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.rl\_trainer\_pytorch module +-------------------------------------------- + +.. automodule:: reagent.training.rl_trainer_pytorch + :members: + :undoc-members: + :show-inheritance: + +reagent.training.sac\_trainer module +------------------------------------ + +.. automodule:: reagent.training.sac_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.slate\_q\_trainer module +----------------------------------------- + +.. automodule:: reagent.training.slate_q_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.td3\_trainer module +------------------------------------ + +.. automodule:: reagent.training.td3_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.trainer module +------------------------------- + +.. automodule:: reagent.training.trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.utils module +----------------------------- + +.. automodule:: reagent.training.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.training + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.training.world_model.rst.txt b/_sources/api/reagent.training.world_model.rst.txt new file mode 100644 index 00000000..7c53a046 --- /dev/null +++ b/_sources/api/reagent.training.world_model.rst.txt @@ -0,0 +1,30 @@ +reagent.training.world\_model package +===================================== + +Submodules +---------- + +reagent.training.world\_model.mdnrnn\_trainer module +---------------------------------------------------- + +.. automodule:: reagent.training.world_model.mdnrnn_trainer + :members: + :undoc-members: + :show-inheritance: + +reagent.training.world\_model.seq2reward\_trainer module +-------------------------------------------------------- + +.. automodule:: reagent.training.world_model.seq2reward_trainer + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.training.world_model + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/api/reagent.workflow.rst.txt b/_sources/api/reagent.workflow.rst.txt new file mode 100644 index 00000000..1ddb8823 --- /dev/null +++ b/_sources/api/reagent.workflow.rst.txt @@ -0,0 +1,118 @@ +reagent.workflow package +======================== + +Submodules +---------- + +reagent.workflow.cli module +--------------------------- + +.. automodule:: reagent.workflow.cli + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.data\_fetcher module +------------------------------------- + +.. automodule:: reagent.workflow.data_fetcher + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.env module +--------------------------- + +.. automodule:: reagent.workflow.env + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.gym\_batch\_rl module +-------------------------------------- + +.. automodule:: reagent.workflow.gym_batch_rl + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.identify\_types\_flow module +--------------------------------------------- + +.. automodule:: reagent.workflow.identify_types_flow + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.result\_registries module +------------------------------------------ + +.. automodule:: reagent.workflow.result_registries + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.result\_types module +------------------------------------- + +.. automodule:: reagent.workflow.result_types + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.spark\_utils module +------------------------------------ + +.. automodule:: reagent.workflow.spark_utils + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.tagged\_union module +------------------------------------- + +.. automodule:: reagent.workflow.tagged_union + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.training module +-------------------------------- + +.. automodule:: reagent.workflow.training + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.training\_reports module +----------------------------------------- + +.. automodule:: reagent.workflow.training_reports + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.types module +----------------------------- + +.. automodule:: reagent.workflow.types + :members: + :undoc-members: + :show-inheritance: + +reagent.workflow.utils module +----------------------------- + +.. automodule:: reagent.workflow.utils + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: reagent.workflow + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/continuous_integration.rst.txt b/_sources/continuous_integration.rst.txt new file mode 100644 index 00000000..53d0036c --- /dev/null +++ b/_sources/continuous_integration.rst.txt @@ -0,0 +1,7 @@ +.. _continuous_integration: + +Continuous Integration +====================== + +We have CI setup on `CircleCI `_. +It's a pretty basic setup. You should follow the local testing instructions in ``.circleci/config.yml``. diff --git a/_sources/distributed.rst.txt b/_sources/distributed.rst.txt index af1da430..7d054b27 100644 --- a/_sources/distributed.rst.txt +++ b/_sources/distributed.rst.txt @@ -12,20 +12,20 @@ How distributed training works With a single GPU and model, training follows this process: 1. Compute the loss from a minibatch of data (the forward pass of the model) -2. Backpropgate that loss through the model to compute gradients (the backward pass of the model) +2. Backpropagate that loss through the model to compute gradients (the backward pass of the model) 3. Sum the gradients across the minibatch. 4. Run the optimizer by calling the "step()" function. Now assume we have several GPUs, and they each have exactly the same model. -#1 and #2 are `embarassingly parallel `_ and can be distribued to many nodes. +#1 and #2 are `embarrassingly parallel `_ and can be distributed to many nodes. As long as we can sum across nodes to complete #3 (this is known as an 'all-reduce'), then each node can run #4 on the same gradients, and the resulting models will again be identical. This is the premise behind distributed training. Training on a single node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Using multiple GPUs on a single node is relatively straightforward. When running either the dqn_workflow or the parametric_dqn_workflow, -set the "use_all_avail_gpus" parameters in the input config (the json file) to true. Horizon will detect the number of available GPUs and +Using multiple GPUs on a single node is relatively straightforward. When running either the dqn_workflow or the parametric_dqn_workflow, +set the "use_all_avail_gpus" parameters in the input config (the json file) to true. ReAgent will detect the number of available GPUs and run on all of them without any additional effort. Training on multiple nodes diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt index 0f571b59..228d7948 100644 --- a/_sources/index.rst.txt +++ b/_sources/index.rst.txt @@ -3,7 +3,7 @@ -.. image:: ../logo/horizon_banner.png +.. image:: ../logo/reagent_banner.png :alt: Banner @@ -11,9 +11,8 @@ ReAgent: Applied Reinforcement Learning Platform ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. image:: https://ci.pytorch.org/jenkins/buildStatus/icon?job=horizon-master - :target: https://ci.pytorch.org/jenkins/job/horizon-master/ - :alt: Build Status +.. image:: https://circleci.com/gh/facebookresearch/ReAgent/tree/master.svg?style=svg + :target: https://circleci.com/gh/facebookresearch/ReAgent/tree/master -------------------------------------------------------------------------------------------------------------------------------------------------------------------- @@ -26,7 +25,7 @@ workflows to train popular deep RL algorithms and includes data preprocessing, f counterfactual policy evaluation, and optimized serving. For more detailed information about ReAgent see the white paper here: `Platform `_. -The source code is available here: `Source code `_. +The source code is available here: `Source code `_. The platform was once named "Horizon" but we have adopted the name "ReAgent" recently to emphasize its broader scope in decision making and reasoning. @@ -59,9 +58,15 @@ License ReAgent is released under a BSD license. Find out more about it here: :ref:`license`. -.. image:: ../logo/horizon_logo.png - :alt: Logo - :width: 200px +Citing +~~~~~~ + +@article{gauci2018horizon, + title={Horizon: Facebook's Open Source Applied Reinforcement Learning Platform}, + author={Gauci, Jason and Conti, Edoardo and Liang, Yitao and Virochsiri, Kittipat and Chen, Zhengxing and He, Yuchen and Kaden, Zachary and Narayanan, Vivek and Ye, Xiaohui}, + journal={arXiv preprint arXiv:1811.00260}, + year={2018} +} Table of Contents ~~~~~~~~~~~~~~~~~~~~~ @@ -77,15 +82,19 @@ Table of Contents :caption: Advanced Topics Distributed Training + Continuous Integration .. toctree:: :caption: Package Reference - Workflow + Evaluation + Models + Prediction Preprocessing + Readers Simulators - Thrift Training + Workflow All Modules .. toctree:: diff --git a/_sources/installation.rst.txt b/_sources/installation.rst.txt index 5d84fefc..cad211b4 100644 --- a/_sources/installation.rst.txt +++ b/_sources/installation.rst.txt @@ -3,168 +3,105 @@ Installation ============ -Anaconda -^^^^^^^^ +ReAgent CLI & Python API +^^^^^^^^^^^^^^^^^^^^^^^^ -First, install anaconda from here (make sure to pick the python 3 version): `Website `_. +We have CLI to launch training & Python API to use programmatically, e.g., in your own script or Jupyter Notebook. +To install this component, you will need to have Python 3.7+ installed on your system. +If you don't have that, you can either install it via `pyenv `_ or +`conda `_. To verify that you have the right version, +type the following command on your shell: -Next, we're going to add some channels that we need for certain software: +.. code-block:: bash + + python --version + +Once you make sure you have the right version, you can simply clone this repo and pip install + +.. code-block:: bash + + git clone https://github.com/facebookresearch/ReAgent.git + cd ReAgent + pip install ".[gym]" + + # install nightly torch (change cpu to cu101/102 if fit) + pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html + +If you don't want need gym dependencies, you can remove :code:`[gym]` + +To verify your setup please run `tox `_. + +.. code-block:: bash + + pip install tox + tox + +Spark preprocessing JAR +^^^^^^^^^^^^^^^^^^^^^^^ + +If you don't want to rebuild the JAR, you can grab `the pre-built version from CircleCI `_, +under `the artifact section of end_to_end_test `_. + +To build from source, you'll need JDK, Scala, & Maven. We will use `SDKMAN! `_ to install them. + +.. code-block:: bash + + curl -s "https://get.sdkman.io" | bash + source "$HOME/.sdkman/bin/sdkman-init.sh" + sdk version + sdk install java 8.0.252.hs-adpt + sdk install scala + sdk install maven + +If you are testing locally, you can also install Spark + +.. code-block:: bash + + sdk install spark 2.4.6 + +Now, you can build our preprocessing JAR + +.. code-block:: bash + + mvn -f preprocessing/pom.xml clean package + +RASP +^^^^ + +RASP (ReAgent Serving Platform) is a decision-serving library. It also has standlone binary. It depends on libtorch, +which cannot be statically linked at the moment. Therefore, we don't have a pre-built version. + +To build the CLI, you'll need `CMake `_ and the following libraries: + +- Nightly build of `libtorch `_ +- `boost `_ +- `gflags `_ +- `glog `_ +- `eigen `_ + +If you don't have those requirements, one easy way to get them is through `conda`. +We recommend `miniconda `_ for this purpose. +If you want to install those requirements via conda, you can type this in the shell: .. code-block:: - conda config --add channels conda-forge # For ONNX/tensorboardX - conda config --add channels pytorch # For PyTorch + conda install --file rasp_requirements.txt -Clone and enter Horizon repo: +To get libtorch, please go to `pytorch `_. +Please make sure to download the "Preview (Nightly)" since our code is tested with that version. +Below, we assumed you put the extracted file at :code:`$HOME/libtorch`. + +You will also need to make sure to init git submodules .. code-block:: - git clone --recurse-submodules https://github.com/facebookresearch/Horizon.git - cd Horizon/ + git submodule update --force --recursive --init --remote -If you already cloned the repo without submodules, they can be added by running this command inside the repository" +Now, you are ready to build .. code-block:: - git submodule update --init --recursive - -Install dependencies: - -.. code-block:: - - conda install --file requirements.txt - -Set JAVA_HOME to the location of your anaconda install - -.. code-block:: - - export JAVA_HOME="$(dirname $(dirname -- `which conda`))" - - echo $JAVA_HOME # Should see something like "/home/jjg/miniconda3" - -Install Spark (the mv command may need to be done as root): - -.. code-block:: - - wget https://archive.apache.org/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz - tar -xzf spark-2.3.3-bin-hadoop2.7.tgz - sudo mv spark-2.3.3-bin-hadoop2.7 /usr/local/spark - -Add the spark bin directory to your path so your terminal can find ``spark-submit``\ : - -.. code-block:: - - export PATH=$PATH:/usr/local/spark/bin - -Install OpenAI Gym if you plan on following our `tutorial `_\ : - -.. code-block:: - - pip install "gym[classic_control,box2d,atari]" - -Download libtorch from https://pytorch.org/get-started/locally/ and extract it to $HOME/libtorch - -As of pytorch 1.3, libtorch is broken on OS/X. To fix (mac only): - -.. code-block:: - - cp ~/miniconda3/lib/libiomp5.dylib $HOME/libtorch/lib/ - -And now, you are ready to install Horizon itself. To install the serving platform: - -.. code-block:: - - mkdir serving/build - cd serving/build - cmake -DCMAKE_PREFIX_PATH=$HOME/libtorch .. - - -Next we must package the models. We use "pip install -e" on the root directory of the repository to create an ephemral package. This means that you can make changes to Horizon and they will be reflected in the package immediately. - -.. code-block:: - - pip install -e . - -At this point, you should be able to run all unit tests: - -.. code-block:: - - pytest - -Docker -^^^^^^ - -We have included a Dockerfile for the CPU-only build and CUDA build under the docker directory. -The CUDA build will need `nvidia-docker `_ to run. - -To build, clone repository and cd into the respective directory: - -.. code-block:: - - git clone https://github.com/facebookresearch/Horizon.git - cd Horizon/ - -On macOS you will need to increase the default memory allocation as the default of 2G is not enough. You can do this by clicking the whale icon in the task bar. We recommend using at least 8G of memory. - -On macOS, you can then build the image: - -.. code-block:: - - docker build -f docker/cpu.Dockerfile -t horizon:dev . - -On Linux you can build the image with specific memory allocations from command line: - -.. code-block:: - - docker build -f docker/cpu.Dockerfile -t horizon:dev --memory=8g --memory-swap=8g . - -To build with CUDA support, use the corresponding dockerfile: - -.. code-block:: - - docker build -f docker/cuda.Dockerfile -t horizon:dev . - -Once the Docker image is built you can start an interactive shell in the container and run the unit tests. To have the ability to edit files locally and have changes be available in the Docker container, mount the local Horizon repo as a volume using the ``-v`` flag. We also add ``-p`` for port mapping so we can view Tensorboard visualizations locally. - -.. code-block:: - - docker run -v $PWD:/home/Horizon -p 0.0.0.0:6006:6006 -it horizon:dev - -To run with GPU, include ``--runtime=nvidia`` after installing `nvidia-docker `_. - -.. code-block:: - - docker run --runtime=nvidia -v $PWD:/home/Horizon -p 0.0.0.0:6006:6006 -it horizon:dev - -If you have SELinux (Fedora, Redhat, etc.) you will have to start docker with the following command (notice the ``:Z`` at the end of path): - -.. code-block:: - - docker run -v $PWD:/home/Horizon:Z -p 0.0.0.0:6006:6006 -it horizon:dev - -To run with GPU, include ``--runtime=nvidia`` after installing `nvidia-docker `_. - -.. code-block:: - - docker run --runtime=nvidia -v $PWD:/home/Horizon:Z -p 0.0.0.0:6006:6006 -it horizon:dev - -Depending on where your local Horizon copy is, you may need to white list your shared path via Docker -> Preferences... -> File Sharing. - -Once inside the container, run the setup file: - -.. code-block:: - - cd Horizon - ./scripts/setup.sh - -Now you can run all the tests: - -.. code-block:: - - python setup.py test - -or try running one specific test: - -.. code-block:: - - python setup.py test -s ml.rl.test.constant_reward.test_constant_reward.TestConstantReward.test_trainer_maxq + mkdir -p serving/build + cd serving/build + cmake -DCMAKE_PREFIX_PATH=$HOME/libtorch -DCMAKE_CXX_STANDARD=17 .. + make diff --git a/_sources/license.rst.txt b/_sources/license.rst.txt index 9136877b..ade8cacc 100644 --- a/_sources/license.rst.txt +++ b/_sources/license.rst.txt @@ -5,7 +5,7 @@ License BSD License -For Horizon software +For ReAgent software Copyright (c) 2017-present, Facebook, Inc. All rights reserved. diff --git a/_sources/rasp_tutorial.rst.txt b/_sources/rasp_tutorial.rst.txt index 2c87d75e..39ed2850 100644 --- a/_sources/rasp_tutorial.rst.txt +++ b/_sources/rasp_tutorial.rst.txt @@ -43,7 +43,7 @@ we will always show bacon (with it’s score of 1.1) and never show ribs (with a score of 0.9). This means we will never know the true performance of recommending ribs and can’t improve our system in the future. This is known as the cold-start or explore-exploit problem -(TODO: Citations). +( https://arxiv.org/abs/1812.00116 ). To avoid that problem, we will use the SoftmaxRanker, which will show bacon 52% of the time and ribs 48% of the time. The SoftmaxRanker @@ -52,11 +52,11 @@ operator is based on the softmax function: :: >>> import numpy as np - >>> + >>> >>> def softmax(x): ... e_x = np.exp(x - np.max(x)) ... return e_x / e_x.sum() - ... + ... >>> print(softmax([1.1, 1.0])) [0.52497919 0.47502081] @@ -121,11 +121,6 @@ We will be using the built-in web service directly for this tutorial. The simulator code can be found at: serving/examples/ecommerce/customer_simulator.py -Here is our RP config file. This tells RP where to find decision plans -and models (coming later): - -(RP config) - Makin’ bacon ------------ @@ -133,7 +128,7 @@ In one terminal window, start the RP server: :: - ➜ ReAgent git:(master) ✗ ./serving/build/RaspCli --logtostderr + ➜ ./serving/build/RaspCli --logtostderr I1014 17:23:19.736086 457250240 DiskConfigProvider.cpp:10] READING CONFIGS FROM serving/examples/ecommerce/plans I1014 17:23:19.738142 457250240 DiskConfigProvider.cpp:42] GOT CONFIG multi_armed_bandit.json AT serving/examples/ecommerce/plans/multi_armed_bandit.json I1014 17:23:19.738286 457250240 DiskConfigProvider.cpp:46] Registered decision config: multi_armed_bandit.json @@ -144,12 +139,11 @@ In one terminal window, start the RP server: I1014 17:23:19.739843 131715072 Server.cpp:58] STARTING SERVER Then in another, run our simulator. The simulator will spawn many -threads and call RASP 10,000 times (this will take a few minutes to -complete): +threads and call RASP 1,000 times: :: - ➜ ReAgent git:(master) ✗ python serving/examples/ecommerce/customer_simulator.py heuristic.json + ➜ python serving/examples/ecommerce/customer_simulator.py heuristic.json 0 200 100 @@ -164,21 +158,20 @@ complete): Action Distribution: {'Ribs': 471, 'Bacon': 529} As expected, we recommend Bacon 52% of the time and Ribs 48% of the -time. We get an average reward (in this case, average # of clicks) of -0.3555. +time. We get an average reward (in this case, average # of clicks) of about 0.36. This is our baseline performance, but can we do better? From the log, we can see that more bacon recommendations were clicked on: :: - ➜ ReAgent git:(master) ✗ cat /tmp/rasp_logging/log.txt | grep '"name":"Ribs"}]' | grep '"reward":0.0' | wc -l + ➜ cat /tmp/rasp_logging/log.txt | grep '"name":"Ribs"}]' | grep '"reward":0.0' | wc -l 390 # Ribs not clicked - ➜ ReAgent git:(master) ✗ cat /tmp/rasp_logging/log.txt | grep '"name":"Ribs"}]' | grep '"reward":1.0' | wc -l + ➜ cat /tmp/rasp_logging/log.txt | grep '"name":"Ribs"}]' | grep '"reward":1.0' | wc -l 88 # Ribs clicked - ➜ ReAgent git:(master) ✗ cat /tmp/rasp_logging/log.txt | grep '"name":"Bacon"}]' | grep '"reward":1.0' | wc -l + ➜ cat /tmp/rasp_logging/log.txt | grep '"name":"Bacon"}]' | grep '"reward":1.0' | wc -l 266 # Bacon clicked - ➜ ReAgent git:(master) ✗ cat /tmp/rasp_logging/log.txt | grep '"name":"Bacon"}]' | grep '"reward":0.0' | wc -l + ➜ cat /tmp/rasp_logging/log.txt | grep '"name":"Bacon"}]' | grep '"reward":0.0' | wc -l 253 # Bacon not clicked This makes sense since, from our simulator definition, most people @@ -197,7 +190,7 @@ Generates this plan: :: - ➜ ReAgent git:(master) ✗ cat serving/examples/ecommerce/plans/multi_armed_bandit.json + ➜ cat serving/examples/ecommerce/plans/multi_armed_bandit.json { "operators": [ { @@ -232,7 +225,7 @@ Running with this new plan gives: :: - ➜ ReAgent git:(master) ✗ python serving/examples/ecommerce/customer_simulator.py multi_armed_bandit.json + ➜ python serving/examples/ecommerce/customer_simulator.py multi_armed_bandit.json 0 200 100 @@ -252,7 +245,7 @@ again: :: - ➜ ReAgent git:(master) ✗ python serving/examples/ecommerce/customer_simulator.py multi_armed_bandit.json + ➜ python serving/examples/ecommerce/customer_simulator.py multi_armed_bandit.json 0 200 100 @@ -291,8 +284,8 @@ instance of RaspCli: … I1014 17:45:36.613893 6602752 Server.cpp:58] STARTING SERVER ^C - ➜ ReAgent git:(master) ✗ rm /tmp/rasp_logging/log.txt - ➜ ReAgent git:(master) ✗ ./serving/build/RaspCli --logtostderr + ➜ rm /tmp/rasp_logging/log.txt + ➜ ./serving/build/RaspCli --logtostderr I1014 17:48:49.674149 144418240 DiskConfigProvider.cpp:10] READING CONFIGS FROM serving/examples/ecommerce/plans I1014 17:48:49.678155 144418240 DiskConfigProvider.cpp:42] GOT CONFIG multi_armed_bandit.json AT serving/examples/ecommerce/plans/multi_armed_bandit.json I1014 17:48:49.679606 144418240 DiskConfigProvider.cpp:46] Registered decision config: multi_armed_bandit.json @@ -308,14 +301,14 @@ Now let’s run the heuristic model a few times to generate enough data :: - ➜ ReAgent git:(master) ✗ for run in {1..10}; do python serving/examples/ecommerce/customer_simulator.py heuristic.json; done + ➜ for run in {1..10}; do python serving/examples/ecommerce/customer_simulator.py heuristic.json; done 0 200 ... 900 Average reward: 0.36 Action Distribution: {'Bacon': 516, 'Ribs': 484} - ➜ ReAgent git:(master) ✗ wc -l /tmp/rasp_logging/log.txt + ➜ wc -l /tmp/rasp_logging/log.txt 10000 /tmp/rasp_logging/log.txt RASP’s logging format and the ReAgent models’ input format is slightly @@ -323,70 +316,43 @@ different. Fortunately, there’s a tool to convert from one to the other: :: - ➜ ReAgent git:(master) ✗ python serving/scripts/rasp_to_model.py /tmp/rasp_logging/log.txt ecom_cb_input_data/input.json - ➜ ReAgent git:(master) ✗ wc -l ecom_cb_input_data/input.json - 10000 ecom_cb_input_data/input.json + ➜ python serving/scripts/rasp_to_model.py /tmp/rasp_logging/log.txt /tmp/input_df.pkl + ... + INFO:__main__: ds mdp_id sequence_number state_features action reward action_probability possible_actions metrics + 0 2019-01-01 1287515757457242569 0 {0: 0.0, 1: 1.0} Ribs 0.0 0.475021 [Bacon, Ribs] {'reward': 0.0} + 1 2019-01-01 -1441171268272508658 0 {0: 0.0, 1: 1.0} Ribs 0.0 0.475021 [Bacon, Ribs] {'reward': 0.0} + 2 2019-01-01 -267723109738500267 0 {0: 0.0, 1: 1.0} Bacon 1.0 0.524979 [Bacon, Ribs] {'reward': 1.0} + 3 2019-01-01 7619952535038766490 0 {0: 0.0, 1: 1.0} Ribs 0.0 0.475021 [Bacon, Ribs] {'reward': 0.0} + 4 2019-01-01 -2393212434904546228 0 {0: 0.0, 1: 1.0} Bacon 0.0 0.524979 [Bacon, Ribs] {'reward': 0.0} + Since we are using the contextual bandit or RL model, we need to build a timeline: :: - rm -Rf spark-warehouse derby.log metastore_db preprocessing/spark-warehouse preprocessing/metastore_db preprocessing/derby.log ; /usr/local/spark/bin/spark-submit \ - --class com.facebook.spark.rl.Preprocessor preprocessing/target/rl-preprocessing-1.1.jar \ - "`cat serving/examples/ecommerce/training/timeline.json`" - ... - 2019-10-14 19:04:18 INFO ShutdownHookManager:54 - Shutdown hook called - 2019-10-14 19:04:18 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/jm/snmq7xfn7llc1tpnjgn7889h6l6pkw/T/spark-2b6a4171-cb60-4d5e-8052-87620a0677a2 - 2019-10-14 19:04:18 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/jm/snmq7xfn7llc1tpnjgn7889h6l6pkw/T/spark-927dae4a-6613-4a28-9d88-4d43a03d1cf3 - ➜ ReAgent git:(master) ✗ + # Set the config + ➜ export CONFIG=serving/examples/ecommerce/training/contextual_bandit.yaml -The spark job creates a directory full of files, so we must merge into -one file for training & evaluation: + # First clean up derby database from last run + ➜ rm -Rf spark-warehouse derby.log metastore_db preprocessing/spark-warehouse preprocessing/metastore_db preprocessing/derby.log + + # Run timeline operator + ➜ ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.timeline_operator "$CONFIG" + +The `Click `_ command submits a Spark job that uploads the timeline table to Hive. +Now we can train the contextual bandit. :: - ➜ ReAgent git:(master) ✗ mkdir -p training_data - ➜ ReAgent git:(master) ✗ cat ecom_cb_training/part* > training_data/train.json - ➜ ReAgent git:(master) ✗ cat ecom_cb_eval/part* > training_data/eval.json + ➜ ./reagent/workflow/cli.py run reagent.workflow.training.identify_and_train_network "$CONFIG" + ... + I0524 112136.208 model_manager.py:213] Saved torchscript model to model_1590344496.torchscript -Now we run our normalization. Any time we use a deep neural network, we -need normalization to prevent some large features from drowning others. - -:: - - ➜ ReAgent git:(master) ✗ python ml/rl/workflow/create_normalization_metadata.py -p serving/examples/ecommerce/training/cb_train.json - - WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode. - INFO:ml.rl.preprocessing.normalization:Got feature: 0 - INFO:ml.rl.preprocessing.normalization:Feature 0 normalization: NormalizationParameters(feature_type='BINARY', boxcox_lambda=None, boxcox_shift=0.0, mean=0.0, stddev=1.0, possible_values=None, quantiles=None, min_value=0.0, max_value=1.0) - INFO:ml.rl.preprocessing.normalization:Got feature: 1 - INFO:ml.rl.preprocessing.normalization:Feature 1 normalization: NormalizationParameters(feature_type='BINARY', boxcox_lambda=None, boxcox_shift=0.0, mean=0.0, stddev=1.0, possible_values=None, quantiles=None, min_value=1.0, max_value=1.0) - INFO:__main__:`state_features` normalization metadata written to training_data/state_features_norm.json - -Now we can train our contextual bandit: - -:: - - ➜ ReAgent git:(master) ✗ rm -Rf "outputs/*" ; python ml/rl/workflow/dqn_workflow.py -p serving/examples/ecommerce/training/cb_train.json - INFO:ml.rl.json_serialize:TYPE: - INFO:ml.rl.json_serialize:{'gamma': 0.0, 'target_update_rate': 1.0, 'maxq_learning': True, 'epsilon': 0.2, 'temperature': 0.35, 'softmax_policy': 0} - ... - INFO:ml.rl.workflow.page_handler:CPE evaluation took 0.26366519927978516 seconds. - INFO:ml.rl.workflow.base_workflow:Training finished. Processed ~6555 examples / s. - INFO:ml.rl.preprocessing.preprocessor:CUDA availability: False - INFO:ml.rl.preprocessing.preprocessor:NOT Using GPU: GPU not requested or not available. - /Users/jjg/github/Horizon/ml/rl/preprocessing/preprocessor.py:546: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! - elif max_value.gt(MAX_FEATURE_VALUE): - /Users/jjg/github/Horizon/ml/rl/preprocessing/preprocessor.py:552: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! - elif min_value.lt(MIN_FEATURE_VALUE): - INFO:__main__:Saving PyTorch trainer to outputs/trainer_1571105504.pt - INFO:ml.rl.workflow.base_workflow:Saving TorchScript predictor to outputs/model_1571105504.torchscript - -At this point, we have a model in ``outputs/model_*.torchscript``. We -are going to combine this scoring model with an e-greedy ranker. The -e-greedy ranker chooses the best actions most of the time, but sometimes -chooses random actions to explore: +At this point, we have a model saved at ``model_*.torchscript``. We +are going to combine this scoring model with an Softmax ranker. The +ranker chooses the best actions most of the time, but rarely +chooses other actions to explore: :: @@ -439,14 +405,14 @@ put the model there so we can find it: :: - ➜ ReAgent git:(master) ✗ mkdir -p /tmp/0 - ➜ ReAgent git:(master) ✗ cp outputs/model_*.torchscript /tmp/0/0 + ➜ mkdir -p /tmp/0 + ➜ cp model_*.torchscript /tmp/0/0 Let’s run with our model: :: - ➜ ReAgent git:(master) ✗ python serving/examples/ecommerce/customer_simulator.py contextual_bandit.json + ➜ python serving/examples/ecommerce/customer_simulator.py contextual_bandit.json 0 200 100 @@ -472,4 +438,3 @@ All of the decisions made so far have been pointwise: we don’t consider repeat visitors. ReAgent can also optimize for long-term value in sequential decisions using reinforcement learning, but that is out of the scope of this starting tutorial. - diff --git a/_sources/usage.rst.txt b/_sources/usage.rst.txt index 8f03e488..edc69569 100644 --- a/_sources/usage.rst.txt +++ b/_sources/usage.rst.txt @@ -3,25 +3,61 @@ Usage ===== -Horizon is designed for large-scale, distributed recommendation/optimization tasks where we don't +ReAgent is designed for large-scale, distributed recommendation/optimization tasks where we don't have access to a simulator. In this environment, it's typically better to train offline on batches of data, and release new policies slowly over time. Because the policy updates slowly and in batches, we use *off-policy* algorithms. To test a new policy without deploying it, we rely on *counter-factual policy evaluation (CPE)*\ , a set of techniques for estimating a policy based on the actions of another policy. -Before we get started using Horizon as it is intended, let's begin with a traditional RL setup with a simulator where we can trivially evaluate new policies: +Quick Start +----------- + +We have set up `Click `_ commands to run our RL workflow. The basic usage pattern is + +.. code-block:: + + ./reagent/workflow/cli.py run + + +To train a model online with OpenAI Gym, simply run the Click command: + +.. code-block:: + + # set the config + export CONFIG=reagent/gym/tests/configs/cartpole/discrete_dqn_cartpole_online.yaml + # train and evaluate model on gym environment + ./reagent/workflow/cli.py run reagent.gym.tests.test_gym.run_test $CONFIG + + +To train a batch RL model, run the following commands: + +.. code-block:: + + # set the config + export CONFIG=reagent/workflow/sample_configs/discrete_dqn_cartpole_offline.yaml + # gather some random transitions (can replace with your own) + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.offline_gym $CONFIG + # convert data to timeline format + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.timeline_operator $CONFIG + # train model based on timeline data + ./reagent/workflow/cli.py run reagent.workflow.training.identify_and_train_network $CONFIG + # evaluate the model + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.evaluate_gym "$CONFIG" + + +Now we will describe how the above commands work, starting with a traditional RL setup with a simulator where we can trivially evaluate new policies: 1 - On-Policy RL Training ------------------------- -Open AI Gym is a set of environments: simulators that can run policies for a given task and generate rewards. If a simulator is accessible, on-policy training (where the latest version of the policy makes new decisions in real-time) can give better results. To train a model on OpenAI Gym, simply run: +OpenAI Gym is a set of environments: simulators that can run policies for a given task and generate rewards. If a simulator is accessible, on-policy training (where the latest version of the policy makes new decisions in real-time) can give better results. We have a suite of benchmarks on OpenAI Gym, which is listed in ``reagent/gym/tests/test_gym.py``'s ``GYM_TESTS``. To train a model on OpenAI Gym, simply run the Click command: .. code-block:: - python ml/rl/test/gym/run_gym.py -p ml/rl/test/gym/discrete_dqn_cartpole_v0.json + ./reagent/workflow/cli.py run reagent.gym.tests.test_gym.run_test reagent/gym/tests/configs/cartpole/discrete_dqn_cartpole_online.yaml -Configs for different environments and algorithms can be found in ``ml/rl/test/gym/``. +Configs for different environments and algorithms can be found in ``reagent/gym/tests/configs//__online.yaml``. While this is typically the set up for people conducting RL research, it isn't always practical to deploy on-policy RL for several reasons: @@ -31,52 +67,47 @@ While this is typically the set up for people conducting RL research, it isn't a #. We want to evaluate the behavior of the policy offline and then keep the policy constant afterwards to reduce the risk that the policy will degrade at odd hours. #. We are building on top of traditional recommender systems that typically rely on a fixed, stochastic policy. -For these reasons, Horizon is designed to support batch, off-policy RL. Let's now walk though how to train a model with Horizon: +For these reasons, ReAgent is designed to support batch, off-policy RL. Let's now walk though how to train a model with ReAgent: 2- Offline RL Training (Batch RL) --------------------------------- -The main use case of Horizon is to train RL models in the **batch** setting. In batch reinforcement learning the data collection and policy learning steps are decoupled. Specifically, we try to learn the best possible policy given the input data. In batch RL, being able to handle thousands of varying feature types and distributions and algorithm performance estimates before deployment are of key importance. +The main use case of ReAgent is to train RL models in the **batch** setting. In batch reinforcement learning the data collection and policy learning steps are decoupled. Specifically, we try to learn the best possible policy given the input data. In batch RL, being able to handle thousands of varying feature types and distributions and algorithm performance estimates before deployment are of key importance. + +In this example, we will train a DQN model on Offline ``CartPole-v0`` data, where Click command config should be set to + +.. code-block:: + + export CONFIG=reagent/workflow/sample_configs/discrete_dqn_cartpole_offline.yaml + + +We now proceed to give pseudo-code to sketch out the main ideas of our batch RL workflow. -In this example, we will train a DQN model on Offline ``Cartpole-v0`` data: Step 1 - Create training data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -First we need to generate the data required to train our RL models. For this example we generate data from the ``Cartpole-v0`` environment in OpenAI Gym. In practice, end users would generate a dataset in a similar format from their production system. +We first generate data from a random policy (chooses random actions) run on the ``CartPole-v0`` environment. +In particular, the following Click command runs 150 episodes of ``CartPole-v0`` (max steps of 200) and stored the pickled dataframe in ``/tmp/tmp_pickle.pkl``, which you may inspect via ``pd_df = pd.read_pickle(pkl_path)``. .. code-block:: - mkdir cartpole_discrete + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.offline_gym $CONFIG - python ml/rl/test/gym/run_gym.py -p ml/rl/test/gym/discrete_dqn_cartpole_v0.json -f cartpole_discrete/training_data.json - -Let's look at one row of data to see the expected input format: +The command essentially performs the following pseudo-code: .. code-block:: - cat cartpole_discrete/training_data.json | head -n1 | python -m json.tool + dataset = ReplayBuffer() + for epoch in range(num_episodes_for_data_batch): + run_episode & store transitions - { - "mdp_id": "10", - "sequence_number": 0, - "state_features": { - "2": 0.0021880944, - "1": -0.015781501, - "0": -0.031933542, - "3": 0.04611974 - }, - "action": "0", - "reward": 1.0, - "possible_actions": [ - "0", - "1" - ], - "action_probability": 0.9, - "ds": "2018-06-25" - } + df = dataset.to_pandas_df() + df.to_pickle(pkl_path) -The input data is a flat file containing a JSON object per-line separated by newlines (the first line is pretty-printed here for readability). This is human-readable, but not the most efficient way to store tabular data. Other ways to store input data is parquet, CSV, or any other format that can be read by Apache Spark. All of these formats are fine, as long as the following schema is maintained: +In practice, end users would generate a dataset in a similar format from their production system. For this example, the data is stored as a pickled Pandas dataframe. + +This is human-readable, but not the most efficient way to store tabular data. Other ways to store input data are parquet, CSV, or any other format that can be read by Apache Spark. All of these formats are fine, as long as the following schema is maintained: .. list-table:: :header-rows: 1 @@ -110,16 +141,14 @@ The input data is a flat file containing a JSON object per-line separated by new - A unique ID for this dataset. -Note that JSON does not support integer keys in objects so in our JSON format we replace the ``map`` with ``map``\ , but even in this case the keys must be strings of integers. +Once you have data on this format, you can move on to Step 2. -Once you have data on this format (or you have generated data using our gym script) you can move on to step 2: - -Step 2 - Convert the data to the ``timeline`` format +Step 2 - Convert the data to the ``Timeline`` format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Models are trained on consecutive pairs of state/action tuples. To assist in creating this table, we have an ``RLTimelineOperator`` spark operator. Let's build and run the timeline operator on the data: -First, we need to build the Spark library that will execute the timeline. Apache Spark is a platform for doing massively-parallel processing. Although we are running this on a single file, Spark is designed to work on thousands of files distribued across many machines. Explaining HDFS, Hive, and Spark are beyond the scope of this tutorial, but for large datasets it's important to understand these concepts and that it's possible to run Horizon in a distributed environment by simply changing the location of the input from a file to an HDFS folder. +First, we need to build the Spark library that will execute the timeline. Apache Spark is a platform for doing massively-parallel processing. Although we are running this on a single file, Spark is designed to work on thousands of files distribued across many machines. Explaining HDFS, Hive, and Spark are beyond the scope of this tutorial, but for large datasets it's important to understand these concepts and that it's possible to run ReAgent in a distributed environment by simply changing the location of the input from a file to an HDFS folder. .. code-block:: @@ -133,49 +162,56 @@ When running spark locally, spark creates a fake "cluster" where it stores all o # Clear last run's spark data (in case of interruption) rm -Rf spark-warehouse derby.log metastore_db preprocessing/spark-warehouse preprocessing/metastore_db preprocessing/derby.log -Now that we are ready, let's run our spark job on our local machine. This will produce a massive amount of logging (because we are running many systems that typically are distributed across many nodes) and there will be some exception stack traces printed because we are running in a psuedo-distributed mode. Generally this is fine as long as the output data is generated: +Now that we are ready, let's run our spark job on our local machine. This will produce a massive amount of logging (because we are running many systems that typically are distributed across many nodes) and there will be some exception stack traces printed because we are running in a psuedo-distributed mode. Generally this is fine as long as the output data is generated. To do so, run the following Click command: .. code-block:: - # Run timelime on pre-timeline data - /usr/local/spark/bin/spark-submit \ - --class com.facebook.spark.rl.Preprocessor preprocessing/target/rl-preprocessing-1.1.jar \ - "`cat ml/rl/workflow/sample_configs/discrete_action/timeline.json`" + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.timeline_operator $CONFIG - # Look at the first row of training & eval - head -n1 cartpole_discrete_training/part* - head -n1 cartpole_discrete_eval/part* - -There are many output files. The reason for this is that Spark expects many input & output files: otherwise it wouldn't be able to efficiently run on many machines and output data in parallel. For this tutorial, we will merge all of this data into a single file, but in a production use-case we would be streaming data from HDFS during training. +The command essentially performs the following pseudo-code: .. code-block:: - # Merge output data to single file - mkdir training_data - cat cartpole_discrete_training/part* > training_data/cartpole_discrete_timeline.json - cat cartpole_discrete_eval/part* > training_data/cartpole_discrete_timeline_eval.json + # load pandas dataframe + pd_df = pd.read_pickle(pkl_path) - # Remove the output data folder - rm -Rf cartpole_discrete_training cartpole_discrete_eval + # convert to Spark dataframe + spark = get_spark_session() + df = spark.createDataFrame(pd_df) -Now that all of our data has been grouped into consecutive pairs, we can run the normalization pipeline. + # run timelime operator + json_params = make_input_to_timeline_operator() + spark._jvm.com.facebook.spark.rl.Timeline.main(json_params) -Step 3 - Create the normalization parameters + +Now that our data is a Spark table in Hive storage, we're ready to run the training workflow (Steps 3-4). These steps are altogether accomplished with the following Click command: + +.. code-block:: + + ./reagent/workflow/cli.py run reagent.workflow.training.identify_and_train_network $CONFIG + + +We now proceed to describing this command and present some pseudo-code. + + +Step 3 - Determine normalization parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Data from production systems is often sparse, noisy and arbitrarily distributed. Literature has shown that neural networks learn faster and better when operating on batches of features that are normally distributed. Horizon includes a workflow that automatically analyzes the training dataset and determines the best transformation function and corresponding normalization parameters for each feature. We can run this workflow on the post timeline data: +Data from production systems is often sparse, noisy and arbitrarily distributed. Literature has shown that neural networks learn faster and better when operating on batches of features that are normally distributed. ReAgent includes a workflow that automatically analyzes the training dataset and determines the best transformation function and corresponding normalization parameters for each feature. We do this via ``ModelManager.run_feature_identification``, where ``input_table_spec`` points to a Spark table with the timeline data. .. code-block:: - python ml/rl/workflow/create_normalization_metadata.py -p ml/rl/workflow/sample_configs/discrete_action/dqn_example.json + model: ModelManager__Union + manager = model.value + manager.run_feature_identification(input_table_spec) -Now we can look at the normalization file. It's a JSON file where each key is a feature id and each value is a string-encoded JSON object describing the normalization: + +The normalization is a Python dictionary where each key is a feature id and each value is NormalizationData. +An example of this, in JSON format, is .. code-block:: - cat training_data/state_features_norm.json | python -m json.tool - { "0": "{\"feature_type\":\"CONTINUOUS\",\"mean\":0.5675003528594971,\"stddev\":1.0,\"min_value\":-0.1467551738023758,\"max_value\":2.1779561042785645}", "1": "{\"feature_type\":\"CONTINUOUS\",\"mean\":0.42259514331817627,\"stddev\":1.0,\"min_value\":-1.3586808443069458,\"max_value\":1.8529225587844849}", @@ -183,28 +219,76 @@ Now we can look at the normalization file. It's a JSON file where each key is a "3": "{\"feature_type\":\"CONTINUOUS\",\"mean\":0.02947876788675785,\"stddev\":1.0,\"min_value\":-2.194336175918579,\"max_value\":2.164193868637085}" } +NB: ``reagent/workflow/training.py`` is what the pseudo-code in Steps 3 and 4 are trying to depict. Models should subclass ``ModelManager`` and implement all abstract methods (including ``run_feature_identification`` and ``query_data``) to be added to our registry of models. + Step 4 - Train model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To train the model, we first save our Spark table to Parquet format, and use `Petastorm `_'s PyTorch DataLoader, which can efficiently read Parquet formatted data. We do this via ``ModelManager.query_data``, which each ``ModelManager`` in our registry of models must implement. In this step, we also process the rewards, i.e. computing multi-step rewards or computing the reward from ``metrics`` columns directly. + +.. code-block:: + train_dataset = manager.query_data( + input_table_spec=input_table_spec, # description of Spark table + sample_range=train_sample_range, # what percentage of data to use for training + reward_options=reward_options, # config to calculate rewards + ) + # train_dataset now points to a Parquet + Now we are ready to train a model by running: .. code-block:: - # Store model outputs here - mkdir outputs + # make preprocessor from the normalization parameters of Step 3 + batch_preprocessor = manager.build_batch_preprocessor() - python ml/rl/workflow/dqn_workflow.py -p ml/rl/workflow/sample_configs/discrete_action/dqn_example.json + # read preprocessed data + data_reader = petastorm.make_batch_reader(train_dataset.parquet_url) + with DataLoader(data_reader, batch_preprocessor) as dataloader: + for batch in dataloader: + trainer.train(batch) -Note that, even in the OpenAI Gym case, we aren't running the gym at this step. We are taking a batch of data that we generated previously and training by looping over that data and interatively learning a better policy than the policy that generated the data. + # Store model outputs + torchscript_output_path = f"model_{round(time.time())}.torchscript" + serving_module = manager.build_serving_module() + torch.jit.save(serving_module, torchscript_output_path) + + # store for later use + training_output.output_path = torchscript_output_path + +Note that the model is trained purely on the randomly generated data we collected in Step 1. +We are taking a batch of data that we generated previously and training by looping over that data and interatively learning a better policy than the policy that generated the data. +Effectively, this is learning to perform a task by observing completely random transitions from an environment! While doing so, we are not even building a dynamics model of the environment. + +NB: We can do the same for the ``eval_dataset`` if we want to perform CPE during training as a diagnosis tool. Step 5 - Evaluate the Model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Now that we have trained a new policy on the offline ``Cartpole-v0`` data, we can try it out to see how it does: +Now that we have trained a new policy on the offline ``CartPole-v0`` data, we can try it out to see how it does: .. code-block:: - python ml/rl/test/workflow/eval_cartpole.py -m outputs/predictor* + ./reagent/workflow/cli.py run reagent.workflow.gym_batch_rl.evaluate_gym $CONFIG + +which performs the following pseudo-code + +.. code-block:: + + # load our previous serving module + jit_model = torch.jit.load(saved_serving_module) + + # wrap around module to fit our gymrunner interface + policy = create_predictor_policy_from_model(env, jit_model) + agent = Agent.create_for_env_with_serving_policy(env, policy=policy) + + # run Agent on environment, and record rewards + rewards = evaluate_for_n_episodes( + n=num_eval_episodes, env=env, agent=agent, max_steps=max_steps + ) + + +Even on completely random data, DQN can learn a policy that obtains scores close to the maximum possible score of 200! + Step 6 - Visualize Results via Tensorboard ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/api/ml.html b/api/ml.html index b0b60d43..53ca62cc 100644 --- a/api/ml.html +++ b/api/ml.html @@ -8,7 +8,7 @@ - ml package — Horizon 1.0 documentation + ml package — ReAgent 1.0 documentation @@ -34,9 +34,7 @@ - - - + @@ -50,7 +48,7 @@ - Horizon + ReAgent @@ -89,25 +87,19 @@

Advanced Topics

Package Reference

-