gum::AdaptiveRMaxPlaner Class Reference

<agrum/FMDP/planning/adaptiveRMaxPlaner.h> More...

#include <adaptiveRMaxPlaner.h>

Inheritance diagram for gum::AdaptiveRMaxPlaner:

Collaboration diagram for gum::AdaptiveRMaxPlaner:

Public Member Functions
Planning Methods
void	initialize (const FMDP< double > *fmdp)
	Initializes data structure needed for making the planning. More...
void	makePlanning (Idx nbStep=1000000)
	Performs a value iteration. More...
Datastructure access methods
INLINE const FMDP< double > *	fmdp ()
	Returns a const ptr on the Factored Markov Decision Process on which we're planning. More...
INLINE const MultiDimFunctionGraph< double > *	vFunction ()
	Returns a const ptr on the value function computed so far. More...
virtual Size	vFunctionSize ()
	Returns vFunction computed so far current size. More...
INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy ()
	Returns the best policy obtained so far. More...
virtual Size	optimalPolicySize ()
	Returns optimalPolicy computed so far current size. More...
std::string	optimalPolicy2String ()
	Provide a better toDot for the optimal policy where the leaves have the action name instead of its id. More...

Static Public Member Functions
static AdaptiveRMaxPlaner *	ReducedAndOrderedInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static AdaptiveRMaxPlaner *	TreeInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	spumddInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	sviInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)

Protected Attributes
const FMDP< double > *	fmdp_
	The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ) More...
MultiDimFunctionGraph< double > *	vFunction_
	The Value Function computed iteratively. More...
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy_
	The associated optimal policy. More...
Set< const DiscreteVariable * >	elVarSeq_
	A Set to eleminate primed variables. More...
double	discountFactor_
	Discount Factor used for infinite horizon planning. More...
IOperatorStrategy< double > *	operator_
bool	verbose_
	Boolean used to indcates whether or not iteration informations should be displayed on terminal. More...

Protected Member Functions
Value Iteration Methods
virtual void	initVFunction_ ()
	Performs a single step of value iteration. More...
virtual MultiDimFunctionGraph< double > *	valueIteration_ ()
	Performs a single step of value iteration. More...
Optimal policy extraction methods
virtual void	evalPolicy_ ()
	Perform the required tasks to extract an optimal policy. More...
Value Iteration Methods
virtual MultiDimFunctionGraph< double > *	evalQaction_ (const MultiDimFunctionGraph< double > *, Idx)
	Performs the P(s'|s,a).V^{t-1}(s') part of the value itération. More...
virtual MultiDimFunctionGraph< double > *	maximiseQactions_ (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs max_a Q(s,a) More...
virtual MultiDimFunctionGraph< double > *	minimiseFunctions_ (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs min_i F_i. More...
virtual MultiDimFunctionGraph< double > *	addReward_ (MultiDimFunctionGraph< double > *function, Idx actionId=0)
	Perform the R(s) + gamma . function. More...
Optimal policy extraction methods
MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	makeArgMax_ (const MultiDimFunctionGraph< double > *Qaction, Idx actionId)
	Creates a copy of given Qaction that can be exploit by a Argmax. More...
virtual MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	argmaximiseQactions_ (std::vector< MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *> &)
	Performs argmax_a Q(s,a) More...
void	extractOptimalPolicy_ (const MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
	From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet. More...

Constructor & destructor.
	AdaptiveRMaxPlaner (IOperatorStrategy< double > *opi, double discountFactor, double epsilon, const ILearningStrategy *learner, bool verbose)
	Default constructor. More...
	~AdaptiveRMaxPlaner ()
	Default destructor. More...

Incremental methods
HashTable< Idx, StatesCounter *>	_counterTable_
HashTable< Idx, bool >	_initializedTable_
bool	_initialized_
void	checkState (const Instantiation &newState, Idx actionId)

Incremental methods
void	setOptimalStrategy (const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *optPol)
virtual ActionSet	stateOptimalPolicy (const Instantiation &curState)
const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optPol_
ActionSet	allActions_

Detailed Description

<agrum/FMDP/planning/adaptiveRMaxPlaner.h>

A class to find optimal policy for a given FMDP.

Perform a RMax planning on given in parameter factored markov decision process

Definition at line 53 of file adaptiveRMaxPlaner.h.

Constructor & Destructor Documentation

AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::AdaptiveRMaxPlaner ( IOperatorStrategy< double > *  opi, double  discountFactor, double  epsilon, const ILearningStrategy *  learner, bool  verbose  )

private

Default constructor.

Definition at line 63 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                               :
      StructuredPlaner(opi, discountFactor, epsilon, verbose),
      IDecisionStrategy(), _fmdpLearner_(learner), _initialized_(false) {
    GUM_CONSTRUCTOR(AdaptiveRMaxPlaner);
  }

Here is the call graph for this function:

~AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::~AdaptiveRMaxPlaner

(

)

Default destructor.

Definition at line 76 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                          {
    GUM_DESTRUCTOR(AdaptiveRMaxPlaner);

    for (HashTableIteratorSafe< Idx, StatesCounter* > scIter = _counterTable_.beginSafe();
         scIter != _counterTable_.endSafe();
         ++scIter)
      delete scIter.val();
  }

Here is the call graph for this function:

Member Function Documentation

_clearTables_()

void gum::AdaptiveRMaxPlaner::_clearTables_ ( )

private

Definition at line 321 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                         {
    for (auto actionIter = this->fmdp()->beginActions(); actionIter != this->fmdp()->endActions();
         ++actionIter) {
      delete _actionsBoolTable_[*actionIter];
      delete _actionsRMaxTable_[*actionIter];
    }
    _actionsRMaxTable_.clear();
    _actionsBoolTable_.clear();
  }

Here is the call graph for this function:

_makeRMaxFunctionGraphs_()

void gum::AdaptiveRMaxPlaner::_makeRMaxFunctionGraphs_ ( )

private

Definition at line 222 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                    {
    _rThreshold_ = _fmdpLearner_->modaMax() * 5 > 30 ? _fmdpLearner_->modaMax() * 5 : 30;
    _rmax_       = _fmdpLearner_->rMax() / (1.0 - this->discountFactor_);

    for (auto actionIter = this->fmdp()->beginActions(); actionIter != this->fmdp()->endActions();
         ++actionIter) {
      std::vector< MultiDimFunctionGraph< double >* > rmaxs;
      std::vector< MultiDimFunctionGraph< double >* > boolQs;

      for (auto varIter = this->fmdp()->beginVariables(); varIter != this->fmdp()->endVariables();
           ++varIter) {
        const IVisitableGraphLearner* visited = _counterTable_[*actionIter];

        MultiDimFunctionGraph< double >* varRMax  = this->operator_->getFunctionInstance();
        MultiDimFunctionGraph< double >* varBoolQ = this->operator_->getFunctionInstance();

        visited->insertSetOfVars(varRMax);
        visited->insertSetOfVars(varBoolQ);

        std::pair< NodeId, NodeId > rooty
           = _visitLearner_(visited, visited->root(), varRMax, varBoolQ);
        varRMax->manager()->setRootNode(rooty.first);
        varRMax->manager()->reduce();
        varRMax->manager()->clean();
        varBoolQ->manager()->setRootNode(rooty.second);
        varBoolQ->manager()->reduce();
        varBoolQ->manager()->clean();

        rmaxs.push_back(varRMax);
        boolQs.push_back(varBoolQ);

        //          std::cout << RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->toDot() << std::endl;
        //          for( auto varIter2 =
        //          RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->variablesSequence().beginSafe(); varIter2 !=
        //          RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->variablesSequence().endSafe(); ++varIter2 )
        //              std::cout << (*varIter2)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varRMax->toDot() << std::endl;
        //          for( auto varIter =
        //          varRMax->variablesSequence().beginSafe(); varIter !=
        //          varRMax->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varBoolQ->toDot() << std::endl;
        //          for( auto varIter =
        //          varBoolQ->variablesSequence().beginSafe(); varIter !=
        //          varBoolQ->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;
      }

      //        std::cout << "Maximising" << std::endl;
      _actionsRMaxTable_.insert(*actionIter, this->maximiseQactions_(rmaxs));
      _actionsBoolTable_.insert(*actionIter, this->minimiseFunctions_(boolQs));
    }
  }

Here is the call graph for this function:

_visitLearner_()

std::pair< NodeId, NodeId > gum::AdaptiveRMaxPlaner::_visitLearner_ ( const IVisitableGraphLearner *  visited, NodeId  currentNodeId, MultiDimFunctionGraph< double > *  rmax, MultiDimFunctionGraph< double > *  boolQ  )

private

Definition at line 288 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                                {
    std::pair< NodeId, NodeId > rep;
    if (visited->isTerminal(currentNodeId)) {
      rep.first = rmax->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < _rThreshold_ ? _rmax_ : 0.0);
      rep.second = boolQ->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < _rThreshold_ ? 0.0 : 1.0);
      return rep;
    }

    NodeId* rmaxsons = static_cast< NodeId* >(
       SOA_ALLOCATE(sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));
    NodeId* bqsons = static_cast< NodeId* >(
       SOA_ALLOCATE(sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));

    for (Idx moda = 0; moda < visited->nodeVar(currentNodeId)->domainSize(); ++moda) {
      std::pair< NodeId, NodeId > sonp
         = _visitLearner_(visited, visited->nodeSon(currentNodeId, moda), rmax, boolQ);
      rmaxsons[moda] = sonp.first;
      bqsons[moda]   = sonp.second;
    }

    rep.first  = rmax->manager()->addInternalNode(visited->nodeVar(currentNodeId), rmaxsons);
    rep.second = boolQ->manager()->addInternalNode(visited->nodeVar(currentNodeId), bqsons);
    return rep;
  }

Here is the call graph for this function:

addReward_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::addReward_ ( MultiDimFunctionGraph< double > *  function, Idx  actionId = 0  )

protectedvirtualinherited

Perform the R(s) + gamma . function.

Warning

function is deleted, new one is returned

Definition at line 395 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                                               {
    // *****************************************************************************************
    // ... we multiply the result by the discount factor, ...
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = operator_->getFunctionInstance();
    newVFunction->copyAndMultiplyByScalar(*Vold, this->discountFactor_);
    delete Vold;

    // *****************************************************************************************
    // ... and finally add reward
    newVFunction = operator_->add(newVFunction, RECAST(fmdp_->reward(actionId)));

    return newVFunction;
  }

argmaximiseQactions_()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::argmaximiseQactions_ ( std::vector< MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * > &  qActionsSet )

protectedvirtualinherited

Performs argmax_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 519 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                                     {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >* newVFunction
       = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >* qAction
         = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->argmaximize(newVFunction, qAction);
    }

    return newVFunction;
  }

checkState()

void gum::AdaptiveRMaxPlaner::checkState ( const Instantiation &  newState, Idx  actionId  )

inlinevirtual

Implements gum::IDecisionStrategy.

Definition at line 198 of file adaptiveRMaxPlaner.h.

                                                                 {
      if (!_initializedTable_[actionId]) {
        _counterTable_[actionId]->reset(newState);
        _initializedTable_[actionId] = true;
      } else
        _counterTable_[actionId]->incState(newState);
    }

evalPolicy_()

void gum::AdaptiveRMaxPlaner::evalPolicy_ ( )

protectedvirtual

Perform the required tasks to extract an optimal policy.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 183 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction = operator_->getFunctionInstance();
    newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());

    std::vector< MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >* >
       argMaxQActionsSet;
    // *****************************************************************************************
    // For each action
    for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter) {
      MultiDimFunctionGraph< double >* qAction = this->evalQaction_(newVFunction, *actionIter);

      qAction = this->addReward_(qAction, *actionIter);

      qAction = this->operator_->maximize(
         _actionsRMaxTable_[*actionIter],
         this->operator_->multiply(qAction, _actionsBoolTable_[*actionIter], 1),
         2);

      argMaxQActionsSet.push_back(makeArgMax_(qAction, *actionIter));
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >* argMaxVFunction
       = argmaximiseQactions_(argMaxQActionsSet);

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    extractOptimalPolicy_(argMaxVFunction);
  }

Here is the call graph for this function:

evalQaction_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::evalQaction_ ( const MultiDimFunctionGraph< double > *  Vold, Idx  actionId  )

protectedvirtualinherited

Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

Definition at line 341 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                {
    // ******************************************************************************
    // Initialisation :
    // Creating a copy of last Vfunction to deduce from the new Qaction
    // And finding the first var to eleminate (the one at the end)

    return operator_->regress(Vold, actionId, this->fmdp_, this->elVarSeq_);
  }

extractOptimalPolicy_()

void gum::StructuredPlaner< double >::extractOptimalPolicy_ ( const MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > *  optimalValueFunction )

protectedinherited

From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.

Warning

deallocate the argmax optimal value function

Definition at line 542 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                    {
    optimalPolicy_->clear();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter
         = argMaxOptimalValueFunction->variablesSequence().beginSafe();
         varIter != argMaxOptimalValueFunction->variablesSequence().endSafe();
         ++varIter)
      optimalPolicy_->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    optimalPolicy_->manager()->setRootNode(_recurExtractOptPol_(argMaxOptimalValueFunction->root(),
                                                                argMaxOptimalValueFunction,
                                                                src2dest));

    delete argMaxOptimalValueFunction;
  }

fmdp()

INLINE const FMDP< double >* gum::StructuredPlaner< double >::fmdp ( )

inlineinherited

Returns a const ptr on the Factored Markov Decision Process on which we're planning.

Definition at line 133 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

{ return fmdp_; }

initialize()

void gum::AdaptiveRMaxPlaner::initialize ( const FMDP< double > *  fmdp )

virtual

Initializes data structure needed for making the planning.

Warning

No calling this methods before starting the first makePlaninng will surely and definitely result in a crash

Reimplemented from gum::IDecisionStrategy.

Definition at line 96 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                {
    if (!_initialized_) {
      StructuredPlaner::initialize(fmdp);
      IDecisionStrategy::initialize(fmdp);
      for (auto actionIter = fmdp->beginActions(); actionIter != fmdp->endActions(); ++actionIter) {
        _counterTable_.insert(*actionIter, new StatesCounter());
        _initializedTable_.insert(*actionIter, false);
      }
      _initialized_ = true;
    }
  }

Here is the call graph for this function:

initVFunction_()

void gum::AdaptiveRMaxPlaner::initVFunction_ ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 130 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                          {
    vFunction_->manager()->setRootNode(vFunction_->manager()->addTerminalNode(0.0));
    for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter)
      vFunction_ = this->operator_->add(vFunction_, RECASTED(this->fmdp_->reward(*actionIter)), 1);
  }

Here is the call graph for this function:

makeArgMax_()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::makeArgMax_ ( const MultiDimFunctionGraph< double > *  Qaction, Idx  actionId  )

protectedinherited

Creates a copy of given Qaction that can be exploit by a Argmax.

Hence, this step consists in replacing each lea by an ArgMaxSet containing the value of the leaf and the actionId of the Qaction

Parameters

Qaction	: the function graph we want to transform
actionId	: the action Id associated to that graph

Warning

delete the original Qaction, returns its conversion

Definition at line 463 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                               {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >* amcpy
       = operator_->getArgMaxFunctionInstance();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter
         = qAction->variablesSequence().beginSafe();
         varIter != qAction->variablesSequence().endSafe();
         ++varIter)
      amcpy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    amcpy->manager()->setRootNode(
       _recurArgMaxCopy_(qAction->root(), actionId, qAction, amcpy, src2dest));

    delete qAction;
    return amcpy;
  }

makePlanning()

void gum::AdaptiveRMaxPlaner::makePlanning ( Idx  nbStep = 1000000 )

virtual

Performs a value iteration.

Parameters

nbStep

: enables you to specify how many value iterations you wish to do. makePlanning will then stop whether when optimal value function is reach or when nbStep have been performed

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 111 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                  {
    _makeRMaxFunctionGraphs_();

    StructuredPlaner::makePlanning(nbStep);

    _clearTables_();
  }

Here is the call graph for this function:

maximiseQactions_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::maximiseQactions_ ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs max_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 356 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                     {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->maximize(newVFunction, qAction);
    }

    return newVFunction;
  }

minimiseFunctions_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::minimiseFunctions_ ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs min_i F_i.

Warning

Performs also the deallocation of the F_i

Definition at line 375 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                     {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->minimize(newVFunction, qAction);
    }

    return newVFunction;
  }

optimalPolicy()

INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::optimalPolicy ( )

inlinevirtualinherited

Returns the best policy obtained so far.

Implements gum::IPlanningStrategy< double >.

Definition at line 148 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                                                                            {
      return optimalPolicy_;
    }

optimalPolicy2String()

std::string gum::StructuredPlaner< double >::optimalPolicy2String ( )

virtualinherited

Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.

Implements gum::IPlanningStrategy< double >.

Definition at line 104 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                 {
    // ************************************************************************
    // Discarding the case where no \pi* have been computed
    if (!optimalPolicy_ || optimalPolicy_->root() == 0) return "NO OPTIMAL POLICY CALCULATED YET";

    // ************************************************************************
    // Initialisation

    // Declaration of the needed string stream
    std::stringstream output;
    std::stringstream terminalStream;
    std::stringstream nonTerminalStream;
    std::stringstream arcstream;

    // First line for the toDot
    output << std::endl << "digraph \" OPTIMAL POLICY \" {" << std::endl;

    // Form line for the internal node stream en the terminal node stream
    terminalStream << "node [shape = box];" << std::endl;
    nonTerminalStream << "node [shape = ellipse];" << std::endl;

    // For somme clarity in the final string
    std::string tab = "\t";

    // To know if we already checked a node or not
    Set< NodeId > visited;

    // FIFO of nodes to visit
    std::queue< NodeId > fifo;

    // Loading the FIFO
    fifo.push(optimalPolicy_->root());
    visited << optimalPolicy_->root();


    // ************************************************************************
    // Main loop
    while (!fifo.empty()) {
      // Node to visit
      NodeId currentNodeId = fifo.front();
      fifo.pop();

      // Checking if it is terminal
      if (optimalPolicy_->isTerminalNode(currentNodeId)) {
        // Get back the associated ActionSet
        ActionSet ase = optimalPolicy_->nodeValue(currentNodeId);

        // Creating a line for this node
        terminalStream << tab << currentNodeId << ";" << tab << currentNodeId << " [label=\""
                       << currentNodeId << " - ";

        // Enumerating and adding to the line the associated optimal actions
        for (SequenceIteratorSafe< Idx > valIter = ase.beginSafe(); valIter != ase.endSafe();
             ++valIter)
          terminalStream << fmdp_->actionName(*valIter) << " ";

        // Terminating line
        terminalStream << "\"];" << std::endl;
        continue;
      }

      // Either wise
      {
        // Geting back the associated internal node
        const InternalNode* currentNode = optimalPolicy_->node(currentNodeId);

        // Creating a line in internalnode stream for this node
        nonTerminalStream << tab << currentNodeId << ";" << tab << currentNodeId << " [label=\""
                          << currentNodeId << " - " << currentNode->nodeVar()->name() << "\"];"
                          << std::endl;

        // Going through the sons and agregating them according the the sons Ids
        HashTable< NodeId, LinkedList< Idx >* > sonMap;
        for (Idx sonIter = 0; sonIter < currentNode->nbSons(); ++sonIter) {
          if (!visited.exists(currentNode->son(sonIter))) {
            fifo.push(currentNode->son(sonIter));
            visited << currentNode->son(sonIter);
          }
          if (!sonMap.exists(currentNode->son(sonIter)))
            sonMap.insert(currentNode->son(sonIter), new LinkedList< Idx >());
          sonMap[currentNode->son(sonIter)]->addLink(sonIter);
        }

        // Adding to the arc stram
        for (auto sonIter = sonMap.beginSafe(); sonIter != sonMap.endSafe(); ++sonIter) {
          arcstream << tab << currentNodeId << " -> " << sonIter.key() << " [label=\" ";
          Link< Idx >* modaIter = sonIter.val()->list();
          while (modaIter) {
            arcstream << currentNode->nodeVar()->label(modaIter->element());
            if (modaIter->nextLink()) arcstream << ", ";
            modaIter = modaIter->nextLink();
          }
          arcstream << "\",color=\"#00ff00\"];" << std::endl;
          delete sonIter.val();
        }
      }
    }

    // Terminating
    output << terminalStream.str() << std::endl
           << nonTerminalStream.str() << std::endl
           << arcstream.str() << std::endl
           << "}" << std::endl;

    return output.str();
  }

optimalPolicySize()

virtual Size gum::StructuredPlaner< double >::optimalPolicySize ( )

inlinevirtualinherited

Returns optimalPolicy computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 155 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                     {
      return optimalPolicy_ != nullptr ? optimalPolicy_->realSize() : 0;
    }

ReducedAndOrderedInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::ReducedAndOrderedInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 62 of file adaptiveRMaxPlaner.h.

                                                                                       {
      return new AdaptiveRMaxPlaner(new MDDOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

setOptimalStrategy()

void gum::IDecisionStrategy::setOptimalStrategy ( const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *  optPol )

inlineinherited

Definition at line 89 of file IDecisionStrategy.h.

                                                                                                   {
      optPol_ = const_cast< MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* >(optPol);
    }

spumddInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::spumddInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 79 of file structuredPlaner.h.

                                                                                            {
      return new StructuredPlaner< GUM_SCALAR >(new MDDOperatorStrategy< GUM_SCALAR >(),
                                                discountFactor,
                                                epsilon,
                                                verbose);
    }

stateOptimalPolicy()

virtual ActionSet gum::IDecisionStrategy::stateOptimalPolicy ( const Instantiation &  curState )

inlinevirtualinherited

Reimplemented in gum::E_GreedyDecider, and gum::RandomDecider.

Definition at line 93 of file IDecisionStrategy.h.

                                                                        {
      return (optPol_ && optPol_->realSize() != 0) ? optPol_->get(curState) : allActions_;
    }

sviInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::sviInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 91 of file structuredPlaner.h.

                                                                                         {
      return new StructuredPlaner< GUM_SCALAR >(new TreeOperatorStrategy< GUM_SCALAR >(),
                                                discountFactor,
                                                epsilon,
                                                verbose);
    }

TreeInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::TreeInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 76 of file adaptiveRMaxPlaner.h.

                                                                                            {
      return new AdaptiveRMaxPlaner(new TreeOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

valueIteration_()

MultiDimFunctionGraph< double > * gum::AdaptiveRMaxPlaner::valueIteration_ ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 139 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction = operator_->getFunctionInstance();
    newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());

    // *****************************************************************************************
    // For each action
    std::vector< MultiDimFunctionGraph< double >* > qActionsSet;
    for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter) {
      MultiDimFunctionGraph< double >* qAction = evalQaction_(newVFunction, *actionIter);

      // *******************************************************************************************
      // Next, we add the reward
      qAction = addReward_(qAction, *actionIter);

      qAction = this->operator_->maximize(
         _actionsRMaxTable_[*actionIter],
         this->operator_->multiply(qAction, _actionsBoolTable_[*actionIter], 1),
         2);

      qActionsSet.push_back(qAction);
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    newVFunction = maximiseQactions_(qActionsSet);

    return newVFunction;
  }

Here is the call graph for this function:

vFunction()

INLINE const MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::vFunction ( )

inlineinherited

Returns a const ptr on the value function computed so far.

Definition at line 138 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

{ return vFunction_; }

vFunctionSize()

virtual Size gum::StructuredPlaner< double >::vFunctionSize ( )

inlinevirtualinherited

Returns vFunction computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 143 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

{ return vFunction_ != nullptr ? vFunction_->realSize() : 0; }

Member Data Documentation

_actionsBoolTable_

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::_actionsBoolTable_

private

Definition at line 186 of file adaptiveRMaxPlaner.h.

_actionsRMaxTable_

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::_actionsRMaxTable_

private

Definition at line 185 of file adaptiveRMaxPlaner.h.

_counterTable_

HashTable< Idx, StatesCounter* > gum::AdaptiveRMaxPlaner::_counterTable_

private

Definition at line 207 of file adaptiveRMaxPlaner.h.

_fmdpLearner_

const ILearningStrategy* gum::AdaptiveRMaxPlaner::_fmdpLearner_

private

Definition at line 187 of file adaptiveRMaxPlaner.h.

_initialized_

bool gum::AdaptiveRMaxPlaner::_initialized_

private

Definition at line 210 of file adaptiveRMaxPlaner.h.

_initializedTable_

HashTable< Idx, bool > gum::AdaptiveRMaxPlaner::_initializedTable_

private

Definition at line 208 of file adaptiveRMaxPlaner.h.

_rmax_

double gum::AdaptiveRMaxPlaner::_rmax_

private

Definition at line 190 of file adaptiveRMaxPlaner.h.

_rThreshold_

double gum::AdaptiveRMaxPlaner::_rThreshold_

private

Definition at line 189 of file adaptiveRMaxPlaner.h.

allActions_

ActionSet gum::IDecisionStrategy::allActions_

protectedinherited

Definition at line 102 of file IDecisionStrategy.h.

discountFactor_

double gum::StructuredPlaner< double >::discountFactor_

protectedinherited

Discount Factor used for infinite horizon planning.

Definition at line 350 of file structuredPlaner.h.

elVarSeq_

Set< const DiscreteVariable* > gum::StructuredPlaner< double >::elVarSeq_

protectedinherited

A Set to eleminate primed variables.

Definition at line 345 of file structuredPlaner.h.

fmdp_

const FMDP< double >* gum::StructuredPlaner< double >::fmdp_

protectedinherited

The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions )

Definition at line 325 of file structuredPlaner.h.

operator_

IOperatorStrategy< double >* gum::StructuredPlaner< double >::operator_

protectedinherited

Definition at line 352 of file structuredPlaner.h.

optimalPolicy_

MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::optimalPolicy_

protectedinherited

The associated optimal policy.

Warning

Leaves are ActionSet which contains the ids of the best actions While this is sufficient to be exploited, to be understood by a human somme translation from the fmdp_ is required. optimalPolicy2String do this job.

Definition at line 340 of file structuredPlaner.h.

optPol_

const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::IDecisionStrategy::optPol_

protectedinherited

Definition at line 99 of file IDecisionStrategy.h.

verbose_

bool gum::StructuredPlaner< double >::verbose_

protectedinherited

Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Definition at line 358 of file structuredPlaner.h.

vFunction_

MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::vFunction_

protectedinherited

The Value Function computed iteratively.

Definition at line 330 of file structuredPlaner.h.

---

The documentation for this class was generated from the following files:

• agrum/FMDP/planning/adaptiveRMaxPlaner.h
• agrum/FMDP/planning/adaptiveRMaxPlaner.cpp