gum::StructuredPlaner< GUM_SCALAR > Class Template Reference

<agrum/FMDP/planning/structuredPlaner.h> More...

#include <structuredPlaner.h>

Inheritance diagram for gum::StructuredPlaner< GUM_SCALAR >:

Collaboration diagram for gum::StructuredPlaner< GUM_SCALAR >:

Public Member Functions
Datastructure access methods
INLINE const FMDP< GUM_SCALAR > *	fmdp ()
	Returns a const ptr on the Factored Markov Decision Process on which we're planning. More...
INLINE const MultiDimFunctionGraph< GUM_SCALAR > *	vFunction ()
	Returns a const ptr on the value function computed so far. More...
virtual Size	vFunctionSize ()
	Returns vFunction computed so far current size. More...
INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy ()
	Returns the best policy obtained so far. More...
virtual Size	optimalPolicySize ()
	Returns optimalPolicy computed so far current size. More...
std::string	optimalPolicy2String ()
	Provide a better toDot for the optimal policy where the leaves have the action name instead of its id. More...
Planning Methods
virtual void	initialize (const FMDP< GUM_SCALAR > *fmdp)
	Initializes data structure needed for making the planning. More...
virtual void	makePlanning (Idx nbStep=1000000)
	Performs a value iteration. More...

Static Public Member Functions
static StructuredPlaner< GUM_SCALAR > *	spumddInstance (GUM_SCALAR discountFactor=0.9, GUM_SCALAR epsilon=0.00001, bool verbose=true)
static StructuredPlaner< GUM_SCALAR > *	sviInstance (GUM_SCALAR discountFactor=0.9, GUM_SCALAR epsilon=0.00001, bool verbose=true)

Protected Attributes
const FMDP< GUM_SCALAR > *	_fmdp
	The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ) More...
MultiDimFunctionGraph< GUM_SCALAR > *	_vFunction
	The Value Function computed iteratively. More...
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	_optimalPolicy
	The associated optimal policy. More...
Set< const DiscreteVariable *>	_elVarSeq
	A Set to eleminate primed variables. More...
GUM_SCALAR	_discountFactor
	Discount Factor used for infinite horizon planning. More...
IOperatorStrategy< GUM_SCALAR > *	_operator
bool	_verbose
	Boolean used to indcates whether or not iteration informations should be displayed on terminal. More...

Protected Member Functions
Value Iteration Methods
virtual void	_initVFunction ()
	Performs a single step of value iteration. More...
virtual MultiDimFunctionGraph< GUM_SCALAR > *	_valueIteration ()
	Performs a single step of value iteration. More...
virtual MultiDimFunctionGraph< GUM_SCALAR > *	_evalQaction (const MultiDimFunctionGraph< GUM_SCALAR > *, Idx)
	Performs the P(s'|s,a).V^{t-1}(s') part of the value itération. More...
virtual MultiDimFunctionGraph< GUM_SCALAR > *	_maximiseQactions (std::vector< MultiDimFunctionGraph< GUM_SCALAR > * > &)
	Performs max_a Q(s,a) More...
virtual MultiDimFunctionGraph< GUM_SCALAR > *	_minimiseFunctions (std::vector< MultiDimFunctionGraph< GUM_SCALAR > * > &)
	Performs min_i F_i. More...
virtual MultiDimFunctionGraph< GUM_SCALAR > *	_addReward (MultiDimFunctionGraph< GUM_SCALAR > *function, Idx actionId=0)
	Perform the R(s) + gamma . function. More...

Constructor & destructor.
	StructuredPlaner (IOperatorStrategy< GUM_SCALAR > *opi, GUM_SCALAR discountFactor, GUM_SCALAR epsilon, bool verbose)
	Default constructor. More...
virtual	~StructuredPlaner ()
	Default destructor. More...

Optimal policy extraction methods
virtual void	_evalPolicy ()
	Perform the required tasks to extract an optimal policy. More...
MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *	_makeArgMax (const MultiDimFunctionGraph< GUM_SCALAR > *Qaction, Idx actionId)
	Creates a copy of given Qaction that can be exploit by a Argmax. More...
virtual MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *	_argmaximiseQactions (std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > * > &)
	Performs argmax_a Q(s,a) More...
void	_extractOptimalPolicy (const MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
	From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet. More...
NodeId	__recurArgMaxCopy (NodeId, Idx, const MultiDimFunctionGraph< GUM_SCALAR > *, MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
	Recursion part for the createArgMaxCopy. More...
NodeId	__recurExtractOptPol (NodeId, const MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
	Recursion part for the createArgMaxCopy. More...
void	__transferActionIds (const ArgMaxSet< GUM_SCALAR, Idx > &, ActionSet &)
	Extract from an ArgMaxSet the associated ActionSet. More...

Detailed Description

template<typename GUM_SCALAR>
class gum::StructuredPlaner< GUM_SCALAR >

<agrum/FMDP/planning/structuredPlaner.h>

A class to find optimal policy for a given FMDP.

Perform a structure value iteration planning

Pure virtual functions : _regress, _maximize, _argmaximize, _add and _subtract are a priori the ones to be respecified according to the used datastructure (MDDs, DTs, BNs, ...)

Definition at line 70 of file structuredPlaner.h.

Constructor & Destructor Documentation

StructuredPlaner()

template<typename GUM_SCALAR>

INLINE gum::StructuredPlaner< GUM_SCALAR >::StructuredPlaner ( IOperatorStrategy< GUM_SCALAR > *  opi, GUM_SCALAR  discountFactor, GUM_SCALAR  epsilon, bool  verbose  )

protected

Default constructor.

Definition at line 64 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::sviInstance().

                                               :
      _discountFactor(discountFactor),
      _operator(opi), _verbose(verbose) {
    GUM_CONSTRUCTOR(StructuredPlaner);

    __threshold = epsilon;
    _vFunction = nullptr;
    _optimalPolicy = nullptr;
  }

Here is the caller graph for this function:

~StructuredPlaner()

template<typename GUM_SCALAR >

INLINE gum::StructuredPlaner< GUM_SCALAR >::~StructuredPlaner ( )

virtual

Default destructor.

Definition at line 82 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::sviInstance().

                                                           {
    GUM_DESTRUCTOR(StructuredPlaner);

    if (_vFunction) { delete _vFunction; }

    if (_optimalPolicy) delete _optimalPolicy;

    delete _operator;
  }

Here is the caller graph for this function:

Member Function Documentation

__recurArgMaxCopy()

template<typename GUM_SCALAR>

NodeId gum::StructuredPlaner< GUM_SCALAR >::__recurArgMaxCopy ( NodeId  currentNodeId, Idx  actionId, const MultiDimFunctionGraph< GUM_SCALAR > *  src, MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *  argMaxCpy, HashTable< NodeId, NodeId > &  visitedNodes  )

private

Recursion part for the createArgMaxCopy.

Definition at line 507 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                {
    if (visitedNodes.exists(currentNodeId)) return visitedNodes[currentNodeId];

    NodeId nody;
    if (src->isTerminalNode(currentNodeId)) {
      ArgMaxSet< GUM_SCALAR, Idx > leaf(src->nodeValue(currentNodeId), actionId);
      nody = argMaxCpy->manager()->addTerminalNode(leaf);
    } else {
      const InternalNode* currentNode = src->node(currentNodeId);
      NodeId*             sonsMap = static_cast< NodeId* >(
         SOA_ALLOCATE(sizeof(NodeId) * currentNode->nodeVar()->domainSize()));
      for (Idx moda = 0; moda < currentNode->nodeVar()->domainSize(); ++moda)
        sonsMap[moda] = __recurArgMaxCopy(
           currentNode->son(moda), actionId, src, argMaxCpy, visitedNodes);
      nody =
         argMaxCpy->manager()->addInternalNode(currentNode->nodeVar(), sonsMap);
    }
    visitedNodes.insert(currentNodeId, nody);
    return nody;
  }

Here is the caller graph for this function:

__recurExtractOptPol()

template<typename GUM_SCALAR>

NodeId gum::StructuredPlaner< GUM_SCALAR >::__recurExtractOptPol ( NodeId  currentNodeId, const MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *  argMaxOptVFunc, HashTable< NodeId, NodeId > &  visitedNodes  )

private

Recursion part for the createArgMaxCopy.

Definition at line 589 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                         {
    if (visitedNodes.exists(currentNodeId)) return visitedNodes[currentNodeId];

    NodeId nody;
    if (argMaxOptVFunc->isTerminalNode(currentNodeId)) {
      ActionSet leaf;
      __transferActionIds(argMaxOptVFunc->nodeValue(currentNodeId), leaf);
      nody = _optimalPolicy->manager()->addTerminalNode(leaf);
    } else {
      const InternalNode* currentNode = argMaxOptVFunc->node(currentNodeId);
      NodeId*             sonsMap = static_cast< NodeId* >(
         SOA_ALLOCATE(sizeof(NodeId) * currentNode->nodeVar()->domainSize()));
      for (Idx moda = 0; moda < currentNode->nodeVar()->domainSize(); ++moda)
        sonsMap[moda] = __recurExtractOptPol(
           currentNode->son(moda), argMaxOptVFunc, visitedNodes);
      nody = _optimalPolicy->manager()->addInternalNode(currentNode->nodeVar(),
                                                        sonsMap);
    }
    visitedNodes.insert(currentNodeId, nody);
    return nody;
  }

Here is the caller graph for this function:

__transferActionIds()

template<typename GUM_SCALAR>

void gum::StructuredPlaner< GUM_SCALAR >::__transferActionIds ( const ArgMaxSet< GUM_SCALAR, Idx > &  src, ActionSet &  dest  )

private

Extract from an ArgMaxSet the associated ActionSet.

Definition at line 619 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                               {
    for (auto idi = src.beginSafe(); idi != src.endSafe(); ++idi)
      dest += *idi;
  }

Here is the caller graph for this function:

_addReward()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< GUM_SCALAR > * gum::StructuredPlaner< GUM_SCALAR >::_addReward ( MultiDimFunctionGraph< GUM_SCALAR > *  function, Idx  actionId = 0  )

protectedvirtual

Perform the R(s) + gamma . function.

Warning

function is deleted, new one is returned

Definition at line 408 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                              {
    // *****************************************************************************************
    // ... we multiply the result by the discount factor, ...
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndMultiplyByScalar(*Vold, this->_discountFactor);
    delete Vold;

    // *****************************************************************************************
    // ... and finally add reward
    newVFunction = _operator->add(newVFunction, RECAST(_fmdp->reward(actionId)));

    return newVFunction;
  }

Here is the caller graph for this function:

_argmaximiseQactions()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< GUM_SCALAR >::_argmaximiseQactions ( std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > * > &  qActionsSet )

protectedvirtual

Performs argmax_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 540 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                        {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
         qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->argmaximize(newVFunction, qAction);
    }

    return newVFunction;
  }

Here is the caller graph for this function:

_evalPolicy()

template<typename GUM_SCALAR >

void gum::StructuredPlaner< GUM_SCALAR >::_evalPolicy ( )

protectedvirtual

Perform the required tasks to extract an optimal policy.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 437 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                   {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndReassign(*_vFunction, _fmdp->mapMainPrime());

    std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >,
                                        SetTerminalNodePolicy >* >
       argMaxQActionsSet;
    // *****************************************************************************************
    // For each action
    for (auto actionIter = _fmdp->beginActions();
         actionIter != _fmdp->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction =
         this->_evalQaction(newVFunction, *actionIter);

      qAction = this->_addReward(qAction);

      argMaxQActionsSet.push_back(_makeArgMax(qAction, *actionIter));
    }
    delete newVFunction;


    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       argMaxVFunction = _argmaximiseQactions(argMaxQActionsSet);

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    _extractOptimalPolicy(argMaxVFunction);
  }

Here is the caller graph for this function:

_evalQaction()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< GUM_SCALAR > * gum::StructuredPlaner< GUM_SCALAR >::_evalQaction ( const MultiDimFunctionGraph< GUM_SCALAR > *  Vold, Idx  actionId  )

protectedvirtual

Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

Definition at line 353 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                       {
    // ******************************************************************************
    // Initialisation :
    // Creating a copy of last Vfunction to deduce from the new Qaction
    // And finding the first var to eleminate (the one at the end)

    return _operator->regress(Vold, actionId, this->_fmdp, this->_elVarSeq);
  }

Here is the caller graph for this function:

_extractOptimalPolicy()

template<typename GUM_SCALAR>

void gum::StructuredPlaner< GUM_SCALAR >::_extractOptimalPolicy ( const MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > *  optimalValueFunction )

protected

From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.

Warning

deallocate the argmax optimal value function

Definition at line 564 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                    {
    _optimalPolicy->clear();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter =
            argMaxOptimalValueFunction->variablesSequence().beginSafe();
         varIter != argMaxOptimalValueFunction->variablesSequence().endSafe();
         ++varIter)
      _optimalPolicy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    _optimalPolicy->manager()->setRootNode(__recurExtractOptPol(
       argMaxOptimalValueFunction->root(), argMaxOptimalValueFunction, src2dest));

    delete argMaxOptimalValueFunction;
  }

Here is the caller graph for this function:

_initVFunction()

template<typename GUM_SCALAR >

void gum::StructuredPlaner< GUM_SCALAR >::_initVFunction ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 298 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                      {
    _vFunction->copy(*(RECAST(_fmdp->reward())));
  }

Here is the caller graph for this function:

_makeArgMax()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< GUM_SCALAR >::_makeArgMax ( const MultiDimFunctionGraph< GUM_SCALAR > *  Qaction, Idx  actionId  )

protected

Creates a copy of given Qaction that can be exploit by a Argmax.

Hence, this step consists in replacing each lea by an ArgMaxSet containing the value of the leaf and the actionId of the Qaction

Parameters

Qaction	: the function graph we want to transform
actionId	: the action Id associated to that graph

Warning

delete the original Qaction, returns its conversion

Definition at line 482 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                          {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       amcpy = _operator->getArgMaxFunctionInstance();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter =
            qAction->variablesSequence().beginSafe();
         varIter != qAction->variablesSequence().endSafe();
         ++varIter)
      amcpy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    amcpy->manager()->setRootNode(
       __recurArgMaxCopy(qAction->root(), actionId, qAction, amcpy, src2dest));

    delete qAction;
    return amcpy;
  }

Here is the caller graph for this function:

_maximiseQactions()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< GUM_SCALAR > * gum::StructuredPlaner< GUM_SCALAR >::_maximiseQactions ( std::vector< MultiDimFunctionGraph< GUM_SCALAR > * > &  qActionsSet )

protectedvirtual

Performs max_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 369 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->maximize(newVFunction, qAction);
    }

    return newVFunction;
  }

Here is the caller graph for this function:

_minimiseFunctions()

template<typename GUM_SCALAR>

MultiDimFunctionGraph< GUM_SCALAR > * gum::StructuredPlaner< GUM_SCALAR >::_minimiseFunctions ( std::vector< MultiDimFunctionGraph< GUM_SCALAR > * > &  qActionsSet )

protectedvirtual

Performs min_i F_i.

Warning

Performs also the deallocation of the F_i

Definition at line 389 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->minimize(newVFunction, qAction);
    }

    return newVFunction;
  }

Here is the caller graph for this function:

_valueIteration()

template<typename GUM_SCALAR >

MultiDimFunctionGraph< GUM_SCALAR > * gum::StructuredPlaner< GUM_SCALAR >::_valueIteration ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 316 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                     {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndReassign(*_vFunction, _fmdp->mapMainPrime());

    // *****************************************************************************************
    // For each action
    std::vector< MultiDimFunctionGraph< GUM_SCALAR >* > qActionsSet;
    for (auto actionIter = _fmdp->beginActions();
         actionIter != _fmdp->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction =
         this->_evalQaction(newVFunction, *actionIter);
      qActionsSet.push_back(qAction);
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    newVFunction = this->_maximiseQactions(qActionsSet);

    // *******************************************************************************************
    // Next, we eval the new function value
    newVFunction = this->_addReward(newVFunction);

    return newVFunction;
  }

Here is the caller graph for this function:

fmdp()

template<typename GUM_SCALAR>

INLINE const FMDP< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::fmdp ( )

inline

Returns a const ptr on the Factored Markov Decision Process on which we're planning.

Definition at line 137 of file structuredPlaner.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

{ return _fmdp; }

Here is the caller graph for this function:

initialize()

template<typename GUM_SCALAR>

void gum::StructuredPlaner< GUM_SCALAR >::initialize ( const FMDP< GUM_SCALAR > *  fmdp )

virtual

Initializes data structure needed for making the planning.

Warning

No calling this methods before starting the first makePlaninng will surely and definitely result in a crash

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 229 of file structuredPlaner_tpl.h.

Referenced by gum::AdaptiveRMaxPlaner::initialize(), and gum::StructuredPlaner< double >::optimalPolicySize().

                                                                                {
    _fmdp = fmdp;

    // Determination of the threshold value
    __threshold *= (1 - _discountFactor) / (2 * _discountFactor);

    // Establishement of sequence of variable elemination
    for (auto varIter = _fmdp->beginVariables(); varIter != _fmdp->endVariables();
         ++varIter)
      _elVarSeq << _fmdp->main2prime(*varIter);

    // Initialisation of the value function
    _vFunction = _operator->getFunctionInstance();
    _optimalPolicy = _operator->getAggregatorInstance();
    __firstTime = true;
  }

Here is the caller graph for this function:

makePlanning()

template<typename GUM_SCALAR >

void gum::StructuredPlaner< GUM_SCALAR >::makePlanning ( Idx  nbStep = 1000000 )

virtual

Performs a value iteration.

Parameters

nbStep

: enables you to specify how many value iterations you wish to do. makePlanning will then stop whether when optimal value function is reach or when nbStep have been performed

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 251 of file structuredPlaner_tpl.h.

Referenced by gum::AdaptiveRMaxPlaner::makePlanning(), and gum::StructuredPlaner< double >::optimalPolicySize().

                                                              {
    if (__firstTime) {
      this->_initVFunction();
      __firstTime = false;
    }

    // *****************************************************************************************
    // Main loop
    // *****************************************************************************************
    Idx        nbIte = 0;
    GUM_SCALAR gap = __threshold + 1;
    while ((gap > __threshold) && (nbIte < nbStep)) {
      ++nbIte;

      MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = this->_valueIteration();

      // *****************************************************************************************
      // Then we compare new value function and the old one
      MultiDimFunctionGraph< GUM_SCALAR >* deltaV =
         _operator->subtract(newVFunction, _vFunction);
      gap = 0;

      for (deltaV->beginValues(); deltaV->hasValue(); deltaV->nextValue())
        if (gap < fabs(deltaV->value())) gap = fabs(deltaV->value());
      delete deltaV;

      if (_verbose)
        std::cout << " ------------------- Fin itération n° " << nbIte << std::endl
                  << " Gap : " << gap << " - " << __threshold << std::endl;

      // *****************************************************************************************
      // And eventually we update pointers for next loop
      delete _vFunction;
      _vFunction = newVFunction;
    }

    // *****************************************************************************************
    // Policy matching value function research
    // *****************************************************************************************
    this->_evalPolicy();
  }

Here is the caller graph for this function:

optimalPolicy()

template<typename GUM_SCALAR>

INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< GUM_SCALAR >::optimalPolicy ( )

inlinevirtual

Returns the best policy obtained so far.

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Definition at line 157 of file structuredPlaner.h.

                                 {
      return _optimalPolicy;
    }

optimalPolicy2String()

template<typename GUM_SCALAR >

std::string gum::StructuredPlaner< GUM_SCALAR >::optimalPolicy2String ( )

virtual

Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Definition at line 105 of file structuredPlaner_tpl.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicySize().

                                                                 {
    // ************************************************************************
    // Discarding the case where no \pi* have been computed
    if (!_optimalPolicy || _optimalPolicy->root() == 0)
      return "NO OPTIMAL POLICY CALCULATED YET";

    // ************************************************************************
    // Initialisation

    // Declaration of the needed string stream
    std::stringstream output;
    std::stringstream terminalStream;
    std::stringstream nonTerminalStream;
    std::stringstream arcstream;

    // First line for the toDot
    output << std::endl << "digraph \" OPTIMAL POLICY \" {" << std::endl;

    // Form line for the internal node stream en the terminal node stream
    terminalStream << "node [shape = box];" << std::endl;
    nonTerminalStream << "node [shape = ellipse];" << std::endl;

    // For somme clarity in the final string
    std::string tab = "\t";

    // To know if we already checked a node or not
    Set< NodeId > visited;

    // FIFO of nodes to visit
    std::queue< NodeId > fifo;

    // Loading the FIFO
    fifo.push(_optimalPolicy->root());
    visited << _optimalPolicy->root();


    // ************************************************************************
    // Main loop
    while (!fifo.empty()) {
      // Node to visit
      NodeId currentNodeId = fifo.front();
      fifo.pop();

      // Checking if it is terminal
      if (_optimalPolicy->isTerminalNode(currentNodeId)) {
        // Get back the associated ActionSet
        ActionSet ase = _optimalPolicy->nodeValue(currentNodeId);

        // Creating a line for this node
        terminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                       << " [label=\"" << currentNodeId << " - ";

        // Enumerating and adding to the line the associated optimal actions
        for (SequenceIteratorSafe< Idx > valIter = ase.beginSafe();
             valIter != ase.endSafe();
             ++valIter)
          terminalStream << _fmdp->actionName(*valIter) << " ";

        // Terminating line
        terminalStream << "\"];" << std::endl;
        continue;
      }

      // Either wise
      {
        // Geting back the associated internal node
        const InternalNode* currentNode = _optimalPolicy->node(currentNodeId);

        // Creating a line in internalnode stream for this node
        nonTerminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                          << " [label=\"" << currentNodeId << " - "
                          << currentNode->nodeVar()->name() << "\"];" << std::endl;

        // Going through the sons and agregating them according the the sons Ids
        HashTable< NodeId, LinkedList< Idx >* > sonMap;
        for (Idx sonIter = 0; sonIter < currentNode->nbSons(); ++sonIter) {
          if (!visited.exists(currentNode->son(sonIter))) {
            fifo.push(currentNode->son(sonIter));
            visited << currentNode->son(sonIter);
          }
          if (!sonMap.exists(currentNode->son(sonIter)))
            sonMap.insert(currentNode->son(sonIter), new LinkedList< Idx >());
          sonMap[currentNode->son(sonIter)]->addLink(sonIter);
        }

        // Adding to the arc stram
        for (auto sonIter = sonMap.beginSafe(); sonIter != sonMap.endSafe();
             ++sonIter) {
          arcstream << tab << currentNodeId << " -> " << sonIter.key()
                    << " [label=\" ";
          Link< Idx >* modaIter = sonIter.val()->list();
          while (modaIter) {
            arcstream << currentNode->nodeVar()->label(modaIter->element());
            if (modaIter->nextLink()) arcstream << ", ";
            modaIter = modaIter->nextLink();
          }
          arcstream << "\",color=\"#00ff00\"];" << std::endl;
          delete sonIter.val();
        }
      }
    }

    // Terminating
    output << terminalStream.str() << std::endl
           << nonTerminalStream.str() << std::endl
           << arcstream.str() << std::endl
           << "}" << std::endl;

    return output.str();
  }

Here is the caller graph for this function:

optimalPolicySize()

template<typename GUM_SCALAR>

virtual Size gum::StructuredPlaner< GUM_SCALAR >::optimalPolicySize ( )

inlinevirtual

Returns optimalPolicy computed so far current size.

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Definition at line 164 of file structuredPlaner.h.

                                     {
      return _optimalPolicy != nullptr ? _optimalPolicy->realSize() : 0;
    }

spumddInstance()

template<typename GUM_SCALAR>

static StructuredPlaner< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::spumddInstance ( GUM_SCALAR  discountFactor = 0.9, GUM_SCALAR  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 80 of file structuredPlaner.h.

Referenced by gum::SDYNA::RandomMDDInstance(), and gum::SDYNA::spimddiInstance().

                                                 {
      return new StructuredPlaner< GUM_SCALAR >(
         new MDDOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

Here is the caller graph for this function:

sviInstance()

template<typename GUM_SCALAR>

static StructuredPlaner< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::sviInstance ( GUM_SCALAR  discountFactor = 0.9, GUM_SCALAR  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 94 of file structuredPlaner.h.

Referenced by gum::SDYNA::RandomTreeInstance(), and gum::SDYNA::spitiInstance().

                                              {
      return new StructuredPlaner< GUM_SCALAR >(
         new TreeOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

Here is the caller graph for this function:

vFunction()

template<typename GUM_SCALAR>

INLINE const MultiDimFunctionGraph< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::vFunction ( )

inline

Returns a const ptr on the value function computed so far.

Definition at line 142 of file structuredPlaner.h.

                                                                  {
      return _vFunction;
    }

vFunctionSize()

template<typename GUM_SCALAR>

virtual Size gum::StructuredPlaner< GUM_SCALAR >::vFunctionSize ( )

inlinevirtual

Returns vFunction computed so far current size.

Implements gum::IPlanningStrategy< GUM_SCALAR >.

Definition at line 149 of file structuredPlaner.h.

                                 {
      return _vFunction != nullptr ? _vFunction->realSize() : 0;
    }

Member Data Documentation

__firstTime

template<typename GUM_SCALAR>

bool gum::StructuredPlaner< GUM_SCALAR >::__firstTime

private

Definition at line 380 of file structuredPlaner.h.

__threshold

template<typename GUM_SCALAR>

GUM_SCALAR gum::StructuredPlaner< GUM_SCALAR >::__threshold

private

The threshold value Whenever | V^{n} - V^{n+1} | < threshold, we consider that V ~ V*.

Definition at line 379 of file structuredPlaner.h.

_discountFactor

template<typename GUM_SCALAR>

GUM_SCALAR gum::StructuredPlaner< GUM_SCALAR >::_discountFactor

protected

Discount Factor used for infinite horizon planning.

Definition at line 363 of file structuredPlaner.h.

_elVarSeq

template<typename GUM_SCALAR>

Set< const DiscreteVariable* > gum::StructuredPlaner< GUM_SCALAR >::_elVarSeq

protected

A Set to eleminate primed variables.

Definition at line 358 of file structuredPlaner.h.

_fmdp

template<typename GUM_SCALAR>

const FMDP< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::_fmdp

protected

The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions )

Definition at line 338 of file structuredPlaner.h.

Referenced by gum::StructuredPlaner< double >::fmdp().

_operator

template<typename GUM_SCALAR>

IOperatorStrategy< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::_operator

protected

Definition at line 365 of file structuredPlaner.h.

_optimalPolicy

template<typename GUM_SCALAR>

MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< GUM_SCALAR >::_optimalPolicy

protected

The associated optimal policy.

Warning

Leaves are ActionSet which contains the ids of the best actions While this is sufficient to be exploited, to be understood by a human somme translation from the _fmdp is required. optimalPolicy2String do this job.

Definition at line 353 of file structuredPlaner.h.

Referenced by gum::StructuredPlaner< double >::optimalPolicy(), and gum::StructuredPlaner< double >::optimalPolicySize().

_verbose

template<typename GUM_SCALAR>

bool gum::StructuredPlaner< GUM_SCALAR >::_verbose

protected

Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Definition at line 371 of file structuredPlaner.h.

_vFunction

template<typename GUM_SCALAR>

MultiDimFunctionGraph< GUM_SCALAR >* gum::StructuredPlaner< GUM_SCALAR >::_vFunction

protected

The Value Function computed iteratively.

Definition at line 343 of file structuredPlaner.h.

Referenced by gum::StructuredPlaner< double >::vFunction(), and gum::StructuredPlaner< double >::vFunctionSize().

---

The documentation for this class was generated from the following files:

• agrum/FMDP/planning/structuredPlaner.h
• agrum/FMDP/planning/structuredPlaner_tpl.h