gum::AdaptiveRMaxPlaner Class Reference

<agrum/FMDP/planning/adaptiveRMaxPlaner.h> More...

#include <adaptiveRMaxPlaner.h>

Inheritance diagram for gum::AdaptiveRMaxPlaner:

Collaboration diagram for gum::AdaptiveRMaxPlaner:

Public Member Functions
Planning Methods
void	initialize (const FMDP< double > *fmdp)
	Initializes data structure needed for making the planning. More...
void	makePlanning (Idx nbStep=1000000)
	Performs a value iteration. More...
Datastructure access methods
INLINE const FMDP< double > *	fmdp ()
	Returns a const ptr on the Factored Markov Decision Process on which we're planning. More...
INLINE const MultiDimFunctionGraph< double > *	vFunction ()
	Returns a const ptr on the value function computed so far. More...
virtual Size	vFunctionSize ()
	Returns vFunction computed so far current size. More...
INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy ()
	Returns the best policy obtained so far. More...
virtual Size	optimalPolicySize ()
	Returns optimalPolicy computed so far current size. More...
std::string	optimalPolicy2String ()
	Provide a better toDot for the optimal policy where the leaves have the action name instead of its id. More...

Static Public Member Functions
static AdaptiveRMaxPlaner *	ReducedAndOrderedInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static AdaptiveRMaxPlaner *	TreeInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	spumddInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	sviInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)

Protected Attributes
const FMDP< double > *	_fmdp
	The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ) More...
MultiDimFunctionGraph< double > *	_vFunction
	The Value Function computed iteratively. More...
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	_optimalPolicy
	The associated optimal policy. More...
Set< const DiscreteVariable * >	_elVarSeq
	A Set to eleminate primed variables. More...
double	_discountFactor
	Discount Factor used for infinite horizon planning. More...
IOperatorStrategy< double > *	_operator
bool	_verbose
	Boolean used to indcates whether or not iteration informations should be displayed on terminal. More...

Protected Member Functions
Value Iteration Methods
virtual void	_initVFunction ()
	Performs a single step of value iteration. More...
virtual MultiDimFunctionGraph< double > *	_valueIteration ()
	Performs a single step of value iteration. More...
Optimal policy extraction methods
virtual void	_evalPolicy ()
	Perform the required tasks to extract an optimal policy. More...
Value Iteration Methods
virtual MultiDimFunctionGraph< double > *	_evalQaction (const MultiDimFunctionGraph< double > *, Idx)
	Performs the P(s'|s,a).V^{t-1}(s') part of the value itération. More...
virtual MultiDimFunctionGraph< double > *	_maximiseQactions (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs max_a Q(s,a) More...
virtual MultiDimFunctionGraph< double > *	_minimiseFunctions (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs min_i F_i. More...
virtual MultiDimFunctionGraph< double > *	_addReward (MultiDimFunctionGraph< double > *function, Idx actionId=0)
	Perform the R(s) + gamma . function. More...
Optimal policy extraction methods
MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	_makeArgMax (const MultiDimFunctionGraph< double > *Qaction, Idx actionId)
	Creates a copy of given Qaction that can be exploit by a Argmax. More...
virtual MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	_argmaximiseQactions (std::vector< MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *> &)
	Performs argmax_a Q(s,a) More...
void	_extractOptimalPolicy (const MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
	From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet. More...

Constructor & destructor.
	AdaptiveRMaxPlaner (IOperatorStrategy< double > *opi, double discountFactor, double epsilon, const ILearningStrategy *learner, bool verbose)
	Default constructor. More...
	~AdaptiveRMaxPlaner ()
	Default destructor. More...

Incremental methods
HashTable< Idx, StatesCounter *>	__counterTable
HashTable< Idx, bool >	__initializedTable
bool	__initialized
void	checkState (const Instantiation &newState, Idx actionId)

Incremental methods
void	setOptimalStrategy (const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *optPol)
virtual ActionSet	stateOptimalPolicy (const Instantiation &curState)
const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	_optPol
ActionSet	_allActions

Detailed Description

<agrum/FMDP/planning/adaptiveRMaxPlaner.h>

A class to find optimal policy for a given FMDP.

Perform a RMax planning on given in parameter factored markov decision process

Definition at line 50 of file adaptiveRMaxPlaner.h.

Constructor & Destructor Documentation

AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::AdaptiveRMaxPlaner ( IOperatorStrategy< double > *  opi, double  discountFactor, double  epsilon, const ILearningStrategy *  learner, bool  verbose  )

private

Default constructor.

Definition at line 60 of file adaptiveRMaxPlaner.cpp.

Referenced by ReducedAndOrderedInstance(), and TreeInstance().

                                                                           :
      StructuredPlaner(opi, discountFactor, epsilon, verbose),
      IDecisionStrategy(), __fmdpLearner(learner), __initialized(false) {
    GUM_CONSTRUCTOR(AdaptiveRMaxPlaner);
  }

Here is the caller graph for this function:

~AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::~AdaptiveRMaxPlaner

(

)

Default destructor.

Definition at line 73 of file adaptiveRMaxPlaner.cpp.

References __counterTable.

Referenced by TreeInstance().

                                          {
    GUM_DESTRUCTOR(AdaptiveRMaxPlaner);

    for (HashTableIteratorSafe< Idx, StatesCounter* > scIter =
            __counterTable.beginSafe();
         scIter != __counterTable.endSafe();
         ++scIter)
      delete scIter.val();
  }

Here is the caller graph for this function:

Member Function Documentation

__clearTables()

void gum::AdaptiveRMaxPlaner::__clearTables ( )

private

Definition at line 342 of file adaptiveRMaxPlaner.cpp.

References __actionsBoolTable, __actionsRMaxTable, gum::FMDP< GUM_SCALAR >::endActions(), and gum::StructuredPlaner< double >::fmdp().

Referenced by makePlanning(), and TreeInstance().

                                         {
    for (auto actionIter = this->fmdp()->beginActions();
         actionIter != this->fmdp()->endActions();
         ++actionIter) {
      delete __actionsBoolTable[*actionIter];
      delete __actionsRMaxTable[*actionIter];
    }
    __actionsRMaxTable.clear();
    __actionsBoolTable.clear();
  }

Here is the call graph for this function:

Here is the caller graph for this function:

__makeRMaxFunctionGraphs()

void gum::AdaptiveRMaxPlaner::__makeRMaxFunctionGraphs ( )

private

Definition at line 235 of file adaptiveRMaxPlaner.cpp.

References __actionsBoolTable, __actionsRMaxTable, __counterTable, __fmdpLearner, __rmax, __rThreshold, __visitLearner(), gum::StructuredPlaner< double >::_discountFactor, gum::StructuredPlaner< double >::_maximiseQactions(), gum::StructuredPlaner< double >::_minimiseFunctions(), gum::StructuredPlaner< double >::_operator, gum::FMDP< GUM_SCALAR >::beginActions(), gum::FMDP< GUM_SCALAR >::beginVariables(), gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::clean(), gum::FMDP< GUM_SCALAR >::endActions(), gum::FMDP< GUM_SCALAR >::endVariables(), gum::StructuredPlaner< double >::fmdp(), gum::IOperatorStrategy< GUM_SCALAR >::getFunctionInstance(), gum::IVisitableGraphLearner::insertSetOfVars(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::manager(), gum::ILearningStrategy::modaMax(), gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::reduce(), gum::ILearningStrategy::rMax(), gum::IVisitableGraphLearner::root(), and gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::setRootNode().

Referenced by makePlanning(), and TreeInstance().

                                                    {
    __rThreshold =
       __fmdpLearner->modaMax() * 5 > 30 ? __fmdpLearner->modaMax() * 5 : 30;
    __rmax = __fmdpLearner->rMax() / (1.0 - this->_discountFactor);

    for (auto actionIter = this->fmdp()->beginActions();
         actionIter != this->fmdp()->endActions();
         ++actionIter) {
      std::vector< MultiDimFunctionGraph< double >* > rmaxs;
      std::vector< MultiDimFunctionGraph< double >* > boolQs;

      for (auto varIter = this->fmdp()->beginVariables();
           varIter != this->fmdp()->endVariables();
           ++varIter) {
        const IVisitableGraphLearner* visited = __counterTable[*actionIter];

        MultiDimFunctionGraph< double >* varRMax =
           this->_operator->getFunctionInstance();
        MultiDimFunctionGraph< double >* varBoolQ =
           this->_operator->getFunctionInstance();

        visited->insertSetOfVars(varRMax);
        visited->insertSetOfVars(varBoolQ);

        std::pair< NodeId, NodeId > rooty =
           __visitLearner(visited, visited->root(), varRMax, varBoolQ);
        varRMax->manager()->setRootNode(rooty.first);
        varRMax->manager()->reduce();
        varRMax->manager()->clean();
        varBoolQ->manager()->setRootNode(rooty.second);
        varBoolQ->manager()->reduce();
        varBoolQ->manager()->clean();

        rmaxs.push_back(varRMax);
        boolQs.push_back(varBoolQ);

        //          std::cout << RECASTED(this->_fmdp->transition(*actionIter,
        //          *varIter))->toDot() << std::endl;
        //          for( auto varIter2 =
        //          RECASTED(this->_fmdp->transition(*actionIter,
        //          *varIter))->variablesSequence().beginSafe(); varIter2 !=
        //          RECASTED(this->_fmdp->transition(*actionIter,
        //          *varIter))->variablesSequence().endSafe(); ++varIter2 )
        //              std::cout << (*varIter2)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varRMax->toDot() << std::endl;
        //          for( auto varIter =
        //          varRMax->variablesSequence().beginSafe(); varIter !=
        //          varRMax->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varBoolQ->toDot() << std::endl;
        //          for( auto varIter =
        //          varBoolQ->variablesSequence().beginSafe(); varIter !=
        //          varBoolQ->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;
      }

      //        std::cout << "Maximising" << std::endl;
      __actionsRMaxTable.insert(*actionIter, this->_maximiseQactions(rmaxs));
      __actionsBoolTable.insert(*actionIter, this->_minimiseFunctions(boolQs));
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

__visitLearner()

std::pair< NodeId, NodeId > gum::AdaptiveRMaxPlaner::__visitLearner ( const IVisitableGraphLearner *  visited, NodeId  currentNodeId, MultiDimFunctionGraph< double > *  rmax, MultiDimFunctionGraph< double > *  boolQ  )

private

Definition at line 306 of file adaptiveRMaxPlaner.cpp.

References __rmax, __rThreshold, gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::addInternalNode(), gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::addTerminalNode(), gum::DiscreteVariable::domainSize(), gum::IVisitableGraphLearner::isTerminal(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::manager(), gum::IVisitableGraphLearner::nodeNbObservation(), gum::IVisitableGraphLearner::nodeSon(), gum::IVisitableGraphLearner::nodeVar(), and SOA_ALLOCATE.

Referenced by __makeRMaxFunctionGraphs(), and TreeInstance().

                                                                                {
    std::pair< NodeId, NodeId > rep;
    if (visited->isTerminal(currentNodeId)) {
      rep.first = rmax->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < __rThreshold ? __rmax : 0.0);
      rep.second = boolQ->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < __rThreshold ? 0.0 : 1.0);
      return rep;
    }

    NodeId* rmaxsons = static_cast< NodeId* >(SOA_ALLOCATE(
       sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));
    NodeId* bqsons = static_cast< NodeId* >(SOA_ALLOCATE(
       sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));

    for (Idx moda = 0; moda < visited->nodeVar(currentNodeId)->domainSize();
         ++moda) {
      std::pair< NodeId, NodeId > sonp = __visitLearner(
         visited, visited->nodeSon(currentNodeId, moda), rmax, boolQ);
      rmaxsons[moda] = sonp.first;
      bqsons[moda] = sonp.second;
    }

    rep.first =
       rmax->manager()->addInternalNode(visited->nodeVar(currentNodeId), rmaxsons);
    rep.second =
       boolQ->manager()->addInternalNode(visited->nodeVar(currentNodeId), bqsons);
    return rep;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

_addReward()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::_addReward ( MultiDimFunctionGraph< double > *  function, Idx  actionId = 0  )

protectedvirtualinherited

Perform the R(s) + gamma . function.

Warning

function is deleted, new one is returned

Definition at line 405 of file structuredPlaner_tpl.h.

References gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::add(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::copyAndMultiplyByScalar(), and RECAST.

Referenced by _evalPolicy(), and _valueIteration().

                                                              {
    // *****************************************************************************************
    // ... we multiply the result by the discount factor, ...
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndMultiplyByScalar(*Vold, this->_discountFactor);
    delete Vold;

    // *****************************************************************************************
    // ... and finally add reward
    newVFunction = _operator->add(newVFunction, RECAST(_fmdp->reward(actionId)));

    return newVFunction;
  }

_argmaximiseQactions()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::_argmaximiseQactions ( std::vector< MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * > &  qActionsSet )

protectedvirtualinherited

Performs argmax_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 537 of file structuredPlaner_tpl.h.

Referenced by _evalPolicy().

                        {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
         qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->argmaximize(newVFunction, qAction);
    }

    return newVFunction;
  }

_evalPolicy()

void gum::AdaptiveRMaxPlaner::_evalPolicy ( )

protectedvirtual

Perform the required tasks to extract an optimal policy.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 191 of file adaptiveRMaxPlaner.cpp.

References __actionsBoolTable, __actionsRMaxTable, gum::StructuredPlaner< double >::_addReward(), gum::StructuredPlaner< double >::_argmaximiseQactions(), gum::StructuredPlaner< double >::_evalQaction(), gum::StructuredPlaner< double >::_extractOptimalPolicy(), gum::StructuredPlaner< double >::_fmdp, gum::StructuredPlaner< double >::_makeArgMax(), gum::StructuredPlaner< double >::_operator, gum::StructuredPlaner< double >::_vFunction, gum::FMDP< GUM_SCALAR >::beginActions(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::copyAndReassign(), gum::FMDP< GUM_SCALAR >::endActions(), gum::IOperatorStrategy< GUM_SCALAR >::getFunctionInstance(), gum::FMDP< GUM_SCALAR >::mapMainPrime(), gum::IOperatorStrategy< GUM_SCALAR >::maximize(), and gum::IOperatorStrategy< GUM_SCALAR >::multiply().

Referenced by TreeInstance().

                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndReassign(*_vFunction, _fmdp->mapMainPrime());

    std::vector<
       MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >* >
       argMaxQActionsSet;
    // *****************************************************************************************
    // For each action
    for (auto actionIter = _fmdp->beginActions();
         actionIter != _fmdp->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< double >* qAction =
         this->_evalQaction(newVFunction, *actionIter);

      qAction = this->_addReward(qAction, *actionIter);

      qAction = this->_operator->maximize(
         __actionsRMaxTable[*actionIter],
         this->_operator->multiply(qAction, __actionsBoolTable[*actionIter], 1),
         2);

      argMaxQActionsSet.push_back(_makeArgMax(qAction, *actionIter));
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >*
       argMaxVFunction = _argmaximiseQactions(argMaxQActionsSet);

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    _extractOptimalPolicy(argMaxVFunction);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

_evalQaction()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::_evalQaction ( const MultiDimFunctionGraph< double > *  Vold, Idx  actionId  )

protectedvirtualinherited

Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

Definition at line 350 of file structuredPlaner_tpl.h.

Referenced by _evalPolicy(), and _valueIteration().

                                                                       {
    // ******************************************************************************
    // Initialisation :
    // Creating a copy of last Vfunction to deduce from the new Qaction
    // And finding the first var to eleminate (the one at the end)

    return _operator->regress(Vold, actionId, this->_fmdp, this->_elVarSeq);
  }

_extractOptimalPolicy()

void gum::StructuredPlaner< double >::_extractOptimalPolicy ( const MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > *  optimalValueFunction )

protectedinherited

From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.

Warning

deallocate the argmax optimal value function

Definition at line 561 of file structuredPlaner_tpl.h.

Referenced by _evalPolicy().

                                    {
    _optimalPolicy->clear();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter =
            argMaxOptimalValueFunction->variablesSequence().beginSafe();
         varIter != argMaxOptimalValueFunction->variablesSequence().endSafe();
         ++varIter)
      _optimalPolicy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    _optimalPolicy->manager()->setRootNode(__recurExtractOptPol(
       argMaxOptimalValueFunction->root(), argMaxOptimalValueFunction, src2dest));

    delete argMaxOptimalValueFunction;
  }

_initVFunction()

void gum::AdaptiveRMaxPlaner::_initVFunction ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 130 of file adaptiveRMaxPlaner.cpp.

References gum::StructuredPlaner< double >::_fmdp, gum::StructuredPlaner< double >::_operator, gum::StructuredPlaner< double >::_vFunction, gum::IOperatorStrategy< GUM_SCALAR >::add(), gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::addTerminalNode(), gum::FMDP< GUM_SCALAR >::beginActions(), gum::FMDP< GUM_SCALAR >::endActions(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::manager(), RECASTED, gum::FMDP< GUM_SCALAR >::reward(), and gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::setRootNode().

Referenced by TreeInstance().

                                          {
    _vFunction->manager()->setRootNode(
       _vFunction->manager()->addTerminalNode(0.0));
    for (auto actionIter = _fmdp->beginActions();
         actionIter != _fmdp->endActions();
         ++actionIter)
      _vFunction = this->_operator->add(
         _vFunction, RECASTED(this->_fmdp->reward(*actionIter)), 1);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

_makeArgMax()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::_makeArgMax ( const MultiDimFunctionGraph< double > *  Qaction, Idx  actionId  )

protectedinherited

Creates a copy of given Qaction that can be exploit by a Argmax.

Hence, this step consists in replacing each lea by an ArgMaxSet containing the value of the leaf and the actionId of the Qaction

Parameters

Qaction	: the function graph we want to transform
actionId	: the action Id associated to that graph

Warning

delete the original Qaction, returns its conversion

Definition at line 479 of file structuredPlaner_tpl.h.

References gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::add(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::manager(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::root(), gum::MultiDimFunctionGraphManager< GUM_SCALAR, TerminalNodePolicy >::setRootNode(), and gum::MultiDimImplementation< GUM_SCALAR >::variablesSequence().

Referenced by _evalPolicy().

                                                                          {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       amcpy = _operator->getArgMaxFunctionInstance();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter =
            qAction->variablesSequence().beginSafe();
         varIter != qAction->variablesSequence().endSafe();
         ++varIter)
      amcpy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    amcpy->manager()->setRootNode(
       __recurArgMaxCopy(qAction->root(), actionId, qAction, amcpy, src2dest));

    delete qAction;
    return amcpy;
  }

_maximiseQactions()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::_maximiseQactions ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs max_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 366 of file structuredPlaner_tpl.h.

Referenced by __makeRMaxFunctionGraphs(), and _valueIteration().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->maximize(newVFunction, qAction);
    }

    return newVFunction;
  }

_minimiseFunctions()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::_minimiseFunctions ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs min_i F_i.

Warning

Performs also the deallocation of the F_i

Definition at line 386 of file structuredPlaner_tpl.h.

Referenced by __makeRMaxFunctionGraphs().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = _operator->minimize(newVFunction, qAction);
    }

    return newVFunction;
  }

_valueIteration()

MultiDimFunctionGraph< double > * gum::AdaptiveRMaxPlaner::_valueIteration ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 143 of file adaptiveRMaxPlaner.cpp.

References __actionsBoolTable, __actionsRMaxTable, gum::StructuredPlaner< double >::_addReward(), gum::StructuredPlaner< double >::_evalQaction(), gum::StructuredPlaner< double >::_fmdp, gum::StructuredPlaner< double >::_maximiseQactions(), gum::StructuredPlaner< double >::_operator, gum::StructuredPlaner< double >::_vFunction, gum::FMDP< GUM_SCALAR >::beginActions(), gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::copyAndReassign(), gum::FMDP< GUM_SCALAR >::endActions(), gum::IOperatorStrategy< GUM_SCALAR >::getFunctionInstance(), gum::FMDP< GUM_SCALAR >::mapMainPrime(), gum::IOperatorStrategy< GUM_SCALAR >::maximize(), and gum::IOperatorStrategy< GUM_SCALAR >::multiply().

Referenced by TreeInstance().

                                                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction =
       _operator->getFunctionInstance();
    newVFunction->copyAndReassign(*_vFunction, _fmdp->mapMainPrime());

    // *****************************************************************************************
    // For each action
    std::vector< MultiDimFunctionGraph< double >* > qActionsSet;
    for (auto actionIter = _fmdp->beginActions();
         actionIter != _fmdp->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< double >* qAction =
         _evalQaction(newVFunction, *actionIter);

      // *******************************************************************************************
      // Next, we add the reward
      qAction = _addReward(qAction, *actionIter);

      qAction = this->_operator->maximize(
         __actionsRMaxTable[*actionIter],
         this->_operator->multiply(qAction, __actionsBoolTable[*actionIter], 1),
         2);

      qActionsSet.push_back(qAction);
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    newVFunction = _maximiseQactions(qActionsSet);

    return newVFunction;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

checkState()

void gum::AdaptiveRMaxPlaner::checkState ( const Instantiation &  newState, Idx  actionId  )

inlinevirtual

Implements gum::IDecisionStrategy.

Definition at line 198 of file adaptiveRMaxPlaner.h.

References __counterTable, and __initializedTable.

                                                                 {
      if (!__initializedTable[actionId]) {
        __counterTable[actionId]->reset(newState);
        __initializedTable[actionId] = true;
      } else
        __counterTable[actionId]->incState(newState);
    }

fmdp()

INLINE const FMDP< double >* gum::StructuredPlaner< double >::fmdp ( )

inlineinherited

Returns a const ptr on the Factored Markov Decision Process on which we're planning.

Definition at line 134 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::_fmdp.

Referenced by __clearTables(), __makeRMaxFunctionGraphs(), and TreeInstance().

{ return _fmdp; }

initialize()

void gum::AdaptiveRMaxPlaner::initialize ( const FMDP< double > *  fmdp )

virtual

Initializes data structure needed for making the planning.

Warning

No calling this methods before starting the first makePlaninng will surely and definitely result in a crash

Reimplemented from gum::IDecisionStrategy.

Definition at line 94 of file adaptiveRMaxPlaner.cpp.

References __counterTable, __initialized, __initializedTable, gum::FMDP< GUM_SCALAR >::beginActions(), gum::FMDP< GUM_SCALAR >::endActions(), gum::IDecisionStrategy::initialize(), gum::StructuredPlaner< GUM_SCALAR >::initialize(), and gum::HashTable< Key, Val, Alloc >::insert().

Referenced by TreeInstance().

                                                                {
    if (!__initialized) {
      StructuredPlaner::initialize(fmdp);
      IDecisionStrategy::initialize(fmdp);
      for (auto actionIter = fmdp->beginActions();
           actionIter != fmdp->endActions();
           ++actionIter) {
        __counterTable.insert(*actionIter, new StatesCounter());
        __initializedTable.insert(*actionIter, false);
      }
      __initialized = true;
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

makePlanning()

void gum::AdaptiveRMaxPlaner::makePlanning ( Idx  nbStep = 1000000 )

virtual

Performs a value iteration.

Parameters

nbStep

: enables you to specify how many value iterations you wish to do. makePlanning will then stop whether when optimal value function is reach or when nbStep have been performed

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 111 of file adaptiveRMaxPlaner.cpp.

References __clearTables(), __makeRMaxFunctionGraphs(), and gum::StructuredPlaner< GUM_SCALAR >::makePlanning().

Referenced by TreeInstance().

                                                  {
    __makeRMaxFunctionGraphs();

    StructuredPlaner::makePlanning(nbStep);

    __clearTables();
  }

Here is the call graph for this function:

Here is the caller graph for this function:

optimalPolicy()

INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::optimalPolicy ( )

inlinevirtualinherited

Returns the best policy obtained so far.

Implements gum::IPlanningStrategy< double >.

Definition at line 154 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::_optimalPolicy.

                                 {
      return _optimalPolicy;
    }

optimalPolicy2String()

std::string gum::StructuredPlaner< double >::optimalPolicy2String ( )

virtualinherited

Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.

Implements gum::IPlanningStrategy< double >.

Definition at line 102 of file structuredPlaner_tpl.h.

References gum::ActionSet::beginSafe(), gum::HashTable< Key, Val, Alloc >::beginSafe(), gum::Link< T >::element(), gum::ActionSet::endSafe(), gum::HashTable< Key, Val, Alloc >::endSafe(), gum::Set< Key, Alloc >::exists(), gum::HashTable< Key, Val, Alloc >::exists(), gum::HashTable< Key, Val, Alloc >::insert(), gum::HashTable< Key, Val, Alloc >::key(), gum::DiscreteVariable::label(), gum::Variable::name(), gum::InternalNode::nbSons(), gum::Link< T >::nextLink(), gum::InternalNode::nodeVar(), and gum::InternalNode::son().

                                                                 {
    // ************************************************************************
    // Discarding the case where no \pi* have been computed
    if (!_optimalPolicy || _optimalPolicy->root() == 0)
      return "NO OPTIMAL POLICY CALCULATED YET";

    // ************************************************************************
    // Initialisation

    // Declaration of the needed string stream
    std::stringstream output;
    std::stringstream terminalStream;
    std::stringstream nonTerminalStream;
    std::stringstream arcstream;

    // First line for the toDot
    output << std::endl << "digraph \" OPTIMAL POLICY \" {" << std::endl;

    // Form line for the internal node stream en the terminal node stream
    terminalStream << "node [shape = box];" << std::endl;
    nonTerminalStream << "node [shape = ellipse];" << std::endl;

    // For somme clarity in the final string
    std::string tab = "\t";

    // To know if we already checked a node or not
    Set< NodeId > visited;

    // FIFO of nodes to visit
    std::queue< NodeId > fifo;

    // Loading the FIFO
    fifo.push(_optimalPolicy->root());
    visited << _optimalPolicy->root();


    // ************************************************************************
    // Main loop
    while (!fifo.empty()) {
      // Node to visit
      NodeId currentNodeId = fifo.front();
      fifo.pop();

      // Checking if it is terminal
      if (_optimalPolicy->isTerminalNode(currentNodeId)) {
        // Get back the associated ActionSet
        ActionSet ase = _optimalPolicy->nodeValue(currentNodeId);

        // Creating a line for this node
        terminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                       << " [label=\"" << currentNodeId << " - ";

        // Enumerating and adding to the line the associated optimal actions
        for (SequenceIteratorSafe< Idx > valIter = ase.beginSafe();
             valIter != ase.endSafe();
             ++valIter)
          terminalStream << _fmdp->actionName(*valIter) << " ";

        // Terminating line
        terminalStream << "\"];" << std::endl;
        continue;
      }

      // Either wise
      {
        // Geting back the associated internal node
        const InternalNode* currentNode = _optimalPolicy->node(currentNodeId);

        // Creating a line in internalnode stream for this node
        nonTerminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                          << " [label=\"" << currentNodeId << " - "
                          << currentNode->nodeVar()->name() << "\"];" << std::endl;

        // Going through the sons and agregating them according the the sons Ids
        HashTable< NodeId, LinkedList< Idx >* > sonMap;
        for (Idx sonIter = 0; sonIter < currentNode->nbSons(); ++sonIter) {
          if (!visited.exists(currentNode->son(sonIter))) {
            fifo.push(currentNode->son(sonIter));
            visited << currentNode->son(sonIter);
          }
          if (!sonMap.exists(currentNode->son(sonIter)))
            sonMap.insert(currentNode->son(sonIter), new LinkedList< Idx >());
          sonMap[currentNode->son(sonIter)]->addLink(sonIter);
        }

        // Adding to the arc stram
        for (auto sonIter = sonMap.beginSafe(); sonIter != sonMap.endSafe();
             ++sonIter) {
          arcstream << tab << currentNodeId << " -> " << sonIter.key()
                    << " [label=\" ";
          Link< Idx >* modaIter = sonIter.val()->list();
          while (modaIter) {
            arcstream << currentNode->nodeVar()->label(modaIter->element());
            if (modaIter->nextLink()) arcstream << ", ";
            modaIter = modaIter->nextLink();
          }
          arcstream << "\",color=\"#00ff00\"];" << std::endl;
          delete sonIter.val();
        }
      }
    }

    // Terminating
    output << terminalStream.str() << std::endl
           << nonTerminalStream.str() << std::endl
           << arcstream.str() << std::endl
           << "}" << std::endl;

    return output.str();
  }

optimalPolicySize()

virtual Size gum::StructuredPlaner< double >::optimalPolicySize ( )

inlinevirtualinherited

Returns optimalPolicy computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 161 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::__recurArgMaxCopy(), gum::StructuredPlaner< GUM_SCALAR >::__recurExtractOptPol(), gum::StructuredPlaner< GUM_SCALAR >::__transferActionIds(), gum::StructuredPlaner< GUM_SCALAR >::_addReward(), gum::StructuredPlaner< GUM_SCALAR >::_argmaximiseQactions(), gum::StructuredPlaner< GUM_SCALAR >::_evalPolicy(), gum::StructuredPlaner< GUM_SCALAR >::_evalQaction(), gum::StructuredPlaner< GUM_SCALAR >::_extractOptimalPolicy(), gum::StructuredPlaner< GUM_SCALAR >::_initVFunction(), gum::StructuredPlaner< GUM_SCALAR >::_makeArgMax(), gum::StructuredPlaner< GUM_SCALAR >::_maximiseQactions(), gum::StructuredPlaner< GUM_SCALAR >::_minimiseFunctions(), gum::StructuredPlaner< GUM_SCALAR >::_optimalPolicy, gum::StructuredPlaner< GUM_SCALAR >::_valueIteration(), gum::StructuredPlaner< GUM_SCALAR >::fmdp(), gum::StructuredPlaner< GUM_SCALAR >::initialize(), gum::StructuredPlaner< GUM_SCALAR >::makePlanning(), and gum::StructuredPlaner< GUM_SCALAR >::optimalPolicy2String().

                                     {
      return _optimalPolicy != nullptr ? _optimalPolicy->realSize() : 0;
    }

ReducedAndOrderedInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::ReducedAndOrderedInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 62 of file adaptiveRMaxPlaner.h.

References AdaptiveRMaxPlaner().

Referenced by gum::SDYNA::RMaxMDDInstance().

                                                                          {
      return new AdaptiveRMaxPlaner(new MDDOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

Here is the call graph for this function:

Here is the caller graph for this function:

setOptimalStrategy()

void gum::IDecisionStrategy::setOptimalStrategy ( const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *  optPol )

inlineinherited

Definition at line 87 of file IDecisionStrategy.h.

References gum::IDecisionStrategy::_optPol.

Referenced by gum::SDYNA::makePlanning().

                                                                                {
      _optPol =
         const_cast< MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* >(
            optPol);
    }

Here is the caller graph for this function:

spumddInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::spumddInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 77 of file structuredPlaner.h.

                                                 {
      return new StructuredPlaner< GUM_SCALAR >(
         new MDDOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

stateOptimalPolicy()

virtual ActionSet gum::IDecisionStrategy::stateOptimalPolicy ( const Instantiation &  curState )

inlinevirtualinherited

Reimplemented in gum::E_GreedyDecider, and gum::RandomDecider.

Definition at line 94 of file IDecisionStrategy.h.

References gum::IDecisionStrategy::_allActions, and gum::IDecisionStrategy::_optPol.

Referenced by gum::E_GreedyDecider::stateOptimalPolicy(), and gum::SDYNA::takeAction().

                                                                        {
      return (_optPol && _optPol->realSize() != 0) ? _optPol->get(curState)
                                                   : _allActions;
    }

Here is the caller graph for this function:

sviInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::sviInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 91 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::StructuredPlaner(), and gum::StructuredPlaner< GUM_SCALAR >::~StructuredPlaner().

                                              {
      return new StructuredPlaner< GUM_SCALAR >(
         new TreeOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

TreeInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::TreeInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 76 of file adaptiveRMaxPlaner.h.

References __clearTables(), __makeRMaxFunctionGraphs(), __visitLearner(), _evalPolicy(), _initVFunction(), _valueIteration(), AdaptiveRMaxPlaner(), gum::StructuredPlaner< double >::fmdp(), initialize(), makePlanning(), and ~AdaptiveRMaxPlaner().

Referenced by gum::SDYNA::RMaxTreeInstance().

                                                                   {
      return new AdaptiveRMaxPlaner(new TreeOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

Here is the call graph for this function:

Here is the caller graph for this function:

vFunction()

INLINE const MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::vFunction ( )

inlineinherited

Returns a const ptr on the value function computed so far.

Definition at line 139 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::_vFunction.

                                                                  {
      return _vFunction;
    }

vFunctionSize()

virtual Size gum::StructuredPlaner< double >::vFunctionSize ( )

inlinevirtualinherited

Returns vFunction computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 146 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::_vFunction, and gum::MultiDimFunctionGraph< GUM_SCALAR, TerminalNodePolicy >::realSize().

                                 {
      return _vFunction != nullptr ? _vFunction->realSize() : 0;
    }

Member Data Documentation

__actionsBoolTable

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::__actionsBoolTable

private

Definition at line 186 of file adaptiveRMaxPlaner.h.

Referenced by __clearTables(), __makeRMaxFunctionGraphs(), _evalPolicy(), and _valueIteration().

__actionsRMaxTable

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::__actionsRMaxTable

private

Definition at line 185 of file adaptiveRMaxPlaner.h.

Referenced by __clearTables(), __makeRMaxFunctionGraphs(), _evalPolicy(), and _valueIteration().

__counterTable

HashTable< Idx, StatesCounter* > gum::AdaptiveRMaxPlaner::__counterTable

private

Definition at line 207 of file adaptiveRMaxPlaner.h.

Referenced by __makeRMaxFunctionGraphs(), checkState(), initialize(), and ~AdaptiveRMaxPlaner().

__fmdpLearner

const ILearningStrategy* gum::AdaptiveRMaxPlaner::__fmdpLearner

private

Definition at line 187 of file adaptiveRMaxPlaner.h.

Referenced by __makeRMaxFunctionGraphs().

__initialized

bool gum::AdaptiveRMaxPlaner::__initialized

private

Definition at line 210 of file adaptiveRMaxPlaner.h.

Referenced by initialize().

__initializedTable

HashTable< Idx, bool > gum::AdaptiveRMaxPlaner::__initializedTable

private

Definition at line 208 of file adaptiveRMaxPlaner.h.

Referenced by checkState(), and initialize().

__rmax

double gum::AdaptiveRMaxPlaner::__rmax

private

Definition at line 190 of file adaptiveRMaxPlaner.h.

Referenced by __makeRMaxFunctionGraphs(), and __visitLearner().

__rThreshold

double gum::AdaptiveRMaxPlaner::__rThreshold

private

Definition at line 189 of file adaptiveRMaxPlaner.h.

Referenced by __makeRMaxFunctionGraphs(), and __visitLearner().

_allActions

ActionSet gum::IDecisionStrategy::_allActions

protectedinherited

Definition at line 104 of file IDecisionStrategy.h.

Referenced by gum::IDecisionStrategy::initialize(), gum::RandomDecider::stateOptimalPolicy(), gum::E_GreedyDecider::stateOptimalPolicy(), and gum::IDecisionStrategy::stateOptimalPolicy().

_discountFactor

double gum::StructuredPlaner< double >::_discountFactor

protectedinherited

Discount Factor used for infinite horizon planning.

Definition at line 360 of file structuredPlaner.h.

Referenced by __makeRMaxFunctionGraphs().

_elVarSeq

Set< const DiscreteVariable* > gum::StructuredPlaner< double >::_elVarSeq

protectedinherited

A Set to eleminate primed variables.

Definition at line 355 of file structuredPlaner.h.

_fmdp

const FMDP< double >* gum::StructuredPlaner< double >::_fmdp

protectedinherited

The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions )

Definition at line 335 of file structuredPlaner.h.

Referenced by _evalPolicy(), _initVFunction(), and _valueIteration().

_operator

IOperatorStrategy< double >* gum::StructuredPlaner< double >::_operator

protectedinherited

Definition at line 362 of file structuredPlaner.h.

Referenced by __makeRMaxFunctionGraphs(), _evalPolicy(), _initVFunction(), and _valueIteration().

_optimalPolicy

MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::_optimalPolicy

protectedinherited

The associated optimal policy.

Warning

Leaves are ActionSet which contains the ids of the best actions While this is sufficient to be exploited, to be understood by a human somme translation from the _fmdp is required. optimalPolicy2String do this job.

Definition at line 350 of file structuredPlaner.h.

_optPol

const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::IDecisionStrategy::_optPol

protectedinherited

Definition at line 101 of file IDecisionStrategy.h.

Referenced by gum::IDecisionStrategy::initialize(), gum::IDecisionStrategy::setOptimalStrategy(), and gum::IDecisionStrategy::stateOptimalPolicy().

_verbose

bool gum::StructuredPlaner< double >::_verbose

protectedinherited

Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Definition at line 368 of file structuredPlaner.h.

_vFunction

MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::_vFunction

protectedinherited

The Value Function computed iteratively.

Definition at line 340 of file structuredPlaner.h.

Referenced by _evalPolicy(), _initVFunction(), and _valueIteration().

---

The documentation for this class was generated from the following files:

• agrum/FMDP/planning/adaptiveRMaxPlaner.h
• agrum/FMDP/planning/adaptiveRMaxPlaner.cpp