gum::AdaptiveRMaxPlaner Class Reference

<agrum/FMDP/planning/adaptiveRMaxPlaner.h> More...

#include <adaptiveRMaxPlaner.h>

Inheritance diagram for gum::AdaptiveRMaxPlaner:

Collaboration diagram for gum::AdaptiveRMaxPlaner:

Public Member Functions
Planning Methods
void	initialize (const FMDP< double > *fmdp)
	Initializes data structure needed for making the planning. More...
void	makePlanning (Idx nbStep=1000000)
	Performs a value iteration. More...
Datastructure access methods
INLINE const FMDP< double > *	fmdp ()
	Returns a const ptr on the Factored Markov Decision Process on which we're planning. More...
INLINE const MultiDimFunctionGraph< double > *	vFunction ()
	Returns a const ptr on the value function computed so far. More...
virtual Size	vFunctionSize ()
	Returns vFunction computed so far current size. More...
INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy ()
	Returns the best policy obtained so far. More...
virtual Size	optimalPolicySize ()
	Returns optimalPolicy computed so far current size. More...
std::string	optimalPolicy2String ()
	Provide a better toDot for the optimal policy where the leaves have the action name instead of its id. More...

Static Public Member Functions
static AdaptiveRMaxPlaner *	ReducedAndOrderedInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static AdaptiveRMaxPlaner *	TreeInstance (const ILearningStrategy *learner, double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	spumddInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)
static StructuredPlaner< double > *	sviInstance (double discountFactor=0.9, double epsilon=0.00001, bool verbose=true)

Protected Attributes
const FMDP< double > *	fmdp_
	The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ) More...
MultiDimFunctionGraph< double > *	vFunction_
	The Value Function computed iteratively. More...
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optimalPolicy_
	The associated optimal policy. More...
Set< const DiscreteVariable * >	elVarSeq_
	A Set to eleminate primed variables. More...
double	discountFactor_
	Discount Factor used for infinite horizon planning. More...
IOperatorStrategy< double > *	operator_
bool	verbose_
	Boolean used to indcates whether or not iteration informations should be displayed on terminal. More...

Protected Member Functions
Value Iteration Methods
virtual void	initVFunction_ ()
	Performs a single step of value iteration. More...
virtual MultiDimFunctionGraph< double > *	valueIteration_ ()
	Performs a single step of value iteration. More...
Optimal policy extraction methods
virtual void	evalPolicy_ ()
	Perform the required tasks to extract an optimal policy. More...
Value Iteration Methods
virtual MultiDimFunctionGraph< double > *	evalQaction_ (const MultiDimFunctionGraph< double > *, Idx)
	Performs the P(s'|s,a).V^{t-1}(s') part of the value itération. More...
virtual MultiDimFunctionGraph< double > *	maximiseQactions_ (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs max_a Q(s,a) More...
virtual MultiDimFunctionGraph< double > *	minimiseFunctions_ (std::vector< MultiDimFunctionGraph< double > *> &)
	Performs min_i F_i. More...
virtual MultiDimFunctionGraph< double > *	addReward_ (MultiDimFunctionGraph< double > *function, Idx actionId=0)
	Perform the R(s) + gamma . function. More...
Optimal policy extraction methods
MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	makeArgMax_ (const MultiDimFunctionGraph< double > *Qaction, Idx actionId)
	Creates a copy of given Qaction that can be exploit by a Argmax. More...
virtual MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *	argmaximiseQactions_ (std::vector< MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *> &)
	Performs argmax_a Q(s,a) More...
void	extractOptimalPolicy_ (const MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
	From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet. More...

Constructor & destructor.
	AdaptiveRMaxPlaner (IOperatorStrategy< double > *opi, double discountFactor, double epsilon, const ILearningStrategy *learner, bool verbose)
	Default constructor. More...
	~AdaptiveRMaxPlaner ()
	Default destructor. More...

Incremental methods
HashTable< Idx, StatesCounter *>	counterTable__
HashTable< Idx, bool >	initializedTable__
bool	initialized__
void	checkState (const Instantiation &newState, Idx actionId)

Incremental methods
void	setOptimalStrategy (const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *optPol)
virtual ActionSet	stateOptimalPolicy (const Instantiation &curState)
const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *	optPol_
ActionSet	allActions_

Detailed Description

<agrum/FMDP/planning/adaptiveRMaxPlaner.h>

A class to find optimal policy for a given FMDP.

Perform a RMax planning on given in parameter factored markov decision process

Definition at line 53 of file adaptiveRMaxPlaner.h.

Constructor & Destructor Documentation

AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::AdaptiveRMaxPlaner ( IOperatorStrategy< double > *  opi, double  discountFactor, double  epsilon, const ILearningStrategy *  learner, bool  verbose  )

private

Default constructor.

Definition at line 63 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                           :
      StructuredPlaner(opi, discountFactor, epsilon, verbose),
      IDecisionStrategy(), fmdpLearner__(learner), initialized__(false) {
    GUM_CONSTRUCTOR(AdaptiveRMaxPlaner);
  }

Here is the call graph for this function:

~AdaptiveRMaxPlaner()

gum::AdaptiveRMaxPlaner::~AdaptiveRMaxPlaner

(

)

Default destructor.

Definition at line 76 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                          {
    GUM_DESTRUCTOR(AdaptiveRMaxPlaner);

    for (HashTableIteratorSafe< Idx, StatesCounter* > scIter
         = counterTable__.beginSafe();
         scIter != counterTable__.endSafe();
         ++scIter)
      delete scIter.val();
  }

Here is the call graph for this function:

Member Function Documentation

addReward_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::addReward_ ( MultiDimFunctionGraph< double > *  function, Idx  actionId = 0  )

protectedvirtualinherited

Perform the R(s) + gamma . function.

Warning

function is deleted, new one is returned

Definition at line 409 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                    {
    // *****************************************************************************************
    // ... we multiply the result by the discount factor, ...
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction
       = operator_->getFunctionInstance();
    newVFunction->copyAndMultiplyByScalar(*Vold, this->discountFactor_);
    delete Vold;

    // *****************************************************************************************
    // ... and finally add reward
    newVFunction = operator_->add(newVFunction, RECAST(fmdp_->reward(actionId)));

    return newVFunction;
  }

argmaximiseQactions_()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::argmaximiseQactions_ ( std::vector< MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * > &  qActionsSet )

protectedvirtualinherited

Performs argmax_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 548 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                        {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       newVFunction
       = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
         qAction
         = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->argmaximize(newVFunction, qAction);
    }

    return newVFunction;
  }

checkState()

void gum::AdaptiveRMaxPlaner::checkState ( const Instantiation &  newState, Idx  actionId  )

inlinevirtual

Implements gum::IDecisionStrategy.

Definition at line 201 of file adaptiveRMaxPlaner.h.

                                                                 {
      if (!initializedTable__[actionId]) {
        counterTable__[actionId]->reset(newState);
        initializedTable__[actionId] = true;
      } else
        counterTable__[actionId]->incState(newState);
    }

clearTables__()

void gum::AdaptiveRMaxPlaner::clearTables__ ( )

private

Definition at line 350 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                         {
    for (auto actionIter = this->fmdp()->beginActions();
         actionIter != this->fmdp()->endActions();
         ++actionIter) {
      delete actionsBoolTable__[*actionIter];
      delete actionsRMaxTable__[*actionIter];
    }
    actionsRMaxTable__.clear();
    actionsBoolTable__.clear();
  }

Here is the call graph for this function:

evalPolicy_()

void gum::AdaptiveRMaxPlaner::evalPolicy_ ( )

protectedvirtual

Perform the required tasks to extract an optimal policy.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 195 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction
       = operator_->getFunctionInstance();
    newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());

    std::vector<
       MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >* >
       argMaxQActionsSet;
    // *****************************************************************************************
    // For each action
    for (auto actionIter = fmdp_->beginActions();
         actionIter != fmdp_->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< double >* qAction
         = this->evalQaction_(newVFunction, *actionIter);

      qAction = this->addReward_(qAction, *actionIter);

      qAction = this->operator_->maximize(
         actionsRMaxTable__[*actionIter],
         this->operator_->multiply(qAction, actionsBoolTable__[*actionIter], 1),
         2);

      argMaxQActionsSet.push_back(makeArgMax_(qAction, *actionIter));
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    MultiDimFunctionGraph< ArgMaxSet< double, Idx >, SetTerminalNodePolicy >*
       argMaxVFunction
       = argmaximiseQactions_(argMaxQActionsSet);

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    extractOptimalPolicy_(argMaxVFunction);
  }

Here is the call graph for this function:

evalQaction_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::evalQaction_ ( const MultiDimFunctionGraph< double > *  Vold, Idx  actionId  )

protectedvirtualinherited

Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

Definition at line 353 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                             {
    // ******************************************************************************
    // Initialisation :
    // Creating a copy of last Vfunction to deduce from the new Qaction
    // And finding the first var to eleminate (the one at the end)

    return operator_->regress(Vold, actionId, this->fmdp_, this->elVarSeq_);
  }

extractOptimalPolicy_()

void gum::StructuredPlaner< double >::extractOptimalPolicy_ ( const MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > *  optimalValueFunction )

protectedinherited

From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.

Warning

deallocate the argmax optimal value function

Definition at line 574 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                    {
    optimalPolicy_->clear();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter
         = argMaxOptimalValueFunction->variablesSequence().beginSafe();
         varIter != argMaxOptimalValueFunction->variablesSequence().endSafe();
         ++varIter)
      optimalPolicy_->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    optimalPolicy_->manager()->setRootNode(
       recurExtractOptPol__(argMaxOptimalValueFunction->root(),
                            argMaxOptimalValueFunction,
                            src2dest));

    delete argMaxOptimalValueFunction;
  }

fmdp()

INLINE const FMDP< double >* gum::StructuredPlaner< double >::fmdp ( )

inlineinherited

Returns a const ptr on the Factored Markov Decision Process on which we're planning.

Definition at line 137 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

{ return fmdp_; }

initialize()

void gum::AdaptiveRMaxPlaner::initialize ( const FMDP< double > *  fmdp )

virtual

Initializes data structure needed for making the planning.

Warning

No calling this methods before starting the first makePlaninng will surely and definitely result in a crash

Reimplemented from gum::IDecisionStrategy.

Definition at line 97 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                {
    if (!initialized__) {
      StructuredPlaner::initialize(fmdp);
      IDecisionStrategy::initialize(fmdp);
      for (auto actionIter = fmdp->beginActions();
           actionIter != fmdp->endActions();
           ++actionIter) {
        counterTable__.insert(*actionIter, new StatesCounter());
        initializedTable__.insert(*actionIter, false);
      }
      initialized__ = true;
    }
  }

Here is the call graph for this function:

initVFunction_()

void gum::AdaptiveRMaxPlaner::initVFunction_ ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 133 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                          {
    vFunction_->manager()->setRootNode(
       vFunction_->manager()->addTerminalNode(0.0));
    for (auto actionIter = fmdp_->beginActions();
         actionIter != fmdp_->endActions();
         ++actionIter)
      vFunction_ = this->operator_->add(vFunction_,
                                        RECASTED(this->fmdp_->reward(*actionIter)),
                                        1);
  }

Here is the call graph for this function:

makeArgMax_()

MultiDimFunctionGraph< ArgMaxSet< double , Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< double >::makeArgMax_ ( const MultiDimFunctionGraph< double > *  Qaction, Idx  actionId  )

protectedinherited

Creates a copy of given Qaction that can be exploit by a Argmax.

Hence, this step consists in replacing each lea by an ArgMaxSet containing the value of the leaf and the actionId of the Qaction

Parameters

Qaction	: the function graph we want to transform
actionId	: the action Id associated to that graph

Warning

delete the original Qaction, returns its conversion

Definition at line 485 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                             {
    MultiDimFunctionGraph< ArgMaxSet< GUM_SCALAR, Idx >, SetTerminalNodePolicy >*
       amcpy
       = operator_->getArgMaxFunctionInstance();

    // Insertion des nouvelles variables
    for (SequenceIteratorSafe< const DiscreteVariable* > varIter
         = qAction->variablesSequence().beginSafe();
         varIter != qAction->variablesSequence().endSafe();
         ++varIter)
      amcpy->add(**varIter);

    HashTable< NodeId, NodeId > src2dest;
    amcpy->manager()->setRootNode(
       recurArgMaxCopy__(qAction->root(), actionId, qAction, amcpy, src2dest));

    delete qAction;
    return amcpy;
  }

makePlanning()

void gum::AdaptiveRMaxPlaner::makePlanning ( Idx  nbStep = 1000000 )

virtual

Performs a value iteration.

Parameters

nbStep

: enables you to specify how many value iterations you wish to do. makePlanning will then stop whether when optimal value function is reach or when nbStep have been performed

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 114 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                  {
    makeRMaxFunctionGraphs__();

    StructuredPlaner::makePlanning(nbStep);

    clearTables__();
  }

Here is the call graph for this function:

makeRMaxFunctionGraphs__()

void gum::AdaptiveRMaxPlaner::makeRMaxFunctionGraphs__ ( )

private

Definition at line 240 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                    {
    rThreshold__
       = fmdpLearner__->modaMax() * 5 > 30 ? fmdpLearner__->modaMax() * 5 : 30;
    rmax__ = fmdpLearner__->rMax() / (1.0 - this->discountFactor_);

    for (auto actionIter = this->fmdp()->beginActions();
         actionIter != this->fmdp()->endActions();
         ++actionIter) {
      std::vector< MultiDimFunctionGraph< double >* > rmaxs;
      std::vector< MultiDimFunctionGraph< double >* > boolQs;

      for (auto varIter = this->fmdp()->beginVariables();
           varIter != this->fmdp()->endVariables();
           ++varIter) {
        const IVisitableGraphLearner* visited = counterTable__[*actionIter];

        MultiDimFunctionGraph< double >* varRMax
           = this->operator_->getFunctionInstance();
        MultiDimFunctionGraph< double >* varBoolQ
           = this->operator_->getFunctionInstance();

        visited->insertSetOfVars(varRMax);
        visited->insertSetOfVars(varBoolQ);

        std::pair< NodeId, NodeId > rooty
           = visitLearner__(visited, visited->root(), varRMax, varBoolQ);
        varRMax->manager()->setRootNode(rooty.first);
        varRMax->manager()->reduce();
        varRMax->manager()->clean();
        varBoolQ->manager()->setRootNode(rooty.second);
        varBoolQ->manager()->reduce();
        varBoolQ->manager()->clean();

        rmaxs.push_back(varRMax);
        boolQs.push_back(varBoolQ);

        //          std::cout << RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->toDot() << std::endl;
        //          for( auto varIter2 =
        //          RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->variablesSequence().beginSafe(); varIter2 !=
        //          RECASTED(this->fmdp_->transition(*actionIter,
        //          *varIter))->variablesSequence().endSafe(); ++varIter2 )
        //              std::cout << (*varIter2)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varRMax->toDot() << std::endl;
        //          for( auto varIter =
        //          varRMax->variablesSequence().beginSafe(); varIter !=
        //          varRMax->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;

        //          std::cout << varBoolQ->toDot() << std::endl;
        //          for( auto varIter =
        //          varBoolQ->variablesSequence().beginSafe(); varIter !=
        //          varBoolQ->variablesSequence().endSafe(); ++varIter )
        //              std::cout << (*varIter)->name() << " | ";
        //          std::cout << std::endl;
      }

      //        std::cout << "Maximising" << std::endl;
      actionsRMaxTable__.insert(*actionIter, this->maximiseQactions_(rmaxs));
      actionsBoolTable__.insert(*actionIter, this->minimiseFunctions_(boolQs));
    }
  }

Here is the call graph for this function:

maximiseQactions_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::maximiseQactions_ ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs max_a Q(s,a)

Warning

Performs also the deallocation of the QActions

Definition at line 370 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->maximize(newVFunction, qAction);
    }

    return newVFunction;
  }

minimiseFunctions_()

MultiDimFunctionGraph< double > * gum::StructuredPlaner< double >::minimiseFunctions_ ( std::vector< MultiDimFunctionGraph< double > * > &  qActionsSet )

protectedvirtualinherited

Performs min_i F_i.

Warning

Performs also the deallocation of the F_i

Definition at line 390 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                        {
    MultiDimFunctionGraph< GUM_SCALAR >* newVFunction = qActionsSet.back();
    qActionsSet.pop_back();

    while (!qActionsSet.empty()) {
      MultiDimFunctionGraph< GUM_SCALAR >* qAction = qActionsSet.back();
      qActionsSet.pop_back();
      newVFunction = operator_->minimize(newVFunction, qAction);
    }

    return newVFunction;
  }

optimalPolicy()

INLINE const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::optimalPolicy ( )

inlinevirtualinherited

Returns the best policy obtained so far.

Implements gum::IPlanningStrategy< double >.

Definition at line 157 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                 {
      return optimalPolicy_;
    }

optimalPolicy2String()

std::string gum::StructuredPlaner< double >::optimalPolicy2String ( )

virtualinherited

Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.

Implements gum::IPlanningStrategy< double >.

Definition at line 105 of file structuredPlaner_tpl.h.

References gum::Set< Key, Alloc >::emplace().

                                                                 {
    // ************************************************************************
    // Discarding the case where no \pi* have been computed
    if (!optimalPolicy_ || optimalPolicy_->root() == 0)
      return "NO OPTIMAL POLICY CALCULATED YET";

    // ************************************************************************
    // Initialisation

    // Declaration of the needed string stream
    std::stringstream output;
    std::stringstream terminalStream;
    std::stringstream nonTerminalStream;
    std::stringstream arcstream;

    // First line for the toDot
    output << std::endl << "digraph \" OPTIMAL POLICY \" {" << std::endl;

    // Form line for the internal node stream en the terminal node stream
    terminalStream << "node [shape = box];" << std::endl;
    nonTerminalStream << "node [shape = ellipse];" << std::endl;

    // For somme clarity in the final string
    std::string tab = "\t";

    // To know if we already checked a node or not
    Set< NodeId > visited;

    // FIFO of nodes to visit
    std::queue< NodeId > fifo;

    // Loading the FIFO
    fifo.push(optimalPolicy_->root());
    visited << optimalPolicy_->root();


    // ************************************************************************
    // Main loop
    while (!fifo.empty()) {
      // Node to visit
      NodeId currentNodeId = fifo.front();
      fifo.pop();

      // Checking if it is terminal
      if (optimalPolicy_->isTerminalNode(currentNodeId)) {
        // Get back the associated ActionSet
        ActionSet ase = optimalPolicy_->nodeValue(currentNodeId);

        // Creating a line for this node
        terminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                       << " [label=\"" << currentNodeId << " - ";

        // Enumerating and adding to the line the associated optimal actions
        for (SequenceIteratorSafe< Idx > valIter = ase.beginSafe();
             valIter != ase.endSafe();
             ++valIter)
          terminalStream << fmdp_->actionName(*valIter) << " ";

        // Terminating line
        terminalStream << "\"];" << std::endl;
        continue;
      }

      // Either wise
      {
        // Geting back the associated internal node
        const InternalNode* currentNode = optimalPolicy_->node(currentNodeId);

        // Creating a line in internalnode stream for this node
        nonTerminalStream << tab << currentNodeId << ";" << tab << currentNodeId
                          << " [label=\"" << currentNodeId << " - "
                          << currentNode->nodeVar()->name() << "\"];" << std::endl;

        // Going through the sons and agregating them according the the sons Ids
        HashTable< NodeId, LinkedList< Idx >* > sonMap;
        for (Idx sonIter = 0; sonIter < currentNode->nbSons(); ++sonIter) {
          if (!visited.exists(currentNode->son(sonIter))) {
            fifo.push(currentNode->son(sonIter));
            visited << currentNode->son(sonIter);
          }
          if (!sonMap.exists(currentNode->son(sonIter)))
            sonMap.insert(currentNode->son(sonIter), new LinkedList< Idx >());
          sonMap[currentNode->son(sonIter)]->addLink(sonIter);
        }

        // Adding to the arc stram
        for (auto sonIter = sonMap.beginSafe(); sonIter != sonMap.endSafe();
             ++sonIter) {
          arcstream << tab << currentNodeId << " -> " << sonIter.key()
                    << " [label=\" ";
          Link< Idx >* modaIter = sonIter.val()->list();
          while (modaIter) {
            arcstream << currentNode->nodeVar()->label(modaIter->element());
            if (modaIter->nextLink()) arcstream << ", ";
            modaIter = modaIter->nextLink();
          }
          arcstream << "\",color=\"#00ff00\"];" << std::endl;
          delete sonIter.val();
        }
      }
    }

    // Terminating
    output << terminalStream.str() << std::endl
           << nonTerminalStream.str() << std::endl
           << arcstream.str() << std::endl
           << "}" << std::endl;

    return output.str();
  }

optimalPolicySize()

virtual Size gum::StructuredPlaner< double >::optimalPolicySize ( )

inlinevirtualinherited

Returns optimalPolicy computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 164 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                     {
      return optimalPolicy_ != nullptr ? optimalPolicy_->realSize() : 0;
    }

ReducedAndOrderedInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::ReducedAndOrderedInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 65 of file adaptiveRMaxPlaner.h.

                                                                                 {
      return new AdaptiveRMaxPlaner(new MDDOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

setOptimalStrategy()

void gum::IDecisionStrategy::setOptimalStrategy ( const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > *  optPol )

inlineinherited

Definition at line 90 of file IDecisionStrategy.h.

                                                                                {
      optPol_ = const_cast<
         MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* >(optPol);
    }

spumddInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::spumddInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 80 of file structuredPlaner.h.

                                                        {
      return new StructuredPlaner< GUM_SCALAR >(
         new MDDOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

stateOptimalPolicy()

virtual ActionSet gum::IDecisionStrategy::stateOptimalPolicy ( const Instantiation &  curState )

inlinevirtualinherited

Reimplemented in gum::E_GreedyDecider, and gum::RandomDecider.

Definition at line 96 of file IDecisionStrategy.h.

                                                                        {
      return (optPol_ && optPol_->realSize() != 0) ? optPol_->get(curState)
                                                   : allActions_;
    }

sviInstance()

static StructuredPlaner< double >* gum::StructuredPlaner< double >::sviInstance ( double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestaticinherited

Definition at line 94 of file structuredPlaner.h.

                                                     {
      return new StructuredPlaner< GUM_SCALAR >(
         new TreeOperatorStrategy< GUM_SCALAR >(),
         discountFactor,
         epsilon,
         verbose);
    }

TreeInstance()

static AdaptiveRMaxPlaner* gum::AdaptiveRMaxPlaner::TreeInstance ( const ILearningStrategy *  learner, double  discountFactor = 0.9, double  epsilon = 0.00001, bool  verbose = true  )

inlinestatic

Definition at line 79 of file adaptiveRMaxPlaner.h.

                                                                          {
      return new AdaptiveRMaxPlaner(new TreeOperatorStrategy< double >(),
                                    discountFactor,
                                    epsilon,
                                    learner,
                                    verbose);
    }

valueIteration_()

MultiDimFunctionGraph< double > * gum::AdaptiveRMaxPlaner::valueIteration_ ( )

protectedvirtual

Performs a single step of value iteration.

Reimplemented from gum::StructuredPlaner< double >.

Definition at line 147 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                       {
    // *****************************************************************************************
    // Loop reset
    MultiDimFunctionGraph< double >* newVFunction
       = operator_->getFunctionInstance();
    newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());

    // *****************************************************************************************
    // For each action
    std::vector< MultiDimFunctionGraph< double >* > qActionsSet;
    for (auto actionIter = fmdp_->beginActions();
         actionIter != fmdp_->endActions();
         ++actionIter) {
      MultiDimFunctionGraph< double >* qAction
         = evalQaction_(newVFunction, *actionIter);

      // *******************************************************************************************
      // Next, we add the reward
      qAction = addReward_(qAction, *actionIter);

      qAction = this->operator_->maximize(
         actionsRMaxTable__[*actionIter],
         this->operator_->multiply(qAction, actionsBoolTable__[*actionIter], 1),
         2);

      qActionsSet.push_back(qAction);
    }
    delete newVFunction;

    // *****************************************************************************************
    // Next to evaluate main value function, we take maximise over all action
    // value, ...
    newVFunction = maximiseQactions_(qActionsSet);

    return newVFunction;
  }

Here is the call graph for this function:

vFunction()

INLINE const MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::vFunction ( )

inlineinherited

Returns a const ptr on the value function computed so far.

Definition at line 142 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                                                  {
      return vFunction_;
    }

vFunctionSize()

virtual Size gum::StructuredPlaner< double >::vFunctionSize ( )

inlinevirtualinherited

Returns vFunction computed so far current size.

Implements gum::IPlanningStrategy< double >.

Definition at line 149 of file structuredPlaner.h.

References gum::StructuredPlaner< GUM_SCALAR >::argmaximiseQactions_().

                                 {
      return vFunction_ != nullptr ? vFunction_->realSize() : 0;
    }

visitLearner__()

std::pair< NodeId, NodeId > gum::AdaptiveRMaxPlaner::visitLearner__ ( const IVisitableGraphLearner *  visited, NodeId  currentNodeId, MultiDimFunctionGraph< double > *  rmax, MultiDimFunctionGraph< double > *  boolQ  )

private

Definition at line 311 of file adaptiveRMaxPlaner.cpp.

References gum::Set< Key, Alloc >::emplace().

                                                                                {
    std::pair< NodeId, NodeId > rep;
    if (visited->isTerminal(currentNodeId)) {
      rep.first = rmax->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < rThreshold__ ? rmax__ : 0.0);
      rep.second = boolQ->manager()->addTerminalNode(
         visited->nodeNbObservation(currentNodeId) < rThreshold__ ? 0.0 : 1.0);
      return rep;
    }

    NodeId* rmaxsons = static_cast< NodeId* >(SOA_ALLOCATE(
       sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));
    NodeId* bqsons   = static_cast< NodeId* >(SOA_ALLOCATE(
       sizeof(NodeId) * visited->nodeVar(currentNodeId)->domainSize()));

    for (Idx moda = 0; moda < visited->nodeVar(currentNodeId)->domainSize();
         ++moda) {
      std::pair< NodeId, NodeId > sonp
         = visitLearner__(visited,
                          visited->nodeSon(currentNodeId, moda),
                          rmax,
                          boolQ);
      rmaxsons[moda] = sonp.first;
      bqsons[moda]   = sonp.second;
    }

    rep.first  = rmax->manager()->addInternalNode(visited->nodeVar(currentNodeId),
                                                 rmaxsons);
    rep.second = boolQ->manager()->addInternalNode(visited->nodeVar(currentNodeId),
                                                   bqsons);
    return rep;
  }

Here is the call graph for this function:

Member Data Documentation

actionsBoolTable__

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::actionsBoolTable__

private

Definition at line 189 of file adaptiveRMaxPlaner.h.

actionsRMaxTable__

HashTable< Idx, MultiDimFunctionGraph< double >* > gum::AdaptiveRMaxPlaner::actionsRMaxTable__

private

Definition at line 188 of file adaptiveRMaxPlaner.h.

allActions_

ActionSet gum::IDecisionStrategy::allActions_

protectedinherited

Definition at line 106 of file IDecisionStrategy.h.

counterTable__

HashTable< Idx, StatesCounter* > gum::AdaptiveRMaxPlaner::counterTable__

private

Definition at line 210 of file adaptiveRMaxPlaner.h.

discountFactor_

double gum::StructuredPlaner< double >::discountFactor_

protectedinherited

Discount Factor used for infinite horizon planning.

Definition at line 363 of file structuredPlaner.h.

elVarSeq_

Set< const DiscreteVariable* > gum::StructuredPlaner< double >::elVarSeq_

protectedinherited

A Set to eleminate primed variables.

Definition at line 358 of file structuredPlaner.h.

fmdp_

const FMDP< double >* gum::StructuredPlaner< double >::fmdp_

protectedinherited

The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions )

Definition at line 338 of file structuredPlaner.h.

fmdpLearner__

const ILearningStrategy* gum::AdaptiveRMaxPlaner::fmdpLearner__

private

Definition at line 190 of file adaptiveRMaxPlaner.h.

initialized__

bool gum::AdaptiveRMaxPlaner::initialized__

private

Definition at line 213 of file adaptiveRMaxPlaner.h.

initializedTable__

HashTable< Idx, bool > gum::AdaptiveRMaxPlaner::initializedTable__

private

Definition at line 211 of file adaptiveRMaxPlaner.h.

operator_

IOperatorStrategy< double >* gum::StructuredPlaner< double >::operator_

protectedinherited

Definition at line 365 of file structuredPlaner.h.

optimalPolicy_

MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< double >::optimalPolicy_

protectedinherited

The associated optimal policy.

Warning

Leaves are ActionSet which contains the ids of the best actions While this is sufficient to be exploited, to be understood by a human somme translation from the fmdp_ is required. optimalPolicy2String do this job.

Definition at line 353 of file structuredPlaner.h.

optPol_

const MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::IDecisionStrategy::optPol_

protectedinherited

Definition at line 103 of file IDecisionStrategy.h.

rmax__

double gum::AdaptiveRMaxPlaner::rmax__

private

Definition at line 193 of file adaptiveRMaxPlaner.h.

rThreshold__

double gum::AdaptiveRMaxPlaner::rThreshold__

private

Definition at line 192 of file adaptiveRMaxPlaner.h.

verbose_

bool gum::StructuredPlaner< double >::verbose_

protectedinherited

Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Definition at line 371 of file structuredPlaner.h.

vFunction_

MultiDimFunctionGraph< double >* gum::StructuredPlaner< double >::vFunction_

protectedinherited

The Value Function computed iteratively.

Definition at line 343 of file structuredPlaner.h.

---

The documentation for this class was generated from the following files:

• agrum/FMDP/planning/adaptiveRMaxPlaner.h
• agrum/FMDP/planning/adaptiveRMaxPlaner.cpp