gum::SDYNA Class Reference

The general SDyna architecture abstract class. More...

#include <agrum/FMDP/SDyna/sdyna.h>

Collaboration diagram for gum::SDYNA:

Public Member Functions
std::string	toString ()
	Returns. More...
std::string	optimalPolicy2String ()
Problem specification methods
void	addAction (const Idx actionId, const std::string &actionName)
	Inserts a new action in the SDyna instance. More...
void	addVariable (const DiscreteVariable *var)
	Inserts a new variable in the SDyna instance. More...
Initialization
void	initialize ()
	Initializes the Sdyna instance. More...
void	initialize (const Instantiation &initialState)
	Initializes the Sdyna instance at given state. More...
Incremental methods
void	setCurrentState (const Instantiation &currentState)
	Sets last state visited to the given state. More...
Idx	takeAction (const Instantiation &curState)
Idx	takeAction ()
void	feedback (const Instantiation &originalState, const Instantiation &reachedState, Idx performedAction, double obtainedReward)
	Performs a feedback on the last transition. More...
void	feedback (const Instantiation &reachedState, double obtainedReward)
	Performs a feedback on the last transition. More...
void	makePlanning (Idx nbStep)
	Starts a new planning. More...
Size methods
just to get the size of the different data structure for performance evaluation purposes only
Size	learnerSize ()
	learnerSize More...
Size	modelSize ()
	modelSize More...
Size	valueFunctionSize ()
	valueFunctionSize More...
Size	optimalPolicySize ()
	optimalPolicySize More...

Static Public Member Functions
static SDYNA *	spitiInstance (double attributeSelectionThreshold=0.99, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...
static SDYNA *	spimddiInstance (double attributeSelectionThreshold=0.99, double similarityThreshold=0.3, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...
static SDYNA *	RMaxMDDInstance (double attributeSelectionThreshold=0.99, double similarityThreshold=0.3, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...
static SDYNA *	RMaxTreeInstance (double attributeSelectionThreshold=0.99, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...
static SDYNA *	RandomMDDInstance (double attributeSelectionThreshold=0.99, double similarityThreshold=0.3, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...
static SDYNA *	RandomTreeInstance (double attributeSelectionThreshold=0.99, double discountFactor=0.9, double epsilon=1, Idx observationPhaseLenght=100, Idx nbValueIterationStep=10)
	@ More...

Protected Attributes
FMDP< double > *	_fmdp
	The learnt Markovian Decision Process. More...
Instantiation	_lastState
	The state in which the system is before we perform a new action. More...

Constructor & destructor.
	SDYNA (ILearningStrategy *learner, IPlanningStrategy< double > *planer, IDecisionStrategy *decider, Idx observationPhaseLenght, Idx nbValueIterationStep, bool actionReward, bool verbose=true)
	Constructor. More...
	~SDYNA ()
	Destructor. More...

Detailed Description

The general SDyna architecture abstract class.

Instance of SDyna architecture should inherit

Definition at line 66 of file sdyna.h.

Constructor & Destructor Documentation

SDYNA()

gum::SDYNA::SDYNA ( ILearningStrategy *  learner, IPlanningStrategy< double > *  planer, IDecisionStrategy *  decider, Idx  observationPhaseLenght, Idx  nbValueIterationStep, bool  actionReward, bool  verbose = true  )

private

Constructor.

Returns

an instance of SDyna architecture

Definition at line 57 of file sdyna.cpp.

References __nbObservation, and _fmdp.

Referenced by RandomMDDInstance(), RandomTreeInstance(), RMaxMDDInstance(), RMaxTreeInstance(), spimddiInstance(), and spitiInstance().

                                                     :
      __learner(learner),
      __planer(planer), __decider(decider),
      __observationPhaseLenght(observationPhaseLenght),
      __nbValueIterationStep(nbValueIterationStep), __actionReward(actionReward),
      _verbose(verbose) {
    GUM_CONSTRUCTOR(SDYNA);

    _fmdp = new FMDP< double >();

    __nbObservation = 1;
  }

Here is the caller graph for this function:

~SDYNA()

gum::SDYNA::~SDYNA

(

)

Destructor.

Definition at line 79 of file sdyna.cpp.

References __bin, __decider, __learner, __planer, and _fmdp.

Referenced by RandomTreeInstance().

                {
    delete __decider;

    delete __learner;

    delete __planer;

    for (auto obsIter = __bin.beginSafe(); obsIter != __bin.endSafe(); ++obsIter)
      delete *obsIter;

    delete _fmdp;

    GUM_DESTRUCTOR(SDYNA);
  }

Here is the caller graph for this function:

Member Function Documentation

addAction()

void gum::SDYNA::addAction ( const Idx  actionId, const std::string &  actionName  )

inline

Inserts a new action in the SDyna instance.

Warning

Without effect until method initialize is called

Parameters

actionId	: an id to identify the action
actionName	: its human name

Definition at line 236 of file sdyna.h.

References _fmdp, and gum::FMDP< GUM_SCALAR >::addAction().

                                                                    {
      _fmdp->addAction(actionId, actionName);
    }

Here is the call graph for this function:

addVariable()

void gum::SDYNA::addVariable ( const DiscreteVariable *  var )

inline

Inserts a new variable in the SDyna instance.

Warning

Without effect until method initialize is called

Parameters

var	: the var to be added. Note that variable may or may not have all its modalities given. If not they will be discovered by the SDyna architecture during the process

Definition at line 250 of file sdyna.h.

References _fmdp, gum::FMDP< GUM_SCALAR >::addVariable(), and initialize().

{ _fmdp->addVariable(var); }

Here is the call graph for this function:

feedback() [1/2]

void gum::SDYNA::feedback	(	const Instantiation &	originalState,
		const Instantiation &	reachedState,
		Idx	performedAction,
		double	obtainedReward
	)

Performs a feedback on the last transition.

Incremental methods.

In extenso, learn from the transition.

Parameters

originalState	: the state we were in before the transition
reachedState	: the state we reached after
performedAction	: the action we performed
obtainedReward	: the reward we obtained

Definition at line 130 of file sdyna.cpp.

References __lastAction, and _lastState.

Referenced by setCurrentState().

                                                    {
    __lastAction = lastAction;
    _lastState = prevState;
    feedback(curState, reward);
  }

Here is the caller graph for this function:

feedback() [2/2]

void gum::SDYNA::feedback	(	const Instantiation &	reachedState,
		double	obtainedReward
	)

Performs a feedback on the last transition.

In extenso, learn from the transition.

Parameters

reachedState	: the state reached after the transition
obtainedReward	: the reward obtained during the transition

Warning

Uses the __originalState and __performedAction stored in cache If you want to specify the original state and the performed action, see below

Definition at line 150 of file sdyna.cpp.

References __actionReward, __bin, __decider, __lastAction, __learner, __nbObservation, __nbValueIterationStep, __observationPhaseLenght, _fmdp, _lastState, gum::ILearningStrategy::addObservation(), gum::IDecisionStrategy::checkState(), gum::FMDP< GUM_SCALAR >::main2prime(), makePlanning(), setCurrentState(), gum::Observation::setModality(), gum::Observation::setReward(), gum::Observation::setRModality(), gum::Instantiation::val(), and gum::Instantiation::variablesSequence().

                                                                   {
    Observation* obs = new Observation();

    for (auto varIter = _lastState.variablesSequence().beginSafe();
         varIter != _lastState.variablesSequence().endSafe();
         ++varIter)
      obs->setModality(*varIter, _lastState.val(**varIter));

    for (auto varIter = newState.variablesSequence().beginSafe();
         varIter != newState.variablesSequence().endSafe();
         ++varIter) {
      obs->setModality(_fmdp->main2prime(*varIter), newState.val(**varIter));

      if (this->__actionReward)
        obs->setRModality(*varIter, _lastState.val(**varIter));
      else
        obs->setRModality(*varIter, newState.val(**varIter));
    }

    obs->setReward(reward);

    __learner->addObservation(__lastAction, obs);
    __bin.insert(obs);

    setCurrentState(newState);
    __decider->checkState(_lastState, __lastAction);

    if (__nbObservation % __observationPhaseLenght == 0)
      makePlanning(__nbValueIterationStep);

    __nbObservation++;
  }

Here is the call graph for this function:

initialize() [1/2]

void gum::SDYNA::initialize

(

)

Initializes the Sdyna instance.

Definition at line 98 of file sdyna.cpp.

References __decider, __learner, __planer, _fmdp, gum::IDecisionStrategy::initialize(), gum::IPlanningStrategy< GUM_SCALAR >::initialize(), and gum::ILearningStrategy::initialize().

Referenced by addVariable(), and initialize().

                         {
    __learner->initialize(_fmdp);
    __planer->initialize(_fmdp);
    __decider->initialize(_fmdp);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

initialize() [2/2]

void gum::SDYNA::initialize

(

const Instantiation &

initialState

)

Initializes the Sdyna instance at given state.

Parameters

initialState

: the state of the studied system from which we will begin the explore, learn and exploit process

Definition at line 111 of file sdyna.cpp.

References initialize(), and setCurrentState().

                                                          {
    initialize();
    setCurrentState(initialState);
  }

Here is the call graph for this function:

learnerSize()

Size gum::SDYNA::learnerSize ( )

inline

learnerSize

Returns

Definition at line 379 of file sdyna.h.

References __learner, and gum::ILearningStrategy::size().

{ return __learner->size(); }

Here is the call graph for this function:

makePlanning()

void gum::SDYNA::makePlanning

(

Idx

nbStep

)

Starts a new planning.

Parameters

nbStep

: the maximal number of value iteration performed in this planning

Definition at line 190 of file sdyna.cpp.

References __decider, __learner, __planer, _verbose, gum::IPlanningStrategy< GUM_SCALAR >::makePlanning(), gum::IPlanningStrategy< GUM_SCALAR >::optimalPolicy(), gum::IDecisionStrategy::setOptimalStrategy(), and gum::ILearningStrategy::updateFMDP().

Referenced by feedback(), and setCurrentState().

                                                   {
    if (_verbose) std::cout << "Updating decision trees ..." << std::endl;
    __learner->updateFMDP();
    // std::cout << << "Done" << std::endl;

    if (_verbose) std::cout << "Planning ..." << std::endl;
    __planer->makePlanning(nbValueIterationStep);
    // std::cout << << "Done" << std::endl;

    __decider->setOptimalStrategy(__planer->optimalPolicy());
  }

Here is the call graph for this function:

Here is the caller graph for this function:

modelSize()

Size gum::SDYNA::modelSize ( )

inline

modelSize

Returns

Definition at line 387 of file sdyna.h.

References _fmdp, and gum::FMDP< GUM_SCALAR >::size().

{ return _fmdp->size(); }

Here is the call graph for this function:

optimalPolicy2String()

std::string gum::SDYNA::optimalPolicy2String ( )

inline

Definition at line 363 of file sdyna.h.

References __planer, and gum::IPlanningStrategy< GUM_SCALAR >::optimalPolicy2String().

{ return __planer->optimalPolicy2String(); }

Here is the call graph for this function:

optimalPolicySize()

Size gum::SDYNA::optimalPolicySize ( )

inline

optimalPolicySize

Returns

Definition at line 403 of file sdyna.h.

References __planer, and gum::IPlanningStrategy< GUM_SCALAR >::optimalPolicySize().

{ return __planer->optimalPolicySize(); }

Here is the call graph for this function:

RandomMDDInstance()

static SDYNA* gum::SDYNA::RandomMDDInstance ( double  attributeSelectionThreshold = 0.99, double  similarityThreshold = 0.3, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 156 of file sdyna.h.

References SDYNA(), and gum::StructuredPlaner< GUM_SCALAR >::spumddInstance().

                                                                      {
      bool               actionReward = true;
      ILearningStrategy* ls = new FMDPLearner< GTEST, GTEST, IMDDILEARNER >(
         attributeSelectionThreshold, actionReward, similarityThreshold);
      IPlanningStrategy< double >* ps =
         StructuredPlaner< double >::spumddInstance(discountFactor, epsilon);
      IDecisionStrategy* ds = new RandomDecider();
      return new SDYNA(
         ls, ps, ds, observationPhaseLenght, nbValueIterationStep, actionReward);
    }

Here is the call graph for this function:

RandomTreeInstance()

static SDYNA* gum::SDYNA::RandomTreeInstance ( double  attributeSelectionThreshold = 0.99, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 175 of file sdyna.h.

References SDYNA(), gum::StructuredPlaner< GUM_SCALAR >::sviInstance(), and ~SDYNA().

                                                                       {
      bool               actionReward = true;
      ILearningStrategy* ls = new FMDPLearner< CHI2TEST, CHI2TEST, ITILEARNER >(
         attributeSelectionThreshold, actionReward);
      IPlanningStrategy< double >* ps =
         StructuredPlaner< double >::sviInstance(discountFactor, epsilon);
      IDecisionStrategy* ds = new RandomDecider();
      return new SDYNA(
         ls, ps, ds, observationPhaseLenght, nbValueIterationStep, actionReward);
    }

Here is the call graph for this function:

RMaxMDDInstance()

static SDYNA* gum::SDYNA::RMaxMDDInstance ( double  attributeSelectionThreshold = 0.99, double  similarityThreshold = 0.3, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 117 of file sdyna.h.

References gum::AdaptiveRMaxPlaner::ReducedAndOrderedInstance(), and SDYNA().

                                                                    {
      bool               actionReward = true;
      ILearningStrategy* ls = new FMDPLearner< GTEST, GTEST, IMDDILEARNER >(
         attributeSelectionThreshold, actionReward, similarityThreshold);
      AdaptiveRMaxPlaner* rm = AdaptiveRMaxPlaner::ReducedAndOrderedInstance(
         ls, discountFactor, epsilon);
      IPlanningStrategy< double >* ps = rm;
      IDecisionStrategy*           ds = rm;
      return new SDYNA(
         ls, ps, ds, observationPhaseLenght, nbValueIterationStep, actionReward);
    }

Here is the call graph for this function:

RMaxTreeInstance()

static SDYNA* gum::SDYNA::RMaxTreeInstance ( double  attributeSelectionThreshold = 0.99, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 137 of file sdyna.h.

References SDYNA(), and gum::AdaptiveRMaxPlaner::TreeInstance().

                                                                     {
      bool               actionReward = true;
      ILearningStrategy* ls = new FMDPLearner< GTEST, GTEST, ITILEARNER >(
         attributeSelectionThreshold, actionReward);
      AdaptiveRMaxPlaner* rm =
         AdaptiveRMaxPlaner::TreeInstance(ls, discountFactor, epsilon);
      IPlanningStrategy< double >* ps = rm;
      IDecisionStrategy*           ds = rm;
      return new SDYNA(
         ls, ps, ds, observationPhaseLenght, nbValueIterationStep, actionReward);
    }

Here is the call graph for this function:

setCurrentState()

void gum::SDYNA::setCurrentState ( const Instantiation &  currentState )

inline

Sets last state visited to the given state.

During the learning process, we will consider that were in this state before the transition.

Parameters

currentState

: the state

Definition at line 292 of file sdyna.h.

References _lastState, feedback(), makePlanning(), takeAction(), and toString().

Referenced by feedback(), and initialize().

                                                            {
      _lastState = currentState;
    }

Here is the call graph for this function:

Here is the caller graph for this function:

spimddiInstance()

static SDYNA* gum::SDYNA::spimddiInstance ( double  attributeSelectionThreshold = 0.99, double  similarityThreshold = 0.3, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 93 of file sdyna.h.

References SDYNA(), and gum::StructuredPlaner< GUM_SCALAR >::spumddInstance().

                                                                    {
      bool               actionReward = false;
      ILearningStrategy* ls = new FMDPLearner< GTEST, GTEST, IMDDILEARNER >(
         attributeSelectionThreshold, actionReward, similarityThreshold);
      IPlanningStrategy< double >* ps = StructuredPlaner< double >::spumddInstance(
         discountFactor, epsilon, false);
      IDecisionStrategy* ds = new E_GreedyDecider();
      return new SDYNA(ls,
                       ps,
                       ds,
                       observationPhaseLenght,
                       nbValueIterationStep,
                       actionReward,
                       false);
    }

Here is the call graph for this function:

spitiInstance()

static SDYNA* gum::SDYNA::spitiInstance ( double  attributeSelectionThreshold = 0.99, double  discountFactor = 0.9, double  epsilon = 1, Idx  observationPhaseLenght = 100, Idx  nbValueIterationStep = 10  )

inlinestatic

@

Definition at line 75 of file sdyna.h.

References SDYNA(), and gum::StructuredPlaner< GUM_SCALAR >::sviInstance().

                                                                  {
      bool               actionReward = false;
      ILearningStrategy* ls = new FMDPLearner< CHI2TEST, CHI2TEST, ITILEARNER >(
         attributeSelectionThreshold, actionReward);
      IPlanningStrategy< double >* ps =
         StructuredPlaner< double >::sviInstance(discountFactor, epsilon);
      IDecisionStrategy* ds = new E_GreedyDecider();
      return new SDYNA(
         ls, ps, ds, observationPhaseLenght, nbValueIterationStep, actionReward);
    }

Here is the call graph for this function:

takeAction() [1/2]

Idx gum::SDYNA::takeAction

(

const Instantiation &

curState

)

Returns

actionId the id of the action the SDyna instance wish to be performed

Parameters

curState

the state in which we currently are

Definition at line 208 of file sdyna.cpp.

References _lastState, and takeAction().

                                                     {
    _lastState = curState;
    return takeAction();
  }

Here is the call graph for this function:

takeAction() [2/2]

Idx gum::SDYNA::takeAction

(

)

Returns

the id of the action the SDyna instance wish to be performed

Definition at line 218 of file sdyna.cpp.

References __decider, __lastAction, _lastState, gum::ActionSet::size(), and gum::IDecisionStrategy::stateOptimalPolicy().

Referenced by setCurrentState(), and takeAction().

                        {
    ActionSet actionSet = __decider->stateOptimalPolicy(_lastState);
    if (actionSet.size() == 1) {
      __lastAction = actionSet[0];
    } else {
      Idx randy = (Idx)((double)std::rand() / (double)RAND_MAX * actionSet.size());
      __lastAction = actionSet[randy == actionSet.size() ? 0 : randy];
    }
    return __lastAction;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

toString()

std::string gum::SDYNA::toString

(

)

Returns.

Returns

a string describing the learned FMDP, and the associated optimal policy. Both in DOT language.

Definition at line 232 of file sdyna.cpp.

References __planer, _fmdp, gum::IPlanningStrategy< GUM_SCALAR >::optimalPolicy2String(), and gum::FMDP< GUM_SCALAR >::toString().

Referenced by setCurrentState().

                            {
    std::stringstream description;

    description << _fmdp->toString() << std::endl;
    description << __planer->optimalPolicy2String() << std::endl;

    return description.str();
  }

Here is the call graph for this function:

Here is the caller graph for this function:

valueFunctionSize()

Size gum::SDYNA::valueFunctionSize ( )

inline

valueFunctionSize

Returns

Definition at line 395 of file sdyna.h.

References __planer, and gum::IPlanningStrategy< GUM_SCALAR >::vFunctionSize().

{ return __planer->vFunctionSize(); }

Here is the call graph for this function:

Member Data Documentation

__actionReward

bool gum::SDYNA::__actionReward

private

Definition at line 441 of file sdyna.h.

Referenced by feedback().

__bin

Set< Observation* > gum::SDYNA::__bin

private

Since SDYNA made these observation, it has to delete them on quitting.

Definition at line 439 of file sdyna.h.

Referenced by feedback(), and ~SDYNA().

__decider

IDecisionStrategy* gum::SDYNA::__decider

private

The decider.

Definition at line 423 of file sdyna.h.

Referenced by feedback(), initialize(), makePlanning(), takeAction(), and ~SDYNA().

__lastAction

Idx gum::SDYNA::__lastAction

private

The last performed action.

Definition at line 436 of file sdyna.h.

Referenced by feedback(), and takeAction().

__learner

ILearningStrategy* gum::SDYNA::__learner

private

The learner used to learn the FMDP.

Definition at line 417 of file sdyna.h.

Referenced by feedback(), initialize(), learnerSize(), makePlanning(), and ~SDYNA().

__nbObservation

Idx gum::SDYNA::__nbObservation

private

The total number of observation made so far.

Definition at line 430 of file sdyna.h.

Referenced by feedback(), and SDYNA().

__nbValueIterationStep

Idx gum::SDYNA::__nbValueIterationStep

private

The number of Value Iteration step we perform.

Definition at line 433 of file sdyna.h.

Referenced by feedback().

__observationPhaseLenght

Idx gum::SDYNA::__observationPhaseLenght

private

The number of observation we make before using again the planer.

Definition at line 427 of file sdyna.h.

Referenced by feedback().

__planer

IPlanningStrategy< double >* gum::SDYNA::__planer

private

The planer used to plan an optimal strategy.

Definition at line 420 of file sdyna.h.

Referenced by initialize(), makePlanning(), optimalPolicy2String(), optimalPolicySize(), toString(), valueFunctionSize(), and ~SDYNA().

_fmdp

FMDP< double >* gum::SDYNA::_fmdp

protected

The learnt Markovian Decision Process.

Definition at line 410 of file sdyna.h.

Referenced by addAction(), addVariable(), feedback(), initialize(), modelSize(), SDYNA(), toString(), and ~SDYNA().

_lastState

Instantiation gum::SDYNA::_lastState

protected

The state in which the system is before we perform a new action.

Definition at line 413 of file sdyna.h.

Referenced by feedback(), setCurrentState(), and takeAction().

_verbose

bool gum::SDYNA::_verbose

private

Definition at line 443 of file sdyna.h.

Referenced by makePlanning().

---

The documentation for this class was generated from the following files:

• agrum/FMDP/SDyna/sdyna.h
• agrum/FMDP/SDyna/sdyna.cpp