Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning Book

List of Figures. List of Tables. Acknowledgements. Preface. 1. Background. 1.1. Why this book was written. 1.2. Simulation-based optimization and modern times. 1.3. How this book is organized. 2. Notation. 2.1. Chapter Overview. 2.2. Some basic conventions. 2.3. Vector notation. 2.4. Notation for matrices. 2.5. Notation for n-tuples. 2.6. Notation for sets. 2.7. Notation for sequences. 2.8. Notation for transformations. 2.9. Max, min and arg max. 2.10. Acronyms and abbreviations. 3. Probability theory: a refresher.3.1. Overview of this chapter. 3.2. Laws of probability. 3.3. Probability distributions. 3.4. Expected value of a random variable. 3.5. Standard deviation of a random variable. 3.6. Limit theorems. 3.7. Review questions. 4. Basic concepts underlying simulation. 4.1. Chapter overview. 4.2. Introductions. 4.3. Models. 4.4. Simulation modeling of random systems. 4.5. Concluding remarks. 4.6. Historical remarks. 4.7. Review questions. 5. Simulation optimization: an overview. 5.1. Chapter overview. 5.2. Shastic parametric optimization. 5.3. Shastic control optimization. 5.4. Historical remarks. 5.5. Review questions. 6. Response surfaces and neural nets. 6.1. Chapter overview. 6.2. RSM: an overview. 6.3. RSM: details. 6.4. Neuro-response surface methods. 6.5. Concluding remarks. 6.6. Bibliographic remarks. 6.7. Review questions. 7. Parametric optimization. 7.1. Chapter overview. 7.2. Continuous optimization. 7.3. Discrete optimization. 7.4. Hybrid solution spaces. 7.5. Concluding remarks. 7.6. Bibliographic remarks. 7.7. Review questions. 8. Dynamic programming. 8.1. Chapter overview. 8.2. Shastic processes. 8.3. Markov processes, Markov chains and semi-Markov processes. 8.4. Markov decision problems. 8.5. How to solve an MDP using exhaustive enumeration. 8.6. Dynamic programming for average reward. 8.7. Dynamic programming and discounted reward. 8.8. The Bellman equation: an intuitive perspective. 8.9. Semi-Markov decision problems. 8.10. Modified policy iteration. 8.11. Miscellaneous topics related to MDPs and SMDPs. 8.12. Conclusions. 8.13. Bibliographic remarks. 8.14. Review questions. 9. Reinforcement learning. 9.1. Chapter overview. 9.2. The need for reinforcement learning. 9.3. Generating the TPM through straightforward counting. 9.4. Reinforcement learning: fundamentals. 9.5. Discounted reward reinforcement learning. 9.6. Average reward reinforcement learning. 9.7. Semi-Markov decision problems and RL. 9.8. RL algorithms and their DP counterparts. 9.9. Actor-critic algorithms. 9.10. Model-building algorithms. 9.11. Finite horizon problems. 9.12. Function approximation. 9.13. Conclusions. 9.14. Bibliographic remarks. 9.15. Review questions. 10. Markov chain automata theory. 10.1. Chapter overview. 10.2. The MCAT framework. 10.3. Concluding remarks. 10.4. Bibliographic remarks. 10.5. Review questions. 11. Convergence: background material. 11.1. Chapter overview. 11.3. Norms. 11.4. Normed vector spaces. 11.5. Functions and mappings. 11.6. Mathematical induction. 11.7. Sequences. 11.8. Sequences in n. 11.9. Cauchy sequences in n. 11.10. Contraction mappings in n. 11.11. Bibliographic remarks. 11.12. Review questions. 12. Convergence: parametric optimization. 12.1. Chapter overview. 12.2. Some definitions and a result. 12.3. Convergence of gradient-descent approaches. 12.4. Perturbation estimates. 12.5. Convergence of simulated annealing. 12.6. Concluding remarks. 12.7. Bibliographic remarks. 12.8. Review questions. 13. Convergence: control optimization. 13.1. Chapter overview. 13.2. Dynamic programming transformations. 13.3. Some definitions. 13.4. Monotonicity of T, Tμ, L, and Lμ. 13.5. Some results for average and discounted MDPs. 13.6. Discounted reward and classical dynamic programming. 13.7. Average reward and classical dynamic programming. 13.8. Convergence of DP schemes for SMDPs. 13.9. Convergence of reinforcement learning schemes. 13.10. Background material for RL convergence. 13.11. Key results for RL convergence. 13.12. Convergence of RL based on value iteration. 13.13. Convergence of Q learning. 13.14. SDMPs. 13.15. Convergence of actor critic algorithms. 13.16. Function approximation and convergence analysis. 13.17. Bibliographic remarks. 13.18. Review questions. 14. Case studies. 14.1. Chapter overview. 14.2. A classical inventory control problem. 14.3. Airline yield management. 14.4. Preventive maintenance. 14.5. Transfer line buffer optimization. 14.6. Inventory control in a supply chain. 14.7. AGV routing. 14.8. Quality control. 14.9. Elevator scheduling. 14.10. Simulation optimization: a comparative perspective. 14.11. Concluding remarks. 14.12. Review questions. 15. Codes. 15.1. Introduction. 15.2. C programming. 15.3. Code organization. 15.4. Random number generators. 15.5. Simultaneous perturbation. 15.6. Dynamic programming codes. 15.7. Codes for neural networks. 15.8. Reinforcement learning codes. 15.9. Codes for the preventative maintenance case study. 15.10. MATLAB codes. 15.11. Concluding remarks. 15.12. Review questions. 16. Concluding remarks. References. Index.