Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing

Alireza Sadeghi, Fatemeh Sheikholeslami, Antonio G. Marques, Georgios B Giannakis

Research output: Contribution to journalArticle

Abstract

Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by caching them at the edge of the network, close to the end users. The ultimate goal is to smartly utilize a limited storage capacity to serve locally contents that are frequently requested instead of fetching them from the cloud, contributing to a better overall network performance and service experience. To enable the SBs with efficient fetch-cache decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs as well as file popularities, an efficient reinforcement learning-based solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of this generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, Q -learning, is employed to find optimal fetch-cache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.

Original languageEnglish (US)
Article number8790766
Pages (from-to)2267-2281
Number of pages15
JournalIEEE Journal on Selected Areas in Communications
Volume37
Issue number10
DOIs
StatePublished - Oct 2019

Fingerprint

Reinforcement learning
Costs
Network performance
Dynamic programming
Base stations
Learning algorithms
Decision making
Availability

Keywords

  • Dynamic caching
  • Q-learning
  • dynamic programming
  • fetching
  • value iteration

Cite this

Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing. / Sadeghi, Alireza; Sheikholeslami, Fatemeh; Marques, Antonio G.; Giannakis, Georgios B.

In: IEEE Journal on Selected Areas in Communications, Vol. 37, No. 10, 8790766, 10.2019, p. 2267-2281.

Research output: Contribution to journalArticle

Sadeghi, Alireza ; Sheikholeslami, Fatemeh ; Marques, Antonio G. ; Giannakis, Georgios B. / Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing. In: IEEE Journal on Selected Areas in Communications. 2019 ; Vol. 37, No. 10. pp. 2267-2281.
@article{da0f4c5a37fb48c0b38721d928abc9b4,
title = "Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing",
abstract = "Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by caching them at the edge of the network, close to the end users. The ultimate goal is to smartly utilize a limited storage capacity to serve locally contents that are frequently requested instead of fetching them from the cloud, contributing to a better overall network performance and service experience. To enable the SBs with efficient fetch-cache decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs as well as file popularities, an efficient reinforcement learning-based solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of this generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, Q -learning, is employed to find optimal fetch-cache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.",
keywords = "Dynamic caching, Q-learning, dynamic programming, fetching, value iteration",
author = "Alireza Sadeghi and Fatemeh Sheikholeslami and Marques, {Antonio G.} and Giannakis, {Georgios B}",
year = "2019",
month = "10",
doi = "10.1109/JSAC.2019.2933780",
language = "English (US)",
volume = "37",
pages = "2267--2281",
journal = "IEEE Journal on Selected Areas in Communications",
issn = "0733-8716",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "10",

}

TY - JOUR

T1 - Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing

AU - Sadeghi, Alireza

AU - Sheikholeslami, Fatemeh

AU - Marques, Antonio G.

AU - Giannakis, Georgios B

PY - 2019/10

Y1 - 2019/10

N2 - Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by caching them at the edge of the network, close to the end users. The ultimate goal is to smartly utilize a limited storage capacity to serve locally contents that are frequently requested instead of fetching them from the cloud, contributing to a better overall network performance and service experience. To enable the SBs with efficient fetch-cache decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs as well as file popularities, an efficient reinforcement learning-based solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of this generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, Q -learning, is employed to find optimal fetch-cache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.

AB - Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by caching them at the edge of the network, close to the end users. The ultimate goal is to smartly utilize a limited storage capacity to serve locally contents that are frequently requested instead of fetching them from the cloud, contributing to a better overall network performance and service experience. To enable the SBs with efficient fetch-cache decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs as well as file popularities, an efficient reinforcement learning-based solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of this generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, Q -learning, is employed to find optimal fetch-cache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.

KW - Dynamic caching

KW - Q-learning

KW - dynamic programming

KW - fetching

KW - value iteration

UR - http://www.scopus.com/inward/record.url?scp=85070720524&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070720524&partnerID=8YFLogxK

U2 - 10.1109/JSAC.2019.2933780

DO - 10.1109/JSAC.2019.2933780

M3 - Article

AN - SCOPUS:85070720524

VL - 37

SP - 2267

EP - 2281

JO - IEEE Journal on Selected Areas in Communications

JF - IEEE Journal on Selected Areas in Communications

SN - 0733-8716

IS - 10

M1 - 8790766

ER -