| In nature, the interaction of particles and microscopic structure of matter have historically been concerned by human being. The Standard Model(SM) widely accepted by the theory of particle physics, describes that the fundamental particles are three generation of quarks, three generation of leptons and their anti-particles. The bosons, including photons (γ), gluons (g), W, Z0, which are the force carriers of the electromagnetic, strong and weak interactions. In 2013 the ATLAS and CMS experiment at the LHC (Large Hadron Collider) observed the Higgs boson, which explains the origin of mass. The strongly interacting force between quarks and gluons described by the theory of Quantum Chromodynamics (QCD), explains that colour confinement, quarks and gluons are restrained in the hadrons. From calculations of the lattice Quantum Chromo-dynamics (LQCD), a deconfined phase transition from hadron to quark-gluon plasma (QGP) will occur. In nature, QGP has ever existed in the centre of neutron stars where the baryon density is extremely high and few microseconds after the Big-Bang where extremely high temperatures are reached.Ultra-relativistic heavy-ion experiments allow to investigate the properties of strongly-interacting matter at extreme conditions of temperature and energy den-sity. In the last forty years, several accelerators have been built to evidence and study of the QGP, including the SPS (Super Proton Synchrotron) which focuses on QGP signals, and RHIC (Relativistic Heavy-Ion Collider) which focuses on QGP properties. These accelerators reached to improve our understanding of the QGP, and serie of experimental phenomena associated to the formation of QGP have been observed, such as collective flow, jet quenching, strangeness enhancement and sup-pression of J/φ production. ALICE(A Large Ion Collider Experiment), one of the four major experiments at LHC which started to be operated end of 2009, is dedi-cated to the study of the heavy-ion collisions. The ALICE experiment investigates lead-lead collisions at (?)= 2.76 TeV, and it is expected that a medium with an extremely high temperature and energy density would be produced in the early stage of the collisions.Amongst the most important probes of the properties about the QGP, heavy quarks (charm and beauty) are particular interest since they are expected to be produced in initial hard scattering processes and experience the full evolution of the medium. They are efficient probes of the medium properties.The heavy-flavour production in proton-proton collisions, allows to test of pertur-bative QCD (pQCD) calculations, and provide the reference for p-Pb (proton-lead) and Pb-Pb collisions. The heavy-flavour production in Pb-Pb collisions, may probe the energy density of the system through energy loss of heavy quark. The in-medium effects are usually quantified by means of the nuclear modification factor RAA of the transverse momentum distribution. The value of RAA are smaller than unity if no nuclear modification is present; a RAA value smaller than unity can arise from partonic energy loss well as other nuclear effects.According to QCD, the radiative energy loss should be larger than that of quarks. Due to the dead cone effect, heavy quark energy loss of gluon should be reduced with respect to that of light quarks. In order to distinguish initial-state effects and medium effects in Pb-Pb collisions, one needs to understand cold nuclear matter effects in the initial and final state. Cold nuclear matter effects can be accessed by studying p-Pb collisions.The work of this thesis is based on the study of open heavy flavours in p-Pb and Pb-Pb collisions via single muons measured with the ALICE forward muon spectrometer. The first chapter consists in a general introduction on heavy-ion col-lisions and QCD phase transitions, and summarizes the motivations for the study of open heavy flavours in nucleon-nucleon, nucleon-nucleus and nucleus-nucleus col-lisions. Chapter 2 gives an overview of the ALICE experiment with a description of the forward muon spectrometer, a short summary of the ALICE online and offline systems. Then the summary of analysis framework and in particular the software developed for the study of open heavy flavours is presented. Chapters 3 and 4 are dedicated to data analysis in p-Pb and Pb-Pb collisions. The nuclear modification factor (-RAA) of high-PT muons from heavy-flavour hadron decays has been mea-sured for the first time in Pb-Pb collisions at (?)=NN=2.76 TeV collected in 2010. The transvers momentum range was limited to 4-10 GeV/c. The results show a strong suppression of a factor 3-4 in the 10% most central collisions, providing cv-idence for the large in-medium effects. Further information can be obtained from the study of initial state effects via the measurement of the nuclear modification factor of muons from heavy-flavour hadron decays in p-Pb (RpPb, forward rapidity) and Pb-p(RPbp, backward rapidity) collisions. In chapter 3, I first focus on pile-up effect, normalization and acceptance x efficiency before to prensent the results con-cerning-RpPb(Pbp) that indicate that cold nuclear matter effects are small. Moreover, comparison results within uncertainties for-Rppb(Pbp) and models are shown. Chap-ter 4 addresses the measurement of heavy-flavour decay muon production in Pb-Pb collisions at (?)=2.76 TeV collected in 2011. Due to high statistics and use of muon triggered events, the nuclear modification factor RAA can be measured over a broad transverse momentum range. The values of RAA are still smaller than unity, and within uncertainties, results of heavy-flavour decay muons at forward rapidity are similar to that obtained for mid-rapidity heavy-flavour decay electrons. Finally, the summary and outlooks based on the work of this thesis arc given in Chapter 5. |